Repository: Kong/kubernetes-ingress-controller
Pull request #2923 (2022-09-12 17:02:59)

chore(tests) use 3.x release images
**What this PR does / why we need it**:

Updates the 3.x compatibility test to use release images. Awaiting https://github.com/Kong/kubernetes-ingress-controller/actions/runs/3039067881 as a test run against this branch.

Changes the the various `/bin/sh -c` invocations in container commands to `/bin/bash -c` as all images provide a bash and the default's `/bin/sh` is not compatible with our scripts.

Fixes the error type check for Knative's IngressClass scan; I forgot it earlier when I fixed the not-Knative version.

**Which issue this PR fixes**:
The Enterprise image pull credentials were not properly configured; this fixes those by virtue of no longer requiring them. We want to start using the release images anyway since they're now the newest versions.

However, it breaks the OSS runs because that image isn't actually available. I'm okay with this; they were passing before, and they will need to pass with that image eventually. This version turns the existing compatibility test into a trial run for the release E2E tests, which will require the released image.

rainest: Didn't work. Docker official library image isn't out yet. Not sure what broke the Enterprise one, lacking diagnostics on anything that failed.
rainest: Postgres tests for Enterprise are failing with https://gist.github.com/rainest/e5aaf08e1e4da19b1fd3dc26520f7170 where the initial migrations job will log
```
openid-connect migrated up to: 000_base_openid_connect (executed)
openid-connect migrated up to: 001_14_to_15 (executed)
openid-connect migrated up to: 002_200_to_210 (executed)
openid-connect migrated up to: 003_280_to_300 (executed)
```
and then the attempt to kong start after will fail with
```
Pending migrations:
openid-connect: 002_200_to_210
New migrations available:
                          openid-connect: 003_280_to_300
...
```
so apparently one of those is lying. Kong never comes up and the tests never really start, but not through obvious fault of the Deployment.

Some other failures stem from the removal of `-enterprise-edition` from the version string, since E2E had its own version sniffer.
rainest: Migrations errors appear to have been a bit misleading and are complaining before migrations finish (this is normal) but not succeeding after (this is not). This is caused by Debian's `/bin/sh` (now the default image) not accepting the syntax we use, so it can never check the return value:

```
$ docker run --rm kong/kong-gateway:3.0 /bin/sh -c '[[ 1 ]] && echo "it works"'
/bin/sh: 1: [[: not found
```

bash does accept it and is available on all our platforms (though sometimes it's BusyBox IIRC):

```
17:05:43-0700 esenin $ docker run --rm kong/kong-gateway:3.0.0.0-alpine /bin/bash -c "echo aaa"                                                                                       
aaa
17:09:48-0700 esenin $ docker run --rm kong/kong-gateway:3.0.0.0-debian /bin/bash -c "echo aaa"  
aaa
17:09:55-0700 esenin $ docker run --rm kong/kong-gateway:3.0.0.0-ubuntu /bin/bash -c "echo aaa"  
aaa
17:09:58-0700 esenin $ docker run --rm kong/kong-gateway:3.0.0.0-rhel /bin/bash -c "echo aaa"  
aaa
```

https://github.com/Kong/kubernetes-ingress-controller/actions/runs/3041434734 is currently running with a fixed version tester and a change to bash. May yet uncover new problems. OSS will still fail since that image is missing, but I'm less concerned with it since it was passing during earlier scheduled runs.
rainest: Failed because these manifests were still using busybox (the image, not the binary), which does not include bash. Now trying with the Kong image instead: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/3041700933

We already replaced busybox with the Kong image long ago when we made a similar change to bash from sh in the chart; probably should have changed this then too.

Patch:
diff --git a/.github/workflows/kong3.0_compatibility.yaml b/.github/workflows/kong3.0_compatibility.yaml
index 2cbb2f414..8d769c74b 100644
--- a/.github/workflows/kong3.0_compatibility.yaml
+++ b/.github/workflows/kong3.0_compatibility.yaml
@@ -44,15 +44,15 @@ jobs:
       id: setup_env_oss
       if: ${{ matrix.kong-edition == 'OSS' }}
       run: |
-        echo "TEST_KONG_IMAGE=kong/kong" >> $GITHUB_ENV
-        echo "TEST_KONG_TAG=3.0.0-alpha.14-alpine" >> $GITHUB_ENV
+        echo "TEST_KONG_IMAGE=kong" >> $GITHUB_ENV
+        echo "TEST_KONG_TAG=3.0" >> $GITHUB_ENV
 
     - name: setup test kong image and env for enterprise edition
       id: setup_env_ee
       if: ${{ matrix.kong-edition == 'enterprise' }}
       run: |
-        echo "TEST_KONG_IMAGE=kong/kong-gateway-internal" >> $GITHUB_ENV
-        echo "TEST_KONG_TAG=3.0.x.x-alpine" >> $GITHUB_ENV
+        echo "TEST_KONG_IMAGE=kong/kong-gateway" >> $GITHUB_ENV
+        echo "TEST_KONG_TAG=3.0" >> $GITHUB_ENV
         echo "TEST_KONG_ENTERPRISE=true" >> $GITHUB_ENV
 
     - name: setup database mode for dbless
@@ -141,7 +141,7 @@ jobs:
       env:
         TEST_KONG_CONTROLLER_IMAGE_LOAD: kong/kubernetes-ingress-controller:ci
         TEST_KONG_CONTROLLER_IMAGE_OVERRIDE: kong/kubernetes-ingress-controller:ci
-        TEST_KONG_IMAGE_OVERRIDE: kong/kong:3.0.0-alpha.14-alpine
+        TEST_KONG_IMAGE_OVERRIDE: kong:3.0
         KONG_CLUSTER_VERSION: "v1.24.2"
         ISTIO_VERSION: "v1.14.1"
         NCPU: 1
@@ -206,7 +206,7 @@ jobs:
       env:
         TEST_KONG_CONTROLLER_IMAGE_LOAD: kong/kubernetes-ingress-controller:ci
         TEST_KONG_CONTROLLER_IMAGE_OVERRIDE: kong/kubernetes-ingress-controller:ci
-        TEST_KONG_IMAGE_OVERRIDE: kong/kong-gateway-internal:3.0.x.x-alpine
+        TEST_KONG_IMAGE_OVERRIDE: kong/kong-gateway:3.0
         KONG_LICENSE_DATA: ${{ steps.license.outputs.license }}
         TEST_KONG_PULL_USERNAME: ${{ secrets.GHA_DOCKERHUB_PULL_USER }}
         TEST_KONG_PULL_PASSWORD: ${{ secrets.GHA_KONG_ORG_DOCKERHUB_PUBLIC_TOKEN }}
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 94793a4f8..27456f318 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -66,6 +66,12 @@
   to preserve existing behavior, add `PathType=prefix` configuration to those
   rules.
   [#2883](https://github.com/Kong/kubernetes-ingress-controller/pull/2883)
+- The GatewayClass objects now require the annotation
+  "konghq.com/gatewayclass-unmanaged" to be reconciled by the controller.
+  The annotation "konghq.com/gateway-unmanaged" is not considered anymore and
+  doesn't need to be set on Gateways to be reconciled. Only the Gateways using
+  an unmanaged GatewayClass are reconciled.
+  [#2917](https://github.com/Kong/kubernetes-ingress-controller/pull/2917)
 
 #### Added
 
@@ -105,6 +111,9 @@
   [#2889](https://github.com/Kong/kubernetes-ingress-controller/issues/2889)
   [#2894](https://github.com/Kong/kubernetes-ingress-controller/issues/2894)
   [#2900](https://github.com/Kong/kubernetes-ingress-controller/issues/2900)
+- Manifests now use `/bin/bash` instead of `/bin/sh` and use bash-based
+  connectivity checks for compatibility with the new Debian Kong images.
+  [#2923](https://github.com/Kong/kubernetes-ingress-controller/issues/2923)
 
 #### Fixed
 
diff --git a/Makefile b/Makefile
index d0703b2ba..8cd775999 100644
--- a/Makefile
+++ b/Makefile
@@ -315,7 +315,7 @@ test.integration.enterprise.postgres.pretty:
 
 .PHONY: test.e2e
 test.e2e:
-	GOFLAGS="-tags=e2e_tests" go test -v \
+	GOFLAGS="-tags=e2e_tests" go test -v $(GOTESTFLAGS) \
 		-race \
 		-parallel $(NCPU) \
 		-timeout $(E2E_TEST_TIMEOUT) \
diff --git a/config/base/kong-ingress-dbless.yaml b/config/base/kong-ingress-dbless.yaml
index 11160c841..a78194408 100644
--- a/config/base/kong-ingress-dbless.yaml
+++ b/config/base/kong-ingress-dbless.yaml
@@ -69,7 +69,7 @@ spec:
         lifecycle:
           preStop:
             exec:
-              command: [ "/bin/sh", "-c", "kong quit" ]
+              command: [ "/bin/bash", "-c", "kong quit" ]
         ports:
         - name: proxy
           containerPort: 8000
diff --git a/config/variants/postgres/kong-ingress-postgres.yaml b/config/variants/postgres/kong-ingress-postgres.yaml
index 90d29cafd..7a7682f0e 100644
--- a/config/variants/postgres/kong-ingress-postgres.yaml
+++ b/config/variants/postgres/kong-ingress-postgres.yaml
@@ -10,7 +10,7 @@ spec:
       - name: wait-for-migrations
         image: kong-placeholder:placeholder
         command:
-        - "/bin/sh"
+        - "/bin/bash"
         - "-c"
         - "while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi; sleep 2;  done;"
         env:
diff --git a/config/variants/postgres/migration.yaml b/config/variants/postgres/migration.yaml
index 8c7a13df9..5d1731099 100644
--- a/config/variants/postgres/migration.yaml
+++ b/config/variants/postgres/migration.yaml
@@ -11,13 +11,13 @@ spec:
     spec:
       initContainers:
       - name: wait-for-postgres
-        image: busybox
+        image: kong-placeholder:placeholder
         env:
         - name: KONG_PG_HOST
           value: postgres
         - name: KONG_PG_PORT
           value: "5432"
-        command: [ "/bin/sh", "-c", "until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db'; sleep 1; done" ]
+        command: [ "/bin/bash", "-c", "until timeout 1 bash 9<>/dev/tcp/${KONG_PG_HOST}/${KONG_PG_PORT}; do echo 'waiting for db'; sleep 1; done" ]
       containers:
       - name: kong-migrations
         image: kong-placeholder:placeholder
@@ -28,5 +28,5 @@ spec:
           value: postgres
         - name: KONG_PG_PORT
           value: "5432"
-        command: [ "/bin/sh", "-c", "kong migrations bootstrap" ]
+        command: [ "/bin/bash", "-c", "kong migrations bootstrap" ]
       restartPolicy: OnFailure
diff --git a/config/variants/postgres/wait/kong-ingress-postgres.yaml b/config/variants/postgres/wait/kong-ingress-postgres.yaml
index 90d29cafd..7a7682f0e 100644
--- a/config/variants/postgres/wait/kong-ingress-postgres.yaml
+++ b/config/variants/postgres/wait/kong-ingress-postgres.yaml
@@ -10,7 +10,7 @@ spec:
       - name: wait-for-migrations
         image: kong-placeholder:placeholder
         command:
-        - "/bin/sh"
+        - "/bin/bash"
         - "-c"
         - "while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi; sleep 2;  done;"
         env:
diff --git a/deploy/single/all-in-one-dbless-k4k8s-enterprise.yaml b/deploy/single/all-in-one-dbless-k4k8s-enterprise.yaml
index 302ccd4f1..8a3c8c343 100644
--- a/deploy/single/all-in-one-dbless-k4k8s-enterprise.yaml
+++ b/deploy/single/all-in-one-dbless-k4k8s-enterprise.yaml
@@ -1601,7 +1601,7 @@ spec:
           preStop:
             exec:
               command:
-              - /bin/sh
+              - /bin/bash
               - -c
               - kong quit
         livenessProbe:
diff --git a/deploy/single/all-in-one-dbless.yaml b/deploy/single/all-in-one-dbless.yaml
index 9996d95a1..9dc94da80 100644
--- a/deploy/single/all-in-one-dbless.yaml
+++ b/deploy/single/all-in-one-dbless.yaml
@@ -1596,7 +1596,7 @@ spec:
           preStop:
             exec:
               command:
-              - /bin/sh
+              - /bin/bash
               - -c
               - kong quit
         livenessProbe:
diff --git a/deploy/single/all-in-one-postgres-enterprise.yaml b/deploy/single/all-in-one-postgres-enterprise.yaml
index b4dd8397f..5eeb0a567 100644
--- a/deploy/single/all-in-one-postgres-enterprise.yaml
+++ b/deploy/single/all-in-one-postgres-enterprise.yaml
@@ -1659,7 +1659,7 @@ spec:
           preStop:
             exec:
               command:
-              - /bin/sh
+              - /bin/bash
               - -c
               - kong quit
         livenessProbe:
@@ -1759,7 +1759,7 @@ spec:
       - name: kong-enterprise-edition-docker
       initContainers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
         - while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi;
           sleep 2;  done;
@@ -1845,7 +1845,7 @@ spec:
     spec:
       containers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
         - kong migrations bootstrap
         env:
@@ -1871,16 +1871,16 @@ spec:
       - name: kong-enterprise-edition-docker
       initContainers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
-        - until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db';
-          sleep 1; done
+        - until timeout 1 bash 9<>/dev/tcp/${KONG_PG_HOST}/${KONG_PG_PORT}; do echo
+          'waiting for db'; sleep 1; done
         env:
         - name: KONG_PG_HOST
           value: postgres
         - name: KONG_PG_PORT
           value: "5432"
-        image: busybox
+        image: kong/kong-gateway:2.8
         name: wait-for-postgres
       restartPolicy: OnFailure
 ---
diff --git a/deploy/single/all-in-one-postgres.yaml b/deploy/single/all-in-one-postgres.yaml
index a564748ba..b754178cb 100644
--- a/deploy/single/all-in-one-postgres.yaml
+++ b/deploy/single/all-in-one-postgres.yaml
@@ -1614,7 +1614,7 @@ spec:
           preStop:
             exec:
               command:
-              - /bin/sh
+              - /bin/bash
               - -c
               - kong quit
         livenessProbe:
@@ -1701,7 +1701,7 @@ spec:
           readOnly: true
       initContainers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
         - while true; do kong migrations list; if [[ 0 -eq $? ]]; then exit 0; fi;
           sleep 2;  done;
@@ -1782,7 +1782,7 @@ spec:
     spec:
       containers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
         - kong migrations bootstrap
         env:
@@ -1796,16 +1796,16 @@ spec:
         name: kong-migrations
       initContainers:
       - command:
-        - /bin/sh
+        - /bin/bash
         - -c
-        - until nc -zv $KONG_PG_HOST $KONG_PG_PORT -w1; do echo 'waiting for db';
-          sleep 1; done
+        - until timeout 1 bash 9<>/dev/tcp/${KONG_PG_HOST}/${KONG_PG_PORT}; do echo
+          'waiting for db'; sleep 1; done
         env:
         - name: KONG_PG_HOST
           value: postgres
         - name: KONG_PG_PORT
           value: "5432"
-        image: busybox
+        image: kong:2.8
         name: wait-for-postgres
       restartPolicy: OnFailure
 ---
diff --git a/examples/gateway-httproute.yaml b/examples/gateway-httproute.yaml
index 4637c5f4f..f96948047 100644
--- a/examples/gateway-httproute.yaml
+++ b/examples/gateway-httproute.yaml
@@ -81,6 +81,8 @@ apiVersion: gateway.networking.k8s.io/v1beta1
 kind: GatewayClass
 metadata:
   name: kong
+  annotations:
+    konghq.com/gatewayclass-unmanaged: "true"
 spec:
   controllerName: konghq.com/kic-gateway-controller
 ---
diff --git a/examples/gateway-tcproute.yaml b/examples/gateway-tcproute.yaml
index 6baceb52f..c1b8f2970 100644
--- a/examples/gateway-tcproute.yaml
+++ b/examples/gateway-tcproute.yaml
@@ -46,6 +46,8 @@ apiVersion: gateway.networking.k8s.io/v1beta1
 kind: GatewayClass
 metadata:
   name: kong
+  annotations:
+    konghq.com/gatewayclass-unmanaged: "true"
 spec:
   controllerName: konghq.com/kic-gateway-controller
 ---
diff --git a/examples/gateway-tlsroute.yaml b/examples/gateway-tlsroute.yaml
index e230c8643..76abf88ba 100644
--- a/examples/gateway-tlsroute.yaml
+++ b/examples/gateway-tlsroute.yaml
@@ -55,6 +55,8 @@ apiVersion: gateway.networking.k8s.io/v1beta1
 kind: GatewayClass
 metadata:
   name: kong
+  annotations:
+    konghq.com/gatewayclass-unmanaged: "true"
 spec:
   controllerName: konghq.com/kic-gateway-controller
 ---
diff --git a/examples/gateway-udproute.yaml b/examples/gateway-udproute.yaml
index 2b37589f1..5c6d057af 100644
--- a/examples/gateway-udproute.yaml
+++ b/examples/gateway-udproute.yaml
@@ -86,6 +86,8 @@ apiVersion: gateway.networking.k8s.io/v1beta1
 kind: GatewayClass
 metadata:
   name: kong
+  annotations:
+    konghq.com/gatewayclass-unmanaged: "true"
 spec:
   controllerName: konghq.com/kic-gateway-controller
 ---
diff --git a/internal/annotations/annotations.go b/internal/annotations/annotations.go
index 901b2428d..ef4d8eec7 100644
--- a/internal/annotations/annotations.go
+++ b/internal/annotations/annotations.go
@@ -17,6 +17,7 @@ limitations under the License.
 package annotations
 
 import (
+	"fmt"
 	"strings"
 
 	netv1 "k8s.io/api/networking/v1"
@@ -55,19 +56,27 @@ const (
 	ResponseBuffering    = "/response-buffering"
 	HostAliasesKey       = "/host-aliases"
 
-	// GatewayUnmanagedAnnotation is an annotation used on a Gateway resource to
-	// indicate that the Gateway should be reconciled according to unmanaged
+	// GatewayClassUnmanagedAnnotationSuffix is an annotation used on a Gateway resource to
+	// indicate that the GatewayClass should be reconciled according to unmanaged
 	// mode.
 	//
-	// NOTE: it's currently required that this annotation be present on all Gateway
+	// NOTE: it's currently required that this annotation be present on all GatewayClass
 	// resources: "unmanaged" mode is the only supported mode at this time.
-	GatewayUnmanagedAnnotation = "/gateway-unmanaged"
+	GatewayClassUnmanagedAnnotationSuffix = "gatewayclass-unmanaged"
 
 	// DefaultIngressClass defines the default class used
 	// by Kong's ingress controller.
 	DefaultIngressClass = "kong"
+
+	// GatewayClassUnmanagedAnnotationPlaceholder is intended to be used as placeholder value for the
+	// GatewayClassUnmanagedAnnotation annotation.
+	GatewayClassUnmanagedAnnotationValuePlaceholder = "true"
 )
 
+// GatewayClassUnmanagedAnnotation is the complete annotations for unmanaged mode made by the konhq.com prefix
+// followed by the gatewayclass-unmanaged GatewayClass suffix.
+var GatewayClassUnmanagedAnnotation = fmt.Sprintf("%s/%s", AnnotationPrefix, GatewayClassUnmanagedAnnotationSuffix)
+
 func validIngress(ingressAnnotationValue, ingressClass string, handling ClassMatching) bool {
 	switch handling {
 	case IgnoreClassMatch:
@@ -243,9 +252,16 @@ func ExtractHostAliases(anns map[string]string) ([]string, bool) {
 	return strings.Split(val, ","), true
 }
 
-// ExtractUnmanagedGatewayMode extracts the value of the unmanaged gateway
+// ExtractUnmanagedGatewayClassMode extracts the value of the unmanaged gateway
 // mode annotation.
-func ExtractUnmanagedGatewayMode(anns map[string]string) (string, bool) {
-	s, ok := anns[AnnotationPrefix+GatewayUnmanagedAnnotation]
-	return s, ok
+func ExtractUnmanagedGatewayClassMode(anns map[string]string) string {
+	if anns == nil {
+		return ""
+	}
+	return anns[GatewayClassUnmanagedAnnotation]
+}
+
+// UpdateUnmanagedAnnotation updates the value of the annotation konghq.com/gatewayclass-unmanaged.
+func UpdateUnmanagedAnnotation(anns map[string]string, annotationValue string) {
+	anns[GatewayClassUnmanagedAnnotation] = annotationValue
 }
diff --git a/internal/controllers/gateway/gateway_controller.go b/internal/controllers/gateway/gateway_controller.go
index 55461b5e3..44b21fed0 100644
--- a/internal/controllers/gateway/gateway_controller.go
+++ b/internal/controllers/gateway/gateway_controller.go
@@ -2,13 +2,14 @@ package gateway
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"reflect"
 
 	"github.com/go-logr/logr"
 	"github.com/kong/go-kong/kong"
 	corev1 "k8s.io/api/core/v1"
-	"k8s.io/apimachinery/pkg/api/errors"
+	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
@@ -31,10 +32,8 @@ import (
 // -----------------------------------------------------------------------------
 
 var (
-	// ErrManagedGatewaysUnsupported is an error used whenever a failure occurs
-	// due to a Gateway that is not properly configured for unmanaged mode.
-	ErrManagedGatewaysUnsupported = fmt.Errorf("invalid gateway spec: managed gateways are not currently supported")
-	gatewayV1beta1Group           = gatewayv1beta1.Group(gatewayv1beta1.GroupName)
+	ErrUnmanagedAnnotation = errors.New("invalid unmanaged annotation value")
+	gatewayV1beta1Group    = gatewayv1beta1.Group(gatewayv1beta1.GroupName)
 )
 
 // -----------------------------------------------------------------------------
@@ -135,7 +134,7 @@ func (r *GatewayReconciler) SetupWithManager(mgr ctrl.Manager) error {
 // -----------------------------------------------------------------------------
 
 // gatewayHasMatchingGatewayClass is a watch predicate which filters out reconciliation events for
-// gateway objects which aren't supported by this controller.
+// gateway objects which aren't supported by this controller or not using an unmanaged GatewayClass.
 func (r *GatewayReconciler) gatewayHasMatchingGatewayClass(obj client.Object) bool {
 	gateway, ok := obj.(*gatewayv1beta1.Gateway)
 	if !ok {
@@ -147,18 +146,18 @@ func (r *GatewayReconciler) gatewayHasMatchingGatewayClass(obj client.Object) bo
 		r.Log.Error(err, "could not retrieve gatewayclass", "gatewayclass", gateway.Spec.GatewayClassName)
 		return false
 	}
-	return gatewayClass.Spec.ControllerName == ControllerName
+	return isGatewayClassControlledAndUmanaged(gatewayClass)
 }
 
 // gatewayClassMatchesController is a watch predicate which filters out events for gatewayclasses which
-// aren't configured with the required ControllerName, e.g. they are not supported by this controller.
+// aren't configured with the required ControllerName or not annotated as unmanaged.
 func (r *GatewayReconciler) gatewayClassMatchesController(obj client.Object) bool {
 	gatewayClass, ok := obj.(*gatewayv1beta1.GatewayClass)
 	if !ok {
 		r.Log.Error(fmt.Errorf("unexpected object type in gatewayclass watch predicates"), "expected", "*gatewayv1beta1.GatewayClass", "found", reflect.TypeOf(obj))
 		return false
 	}
-	return gatewayClass.Spec.ControllerName == ControllerName
+	return isGatewayClassControlledAndUmanaged(gatewayClass)
 }
 
 // listGatewaysForGatewayClass is a watch predicate which finds all the gateway objects reference
@@ -221,7 +220,7 @@ func (r *GatewayReconciler) listGatewaysForService(svc client.Object) (recs []re
 			r.Log.Error(err, "failed to retrieve gateway class in watch predicates", "gatewayclass", gateway.Spec.GatewayClassName)
 			return
 		}
-		if isGatewayInClassAndUnmanaged(gatewayClass, gateway) {
+		if isGatewayClassControlledAndUmanaged(gatewayClass) {
 			recs = append(recs, reconcile.Request{
 				NamespacedName: types.NamespacedName{
 					Namespace: gateway.Namespace,
@@ -268,7 +267,7 @@ func (r *GatewayReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 	// to be gone at this point in which case it will be ignored.
 	gateway := new(gatewayv1beta1.Gateway)
 	if err := r.Get(ctx, req.NamespacedName, gateway); err != nil {
-		if errors.IsNotFound(err) {
+		if k8serrors.IsNotFound(err) {
 			debug(log, gateway, "reconciliation triggered but gateway does not exist, ignoring")
 			return ctrl.Result{Requeue: false}, nil
 		}
@@ -310,6 +309,14 @@ func (r *GatewayReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ct
 		return ctrl.Result{Requeue: false}, nil
 	}
 
+	// ensure that the GatewayClass matches the ControllerName and is unmanaged.
+	// This check has already been performed by predicates, but we need to ensure this condition
+	// as the reconciliation loop may be triggered by objects in which predicates we
+	// cannot check the ControllerName and the unmanaged mode (e.g., ReferenceGrants).
+	if !isGatewayClassControlledAndUmanaged(gwc) {
+		return reconcile.Result{}, nil
+	}
+
 	// reconciliation assumes unmanaged mode, in the future we may have a slot here for
 	// other gateway management modes.
 	result, err := r.reconcileUnmanagedGateway(ctx, log, gateway)
@@ -332,26 +339,23 @@ func (r *GatewayReconciler) reconcileUnmanagedGateway(ctx context.Context, log l
 	// any Gateway object that comes to us is configured appropriately, and if not reject it
 	// with a clear status condition and message.
 	debug(log, gateway, "validating management mode for gateway") // this will also be done by the validating webhook, this is a fallback
-	unmanagedAnnotation := annotations.AnnotationPrefix + annotations.GatewayUnmanagedAnnotation
-	existingGatewayEnabled, ok := annotations.ExtractUnmanagedGatewayMode(gateway.GetAnnotations())
 
-	// allow for Gateway resources to be configured with "true" in place of the publish service
-	// reference as a placeholder to automatically populate the annotation with the namespace/name
-	// that was provided to the controller manager via --publish-service.
+	// enforce the service reference as the annotation value for the key UnmanagedGateway.
 	debug(log, gateway, "initializing admin service annotation if unset")
-	if !ok || existingGatewayEnabled == "true" { // true is a placeholder which triggers auto-initialization of the ref
-		debug(log, gateway, fmt.Sprintf("a placeholder value was provided for %s, adding the default service ref %s", unmanagedAnnotation, r.PublishService))
+	if !isObjectUnmanaged(gateway.GetAnnotations()) {
+		debug(log, gateway, fmt.Sprintf("a placeholder value was provided for %s, adding the default service ref %s", annotations.GatewayClassUnmanagedAnnotation, r.PublishService))
 		if gateway.Annotations == nil {
-			gateway.Annotations = make(map[string]string)
+			gateway.Annotations = map[string]string{}
 		}
-		gateway.Annotations[unmanagedAnnotation] = r.PublishService
+		annotations.UpdateUnmanagedAnnotation(gateway.Annotations, r.PublishService)
 		return ctrl.Result{}, r.Update(ctx, gateway)
 	}
 
+	serviceRef := annotations.ExtractUnmanagedGatewayClassMode(gateway.Annotations)
 	// validation check of the Gateway to ensure that the publish service is actually available
 	// in the cluster. If it is not the object will be requeued until it exists (or is otherwise retrievable).
 	debug(log, gateway, "gathering the gateway publish service") // this will also be done by the validating webhook, this is a fallback
-	svc, err := r.determineServiceForGateway(ctx, existingGatewayEnabled)
+	svc, err := r.determineServiceForGateway(ctx, serviceRef)
 	if err != nil {
 		log.Error(err, "could not determine service for gateway", "namespace", gateway.Namespace, "name", gateway.Name)
 		return ctrl.Result{Requeue: true}, err
@@ -394,7 +398,7 @@ func (r *GatewayReconciler) reconcileUnmanagedGateway(ctx context.Context, log l
 		debug(log, gateway, "updating addresses to match Kong proxy Service")
 		gateway.Spec.Addresses = kongAddresses
 		if err := r.Update(ctx, gateway); err != nil {
-			if errors.IsConflict(err) {
+			if k8serrors.IsConflict(err) {
 				// if there's a conflict that's normal just requeue to retry, no need to make noise.
 				return ctrl.Result{Requeue: true}, nil
 			}
@@ -427,7 +431,7 @@ func (r *GatewayReconciler) reconcileUnmanagedGateway(ctx context.Context, log l
 	debug(log, gateway, "updating the gateway status if necessary")
 	isChanged, err := r.updateAddressesAndListenersStatus(ctx, gateway, listenerStatuses)
 	if err != nil {
-		if errors.IsConflict(err) {
+		if k8serrors.IsConflict(err) {
 			// if there's a conflict that's normal just requeue to retry, no need to make noise.
 			return ctrl.Result{Requeue: true}, nil
 		}
diff --git a/internal/controllers/gateway/gateway_utils.go b/internal/controllers/gateway/gateway_utils.go
index 1d2f4f315..cd3923226 100644
--- a/internal/controllers/gateway/gateway_utils.go
+++ b/internal/controllers/gateway/gateway_utils.go
@@ -66,11 +66,17 @@ func isGatewayReady(gateway *Gateway) bool {
 	return false
 }
 
-// isGatewayInClassAndUnmanaged returns boolean if the provided combination of gateway and class
-// is controlled by this controller and the gateway is configured for unmanaged mode.
-func isGatewayInClassAndUnmanaged(gatewayClass *GatewayClass, gateway Gateway) bool {
-	_, ok := annotations.ExtractUnmanagedGatewayMode(gateway.Annotations)
-	return ok && gatewayClass.Spec.ControllerName == ControllerName
+// isObjectUnmanaged returns boolean if the object is configured
+// for unmanaged mode.
+func isObjectUnmanaged(anns map[string]string) bool {
+	annotationValue := annotations.ExtractUnmanagedGatewayClassMode(anns)
+	return annotationValue != ""
+}
+
+// isGatewayClassControlledAndUmanaged returns boolean if the GatewayClass
+// is controlled by this controller and is configured for unmanaged mode.
+func isGatewayClassControlledAndUmanaged(gatewayClass *GatewayClass) bool {
+	return gatewayClass.Spec.ControllerName == ControllerName && isObjectUnmanaged(gatewayClass.Annotations)
 }
 
 // getRefFromPublishService splits a publish service string in the format namespace/name into a types.NamespacedName
diff --git a/internal/controllers/gateway/gatewayclass_controller.go b/internal/controllers/gateway/gatewayclass_controller.go
index e1ea622d6..3a9b22a82 100644
--- a/internal/controllers/gateway/gatewayclass_controller.go
+++ b/internal/controllers/gateway/gatewayclass_controller.go
@@ -2,6 +2,8 @@ package gateway
 
 import (
 	"context"
+	"fmt"
+	"reflect"
 
 	"github.com/go-logr/logr"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -11,6 +13,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 	"sigs.k8s.io/controller-runtime/pkg/source"
 	gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
@@ -22,12 +25,10 @@ import (
 // GatewayClass Controller - Vars & Consts
 // -----------------------------------------------------------------------------
 
-const (
-	// ControllerName is the unique identifier for this controller and is used
-	// within GatewayClass resources to indicate that this controller should
-	// support connected Gateway resources.
-	ControllerName gatewayv1beta1.GatewayController = "konghq.com/kic-gateway-controller"
-)
+// ControllerName is the unique identifier for this controller and is used
+// within GatewayClass resources to indicate that this controller should
+// support connected Gateway resources.
+var ControllerName gatewayv1beta1.GatewayController = "konghq.com/kic-gateway-controller"
 
 // -----------------------------------------------------------------------------
 // GatewayClass Controller - Reconciler
@@ -54,11 +55,28 @@ func (r *GatewayClassReconciler) SetupWithManager(mgr ctrl.Manager) error {
 	return c.Watch(
 		&source.Kind{Type: &gatewayv1beta1.GatewayClass{}},
 		&handler.EnqueueRequestForObject{},
+		predicate.NewPredicateFuncs(r.GatewayClassIsUnmanaged),
 	)
 }
 
 // -----------------------------------------------------------------------------
-// Gateway Controller - Reconciliation
+// GatewayClass Controller - Watch Predicates
+// -----------------------------------------------------------------------------
+
+// GatewayClassIsUnmanaged is a watch predicate which filters out reconciliation events for
+// gateway objects which aren't annotated as unmanaged.
+func (r *GatewayClassReconciler) GatewayClassIsUnmanaged(obj client.Object) bool {
+	gatewayClass, ok := obj.(*gatewayv1beta1.GatewayClass)
+	if !ok {
+		r.Log.Error(fmt.Errorf("unexpected object type in gateway watch predicates"), "expected", "*gatewayv1beta1.GatewayClass", "found", reflect.TypeOf(obj))
+		return false
+	}
+
+	return isGatewayClassControlledAndUmanaged(gatewayClass)
+}
+
+// -----------------------------------------------------------------------------
+// GatewayClass Controller - Reconciliation
 // -----------------------------------------------------------------------------
 
 //+kubebuilder:rbac:groups=gateway.networking.k8s.io,resources=gatewayclasses,verbs=get;list;watch
@@ -79,7 +97,7 @@ func (r *GatewayClassReconciler) Reconcile(ctx context.Context, req ctrl.Request
 	}
 	log.V(util.DebugLevel).Info("processing gatewayclass", "name", req.Name)
 
-	if gwc.Spec.ControllerName == ControllerName {
+	if isGatewayClassControlledAndUmanaged(gwc) {
 		alreadyAccepted := false
 		for _, cond := range gwc.Status.Conditions {
 			if cond.Reason == string(gatewayv1beta1.GatewayClassConditionStatusAccepted) {
diff --git a/internal/dataplane/parser/translate_knative.go b/internal/dataplane/parser/translate_knative.go
index 16437cfdd..c41c517eb 100644
--- a/internal/dataplane/parser/translate_knative.go
+++ b/internal/dataplane/parser/translate_knative.go
@@ -22,7 +22,7 @@ func (p *Parser) ingressRulesFromKnativeIngress() ingressRules {
 	// well use the stock Kubernetes resource.
 	icp, err := getIngressClassParametersOrDefault(p.storer)
 	if err != nil {
-		if errors.Is(err, store.ErrNotFound{}) {
+		if errors.As(err, &store.ErrNotFound{}) {
 			// not found is expected if no IngressClass exists or IngressClassParameters isn't configured
 			p.logger.Debugf("could not find IngressClassParameters, using defaults: %s", err)
 		} else {
diff --git a/internal/manager/config.go b/internal/manager/config.go
index ceda286bd..55b79d248 100644
--- a/internal/manager/config.go
+++ b/internal/manager/config.go
@@ -15,6 +15,7 @@ import (
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/adminapi"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/admission"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/annotations"
+	"github.com/kong/kubernetes-ingress-controller/v2/internal/controllers/gateway"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/dataplane"
 )
 
@@ -54,14 +55,15 @@ type Config struct {
 	KongCustomEntitiesSecret string
 
 	// Kubernetes configurations
-	KubeconfigPath          string
-	IngressClassName        string
-	EnableLeaderElection    bool
-	LeaderElectionNamespace string
-	LeaderElectionID        string
-	Concurrency             int
-	FilterTags              []string
-	WatchNamespaces         []string
+	KubeconfigPath           string
+	IngressClassName         string
+	EnableLeaderElection     bool
+	LeaderElectionNamespace  string
+	LeaderElectionID         string
+	Concurrency              int
+	FilterTags               []string
+	WatchNamespaces          []string
+	GatewayAPIControllerName string
 
 	// Ingress status
 	PublishService       string
@@ -152,6 +154,7 @@ func (c *Config) FlagSet() *pflag.FlagSet {
 	flagSet.StringVar(&c.KongCustomEntitiesSecret, "kong-custom-entities-secret", "", `A Secret containing custom entities for DB-less mode, in "namespace/name" format`)
 
 	// Kubernetes configurations
+	flagSet.StringVar(&c.GatewayAPIControllerName, "gateway-api-controller-name", string(gateway.ControllerName), "The controller name to match on Gateway API resources.")
 	flagSet.StringVar(&c.KubeconfigPath, "kubeconfig", "", "Path to the kubeconfig file.")
 	flagSet.StringVar(&c.IngressClassName, "ingress-class", annotations.DefaultIngressClass, `Name of the ingress class to route through this controller.`)
 	flagSet.BoolVar(&c.EnableLeaderElection, "leader-elect", false, "DEPRECATED as of 2.1.0 leader election behavior is determined automatically and this flag has no effect")
diff --git a/internal/manager/run.go b/internal/manager/run.go
index 10bafdfa1..08f4ff8d3 100644
--- a/internal/manager/run.go
+++ b/internal/manager/run.go
@@ -6,6 +6,7 @@ import (
 	"errors"
 	"fmt"
 	"net/http"
+	"regexp"
 	"time"
 
 	"github.com/avast/retry-go/v4"
@@ -19,6 +20,7 @@ import (
 	gatewayv1alpha2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
 	gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
 
+	"github.com/kong/kubernetes-ingress-controller/v2/internal/controllers/gateway"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/dataplane"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/manager/metadata"
 	mgrutils "github.com/kong/kubernetes-ingress-controller/v2/internal/manager/utils"
@@ -189,6 +191,11 @@ func Run(ctx context.Context, c *Config, diagnostic util.ConfigDumpDiagnostic) e
 		}
 	}
 
+	if !isControllerNameValid(c.GatewayAPIControllerName) {
+		return errors.New("--gateway-api-controller-name is invalid. The expected format is example.com/controller-name")
+	}
+	gateway.ControllerName = gatewayv1beta1.GatewayController(c.GatewayAPIControllerName)
+
 	setupLog.Info("Starting Enabled Controllers")
 	controllers, err := setupControllers(mgr, dataplaneClient, dataplaneAddressFinder, kubernetesStatusQueue, c, featureGates)
 	if err != nil {
@@ -229,3 +236,9 @@ func Run(ctx context.Context, c *Config, diagnostic util.ConfigDumpDiagnostic) e
 	setupLog.Info("Starting manager")
 	return mgr.Start(ctx)
 }
+
+func isControllerNameValid(controllerName string) bool {
+	// https://github.com/kubernetes-sigs/gateway-api/blob/547122f7f55ac0464685552898c560658fb40073/apis/v1beta1/shared_types.go#L448-L463
+	re := regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*\/[A-Za-z0-9\/\-._~%!$&'()*+,;=:]+$`)
+	return re.Match([]byte(controllerName))
+}


Test patch:
diff --git a/internal/controllers/gateway/gateway_controller_test.go b/internal/controllers/gateway/gateway_controller_test.go
index 84814ef36..2b78ecac2 100644
--- a/internal/controllers/gateway/gateway_controller_test.go
+++ b/internal/controllers/gateway/gateway_controller_test.go
@@ -325,41 +325,76 @@ func Test_reconcileGatewaysIfClassMatches(t *testing.T) {
 }
 
 func Test_isGatewayControlledAndUnmanagedMode(t *testing.T) {
-	t.Log("generating a gatewayclass controlled by this controller implementation")
-	controlledGatewayClass := &gatewayv1beta1.GatewayClass{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: "us",
-		},
-		Spec: gatewayv1beta1.GatewayClassSpec{
-			ControllerName: ControllerName,
-		},
-	}
+	var testControllerName gatewayv1beta1.GatewayController = "acme.io/gateway-controller"
 
-	t.Log("generating a gatewayclass not controlled by this implementation")
-	uncontrolledGatewayClass := &gatewayv1beta1.GatewayClass{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: "eu",
+	testCases := []struct {
+		name           string
+		GatewayClass   *gatewayv1beta1.GatewayClass
+		expectedResult bool
+	}{
+		{
+			name: "uncontrolled managed GatewayClass",
+			GatewayClass: &gatewayv1beta1.GatewayClass{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "uncontrolled-managed",
+				},
+				Spec: gatewayv1beta1.GatewayClassSpec{
+					ControllerName: testControllerName,
+				},
+			},
+			expectedResult: false,
 		},
-		Spec: gatewayv1beta1.GatewayClassSpec{
-			ControllerName: "acme.io/gateway-controller",
+		{
+			name: "uncontrolled unmanaged GatewayClass",
+			GatewayClass: &gatewayv1beta1.GatewayClass{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "uncontrolled-unmanaged",
+					Annotations: map[string]string{
+						annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+					},
+				},
+				Spec: gatewayv1beta1.GatewayClassSpec{
+					ControllerName: testControllerName,
+				},
+			},
+			expectedResult: false,
 		},
-	}
-
-	t.Log("creating an unmanaged mode enabled gateway")
-	unmanagedGateway := gatewayv1beta1.Gateway{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: "test",
-			Annotations: map[string]string{
-				annotations.AnnotationPrefix + annotations.GatewayUnmanagedAnnotation: "true",
+		{
+			name: "controlled managed GatewayClass",
+			GatewayClass: &gatewayv1beta1.GatewayClass{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "controlled-managed",
+				},
+				Spec: gatewayv1beta1.GatewayClassSpec{
+					ControllerName: ControllerName,
+				},
+			},
+			expectedResult: false,
+		},
+		{
+			name: "controlled unmanaged GatewayClass",
+			GatewayClass: &gatewayv1beta1.GatewayClass{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "controlled-unmanaged",
+					Annotations: map[string]string{
+						annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+					},
+				},
+				Spec: gatewayv1beta1.GatewayClassSpec{
+					ControllerName: ControllerName,
+				},
 			},
+			expectedResult: true,
 		},
 	}
 
-	t.Log("verifying the results for several gateways")
-	assert.False(t, isGatewayInClassAndUnmanaged(controlledGatewayClass, gatewayv1beta1.Gateway{}))
-	assert.False(t, isGatewayInClassAndUnmanaged(uncontrolledGatewayClass, gatewayv1beta1.Gateway{}))
-	assert.False(t, isGatewayInClassAndUnmanaged(uncontrolledGatewayClass, unmanagedGateway))
-	assert.True(t, isGatewayInClassAndUnmanaged(controlledGatewayClass, unmanagedGateway))
+	for _, tc := range testCases {
+		tc := tc
+		t.Run(tc.name, func(t *testing.T) {
+			t.Parallel()
+			assert.Equal(t, tc.expectedResult, isGatewayClassControlledAndUmanaged(tc.GatewayClass))
+		})
+	}
 }
 
 func Test_areAllowedRoutesConsistentByProtocol(t *testing.T) {
diff --git a/test/conformance/gateway_conformance_test.go b/test/conformance/gateway_conformance_test.go
index 6fa6458b1..99448f047 100644
--- a/test/conformance/gateway_conformance_test.go
+++ b/test/conformance/gateway_conformance_test.go
@@ -16,6 +16,7 @@ import (
 	"sigs.k8s.io/gateway-api/conformance/tests"
 	"sigs.k8s.io/gateway-api/conformance/utils/suite"
 
+	"github.com/kong/kubernetes-ingress-controller/v2/internal/annotations"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/controllers/gateway"
 	"github.com/kong/kubernetes-ingress-controller/v2/test/consts"
 )
@@ -40,6 +41,9 @@ func TestGatewayConformance(t *testing.T) {
 	gwc := &gatewayv1beta1.GatewayClass{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: uuid.NewString(),
+			Annotations: map[string]string{
+				annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+			},
 		},
 		Spec: gatewayv1beta1.GatewayClassSpec{
 			ControllerName: gateway.ControllerName,
diff --git a/test/e2e/all_in_one_test.go b/test/e2e/all_in_one_test.go
index 1ed16a2bd..73094fc04 100644
--- a/test/e2e/all_in_one_test.go
+++ b/test/e2e/all_in_one_test.go
@@ -92,6 +92,17 @@ func TestDeployAllInOneDBLESS(t *testing.T) {
 		assert.NoError(t, env.Cleanup(ctx))
 	}()
 
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
+	defer func() {
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
+	}()
+
 	t.Log("deploying kong components")
 	manifest, err := getTestManifest(t, dblessPath)
 	require.NoError(t, err)
@@ -148,6 +159,17 @@ func TestDeployAndUpgradeAllInOneDBLESS(t *testing.T) {
 		assert.NoError(t, env.Cleanup(ctx))
 	}()
 
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
+	defer func() {
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
+	}()
+
 	t.Logf("deploying previous version %s kong manifest", preTag)
 	deployKong(ctx, t, env, oldManifest.Body)
 
@@ -187,8 +209,15 @@ func TestDeployAllInOneEnterpriseDBLESS(t *testing.T) {
 
 	createKongImagePullSecret(ctx, t, env)
 
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("generating a superuser password")
@@ -233,8 +262,16 @@ func TestDeployAllInOnePostgres(t *testing.T) {
 		environments.NewBuilder().WithAddons(addons...), clusterVersionStr)
 	env, err := builder.Build(ctx)
 	require.NoError(t, err)
+
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("deploying kong components")
@@ -270,7 +307,12 @@ func TestDeployAllInOnePostgresWithMultipleReplicas(t *testing.T) {
 	t.Logf("build a cleaner to dump diagnostics...")
 	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("deploying kong components")
@@ -419,8 +461,15 @@ func TestDeployAllInOneEnterprisePostgres(t *testing.T) {
 	require.NoError(t, err)
 	createKongImagePullSecret(ctx, t, env)
 
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("generating a superuser password")
diff --git a/test/e2e/features_test.go b/test/e2e/features_test.go
index c7e850c64..e2a25a514 100644
--- a/test/e2e/features_test.go
+++ b/test/e2e/features_test.go
@@ -161,14 +161,16 @@ func TestWebhookUpdate(t *testing.T) {
 	builder := environments.NewBuilder().WithExistingCluster(cluster).WithAddons(addons...)
 	env, err := builder.Build(ctx)
 	require.NoError(t, err)
-	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
-	}()
 
 	t.Logf("build a cleaner to dump diagnostics...")
 	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("deploying kong components")
@@ -310,9 +312,15 @@ func TestDeployAllInOneDBLESSGateway(t *testing.T) {
 	env, err := builder.Build(ctx)
 	require.NoError(t, err)
 
+	t.Logf("build a cleaner to dump diagnostics...")
+	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		t.Logf("cleaning up environment for cluster %s", env.Cluster().Name())
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("deploying kong components")
@@ -490,8 +498,14 @@ func TestDeployAllInOneDBLESSNoLoadBalancer(t *testing.T) {
 		environments.NewBuilder().WithAddons(addons...), clusterVersionStr)
 	env, err := builder.Build(ctx)
 	require.NoError(t, err)
+	cleaner := clusters.NewCleaner(env.Cluster())
 	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
+		if t.Failed() {
+			output, err := cleaner.DumpDiagnostics(ctx, t.Name())
+			t.Logf("%s failed, dumped diagnostics to %s", t.Name(), output)
+			assert.NoError(t, err)
+		}
+		assert.NoError(t, cleaner.Cleanup(ctx))
 	}()
 
 	t.Log("deploying kong components")
@@ -570,9 +584,6 @@ func TestDefaultIngressClass(t *testing.T) {
 	builder := environments.NewBuilder().WithExistingCluster(cluster).WithAddons(addons...)
 	env, err := builder.Build(ctx)
 	require.NoError(t, err)
-	defer func() {
-		assert.NoError(t, env.Cleanup(ctx))
-	}()
 
 	t.Logf("build a cleaner to dump diagnostics...")
 	cleaner := clusters.NewCleaner(env.Cluster())
diff --git a/test/e2e/helpers_gateway_test.go b/test/e2e/helpers_gateway_test.go
index 38f50d2a9..5b7740656 100644
--- a/test/e2e/helpers_gateway_test.go
+++ b/test/e2e/helpers_gateway_test.go
@@ -40,6 +40,10 @@ func deployGateway(ctx context.Context, t *testing.T, env environments.Environme
 	supportedGatewayClass := &gatewayv1beta1.GatewayClass{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: uuid.NewString(),
+			Annotations: map[string]string{
+				// annotate the gatewayclass to unmanaged.
+				annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+			},
 		},
 		Spec: gatewayv1beta1.GatewayClassSpec{
 			ControllerName: gateway.ControllerName,
@@ -52,9 +56,6 @@ func deployGateway(ctx context.Context, t *testing.T, env environments.Environme
 	gw := &gatewayv1beta1.Gateway{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: "kong",
-			Annotations: map[string]string{
-				annotations.AnnotationPrefix + annotations.GatewayUnmanagedAnnotation: "true", // trigger the unmanaged gateway mode
-			},
 		},
 		Spec: gatewayv1beta1.GatewaySpec{
 			GatewayClassName: gatewayv1beta1.ObjectName(supportedGatewayClass.Name),
@@ -84,6 +85,7 @@ func verifyGateway(ctx context.Context, t *testing.T, env environments.Environme
 				return true
 			}
 		}
+		t.Logf("conditions: %v", gw.Status.Conditions)
 		return false
 	}, gatewayUpdateWaitTime, time.Second)
 }
@@ -97,6 +99,10 @@ func deployGatewayWithTCPListener(ctx context.Context, t *testing.T, env environ
 	supportedGatewayClass := &gatewayv1beta1.GatewayClass{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: uuid.NewString(),
+			Annotations: map[string]string{
+				// annotate the gatewayclass to unmanaged.
+				annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+			},
 		},
 		Spec: gatewayv1beta1.GatewayClassSpec{
 			ControllerName: gateway.ControllerName,
diff --git a/test/e2e/helpers_test.go b/test/e2e/helpers_test.go
index fa58d8d0c..004dcd29c 100644
--- a/test/e2e/helpers_test.go
+++ b/test/e2e/helpers_test.go
@@ -218,7 +218,12 @@ func verifyEnterprise(ctx context.Context, t *testing.T, env environments.Enviro
 		}
 		return adminOutput.Version != ""
 	}, adminAPIWait, time.Second)
-	require.Contains(t, adminOutput.Version, "enterprise-edition")
+	if string(adminOutput.Version[0]) == "3" {
+		// 3.x removed the "-enterprise-edition" string but provided no other indication that something is enterprise
+		require.Len(t, strings.Split(adminOutput.Version, "."), 4)
+	} else {
+		require.Contains(t, adminOutput.Version, "enterprise-edition")
+	}
 }
 
 func verifyEnterpriseWithPostgres(ctx context.Context, t *testing.T, env environments.Environment, adminPassword string) {
@@ -275,8 +280,12 @@ func killKong(ctx context.Context, t *testing.T, env environments.Environment, p
 	written, err := kubeconfigFile.Write(kubeconfig)
 	require.NoError(t, err)
 	require.Equal(t, len(kubeconfig), written)
-	cmd := exec.Command("kubectl", "--kubeconfig", kubeconfigFile.Name(), "exec", "-n", pod.Namespace, pod.Name, "--", "kill", "1") //nolint:gosec
-	require.NoError(t, cmd.Run())
+	cmd := exec.Command("kubectl", "--kubeconfig", kubeconfigFile.Name(), "exec", "-n", pod.Namespace, pod.Name, "--", "bash", "-c", "kill 1") //nolint:gosec
+	stdout, stderr := new(bytes.Buffer), new(bytes.Buffer)
+	cmd.Stdout = stdout
+	cmd.Stderr = stderr
+	err = cmd.Run()
+	require.NoErrorf(t, err, "kill failed: STDOUT(%s) STDERR(%s)", stdout.String(), stderr.String())
 	require.Eventually(t, func() bool {
 		pod, err = env.Cluster().Client().CoreV1().Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{})
 		require.NoError(t, err)
diff --git a/test/e2e/istio_test.go b/test/e2e/istio_test.go
index c13b32acd..bfcca7409 100644
--- a/test/e2e/istio_test.go
+++ b/test/e2e/istio_test.go
@@ -55,6 +55,12 @@ var (
 // See: https://docs.konghq.com/kubernetes-ingress-controller/latest/references/version-compatibility/#istio
 func TestIstioWithKongIngressGateway(t *testing.T) {
 	t.Parallel()
+	// Istio's test is unique in that it operates like the integration tests, and runs the controller manager from the
+	// test, whereas most E2E tests deploy it to the cluster normally. The upshot of this is that the Istio test
+	// pollutes E2E logs with a bunch of controller log nonsense and a boatload of goroutines that litter the panic
+	// logs. Temporarily skip it because it's not failing and it's making it harder to read the failure results.
+	// Ultimately we should probably move it elsewhere.
+
 	ctx, cancel := context.WithCancel(context.Background())
 	defer cancel()
 
diff --git a/test/integration/gateway_test.go b/test/integration/gateway_test.go
index 411edfa53..f14b83e0e 100644
--- a/test/integration/gateway_test.go
+++ b/test/integration/gateway_test.go
@@ -32,8 +32,6 @@ const (
 	// gatewayUpdateWaitTime is the amount of time to wait for updates to the Gateway, or to its
 	// parent Service to fully resolve into ready state.
 	gatewayUpdateWaitTime = time.Minute * 3
-
-	unmanagedAnnotation = annotations.AnnotationPrefix + annotations.GatewayUnmanagedAnnotation
 )
 
 func TestUnmanagedGatewayBasics(t *testing.T) {
@@ -49,7 +47,7 @@ func TestUnmanagedGatewayBasics(t *testing.T) {
 	require.NoError(t, err)
 
 	t.Log("deploying a new gateway")
-	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, managedGatewayClassName)
+	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, unmanagedGatewayClassName)
 	require.NoError(t, err)
 	cleaner.Add(gateway)
 	err = gatewayHealthCheck(ctx, gatewayClient, gateway.Name, ns.Name)
@@ -59,7 +57,7 @@ func TestUnmanagedGatewayBasics(t *testing.T) {
 	require.Eventually(t, func() bool {
 		gw, err = gatewayClient.GatewayV1beta1().Gateways(ns.Name).Get(ctx, defaultGatewayName, metav1.GetOptions{})
 		require.NoError(t, err)
-		return gw.Annotations[unmanagedAnnotation] == "kong-system/ingress-controller-kong-proxy"
+		return gw.Annotations[annotations.GatewayClassUnmanagedAnnotation] == "kong-system/ingress-controller-kong-proxy"
 	}, gatewayUpdateWaitTime, time.Second)
 
 	t.Log("verifying that the gateway address is populated from the publish service")
@@ -136,8 +134,8 @@ func TestGatewayListenerConflicts(t *testing.T) {
 	gatewayClient, err := gatewayclient.NewForConfig(env.Cluster().Config())
 	require.NoError(t, err)
 
-	t.Log("deploying a new gateway using the default gateway")
-	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, managedGatewayClassName)
+	t.Log("deploying a new Gateway using the default GatewayClass")
+	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, unmanagedGatewayClassName)
 	require.NoError(t, err)
 	cleaner.Add(gateway)
 	err = gatewayHealthCheck(ctx, gatewayClient, gateway.Name, ns.Name)
@@ -362,7 +360,7 @@ func TestUnmanagedGatewayControllerSupport(t *testing.T) {
 
 	t.Log("deploying an unsupported gatewayclass to the test cluster")
 	unsupportedGatewayClass, err := DeployGatewayClass(ctx, gatewayClient, uuid.NewString(), func(gc *gatewayv1beta1.GatewayClass) {
-		gc.Spec.ControllerName = unmanagedControllerName
+		gc.Spec.ControllerName = unsupportedControllerName
 	})
 	require.NoError(t, err)
 	cleaner.Add(unsupportedGatewayClass)
@@ -426,6 +424,56 @@ func TestUnmanagedGatewayClass(t *testing.T) {
 	}, gatewayUpdateWaitTime, time.Second)
 }
 
+func TestManagedGatewayClass(t *testing.T) {
+	ns, cleaner := setup(t)
+	defer func() { assert.NoError(t, cleaner.Cleanup(ctx)) }()
+
+	t.Log("generating a gateway kubernetes client")
+	gatewayClient, err := gatewayclient.NewForConfig(env.Cluster().Config())
+	require.NoError(t, err)
+
+	t.Log("deploying a gateway to the test cluster, but with no valid gatewayclass yet")
+	gatewayClassName := uuid.NewString()
+	gatewayName := uuid.NewString()
+	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, gatewayClassName, func(gw *gatewayv1beta1.Gateway) {
+		gw.Name = gatewayName
+	})
+	require.NoError(t, err)
+	cleaner.Add(gateway)
+
+	t.Log("verifying that the Gateway object does not get scheduled by the controller due to missing its GatewayClass")
+	timeout := time.Now().Add(gatewayWaitTimeToVerifyScheduling)
+	for timeout.After(time.Now()) {
+		gateway, err = gatewayClient.GatewayV1beta1().Gateways(ns.Name).Get(ctx, gateway.Name, metav1.GetOptions{})
+		require.NoError(t, err)
+		require.Len(t, gateway.Status.Conditions, 1)
+		require.Equal(t, string(gatewayv1beta1.GatewayReasonNotReconciled), gateway.Status.Conditions[0].Reason)
+	}
+
+	t.Log("deploying a missing managed gatewayclass to the test cluster")
+	gwc, err := DeployGatewayClass(ctx, gatewayClient, gatewayClassName, func(gc *gatewayv1beta1.GatewayClass) {
+		gc.Annotations = nil
+	})
+	require.NoError(t, err)
+	cleaner.Add(gwc)
+
+	finished := make(chan struct{})
+
+	// Let's wait for one minute and check that the Gateway hasn't reconciled by the operator. It should never get ready.
+	t.Log("the Gateway must not be reconciled as it is using a managed GatewayClass")
+	time.AfterFunc(time.Minute, func() {
+		defer close(finished)
+		gateway, err = gatewayClient.GatewayV1beta1().Gateways(ns.Name).Get(ctx, gateway.Name, metav1.GetOptions{})
+		require.NoError(t, err)
+		for _, cond := range gateway.Status.Conditions {
+			if cond.Type == string(gatewayv1beta1.GatewayConditionReady) {
+				require.Equal(t, cond.Status, metav1.ConditionFalse)
+			}
+		}
+	})
+	<-finished
+}
+
 func TestGatewayFilters(t *testing.T) {
 	ns, cleaner := setup(t)
 	defer func() { assert.NoError(t, cleaner.Cleanup(ctx)) }()
@@ -443,7 +491,7 @@ func TestGatewayFilters(t *testing.T) {
 	t.Log("deploying a gateway that allows routes in all namespaces")
 	gatewayName := uuid.NewString()
 	fromAll := gatewayv1beta1.NamespacesFromAll
-	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, managedGatewayClassName, func(gw *gatewayv1beta1.Gateway) {
+	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, unmanagedGatewayClassName, func(gw *gatewayv1beta1.Gateway) {
 		gw.Name = gatewayName
 		gw.Spec.Listeners = []gatewayv1beta1.Listener{
 			{
diff --git a/test/integration/gateway_webhook_test.go b/test/integration/gateway_webhook_test.go
index 58daa2732..60737d2e9 100644
--- a/test/integration/gateway_webhook_test.go
+++ b/test/integration/gateway_webhook_test.go
@@ -70,11 +70,11 @@ func TestGatewayValidationWebhook(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name: uuid.NewString(),
 					Annotations: map[string]string{
-						annotations.AnnotationPrefix + annotations.GatewayUnmanagedAnnotation: "true",
+						annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
 					},
 				},
 				Spec: gatewayv1beta1.GatewaySpec{
-					GatewayClassName: gatewayv1beta1.ObjectName(managedGatewayClassName),
+					GatewayClassName: gatewayv1beta1.ObjectName(unmanagedGatewayClassName),
 					Listeners: []gatewayv1beta1.Listener{{
 						Name:     "http",
 						Protocol: gatewayv1beta1.HTTPProtocolType,
diff --git a/test/integration/helpers_test.go b/test/integration/helpers_test.go
index 81b0dea17..50f0fa936 100644
--- a/test/integration/helpers_test.go
+++ b/test/integration/helpers_test.go
@@ -20,17 +20,18 @@ import (
 	gatewayv1beta1 "sigs.k8s.io/gateway-api/apis/v1beta1"
 	gatewayclient "sigs.k8s.io/gateway-api/pkg/client/clientset/versioned"
 
+	"github.com/kong/kubernetes-ingress-controller/v2/internal/annotations"
 	"github.com/kong/kubernetes-ingress-controller/v2/internal/controllers/gateway"
 )
 
 const (
 	// defaultGatewayName is the default name for the Gateways created during tests.
 	defaultGatewayName = "kong"
-	// managedGatewayClassName is the name of the default GatewayClass created during the test environment setup.
-	managedGatewayClassName = "kong-managed"
-	// unmanagedControllerName is the name of the controller used for those gateways that are not supported
+	// unmanagedGatewayClassName is the name of the default GatewayClass created during the test environment setup.
+	unmanagedGatewayClassName = "kong-unmanaged"
+	// unsupportedControllerName is the name of the controller used for those gateways that are not supported
 	// by an actual controller (i.e., they won't be scheduled).
-	unmanagedControllerName gatewayv1beta1.GatewayController = "example.com/unmanaged-gateway-controller"
+	unsupportedControllerName gatewayv1beta1.GatewayController = "example.com/unsupported-gateway-controller"
 )
 
 // DeployGateway creates a default gatewayClass, accepts a variadic set of options,
@@ -39,6 +40,9 @@ func DeployGatewayClass(ctx context.Context, client *gatewayclient.Clientset, ga
 	gwc := &gatewayv1beta1.GatewayClass{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: gatewayClassName,
+			Annotations: map[string]string{
+				annotations.GatewayClassUnmanagedAnnotation: annotations.GatewayClassUnmanagedAnnotationValuePlaceholder,
+			},
 		},
 		Spec: gatewayv1beta1.GatewayClassSpec{
 			ControllerName: gateway.ControllerName,
@@ -67,9 +71,6 @@ func DeployGateway(ctx context.Context, client *gatewayclient.Clientset, namespa
 	gw := &gatewayv1beta1.Gateway{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: defaultGatewayName,
-			Annotations: map[string]string{
-				unmanagedAnnotation: "true", // trigger the unmanaged gateway mode
-			},
 		},
 		Spec: gatewayv1beta1.GatewaySpec{
 			GatewayClassName: gatewayv1beta1.ObjectName(gatewayClassName),
diff --git a/test/integration/httproute_webhook_test.go b/test/integration/httproute_webhook_test.go
index b82f92b37..b9dcc0da2 100644
--- a/test/integration/httproute_webhook_test.go
+++ b/test/integration/httproute_webhook_test.go
@@ -49,7 +49,7 @@ func TestHTTPRouteValidationWebhook(t *testing.T) {
 	require.NoError(t, err)
 
 	t.Log("creating a managed gateway")
-	managedGateway, err := DeployGateway(ctx, gatewayClient, ns.Name, managedGatewayClassName, func(g *gatewayv1beta1.Gateway) {
+	managedGateway, err := DeployGateway(ctx, gatewayClient, ns.Name, unmanagedGatewayClassName, func(g *gatewayv1beta1.Gateway) {
 		g.Name = uuid.NewString()
 	})
 	require.NoError(t, err)
@@ -57,7 +57,7 @@ func TestHTTPRouteValidationWebhook(t *testing.T) {
 
 	t.Log("creating an unmanaged gatewayclass")
 	unmanagedGatewayClass, err := DeployGatewayClass(ctx, gatewayClient, uuid.NewString(), func(gc *gatewayv1beta1.GatewayClass) {
-		gc.Spec.ControllerName = unmanagedControllerName
+		gc.Spec.ControllerName = unsupportedControllerName
 	})
 	require.NoError(t, err)
 	cleaner.Add(unmanagedGatewayClass)
diff --git a/test/integration/suite_test.go b/test/integration/suite_test.go
index f805b72eb..c4a1b2da6 100644
--- a/test/integration/suite_test.go
+++ b/test/integration/suite_test.go
@@ -224,7 +224,7 @@ func TestMain(m *testing.M) {
 	exitOnErr(err)
 
 	fmt.Println("INFO: Deploying the default GatewayClass")
-	gwc, err := DeployGatewayClass(ctx, gatewayClient, managedGatewayClassName)
+	gwc, err := DeployGatewayClass(ctx, gatewayClient, unmanagedGatewayClassName)
 	exitOnErr(err)
 	cleaner.Add(gwc)
 
diff --git a/test/integration/tlsroute_test.go b/test/integration/tlsroute_test.go
index eb53bc757..882fb08ac 100644
--- a/test/integration/tlsroute_test.go
+++ b/test/integration/tlsroute_test.go
@@ -459,7 +459,7 @@ func TestTLSRouteReferenceGrant(t *testing.T) {
 	cleaner.Add(secret2)
 
 	t.Log("deploying a gateway to the test cluster using unmanaged gateway mode")
-	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, managedGatewayClassName, func(gw *gatewayv1beta1.Gateway) {
+	gateway, err := DeployGateway(ctx, gatewayClient, ns.Name, unmanagedGatewayClassName, func(gw *gatewayv1beta1.Gateway) {
 		hostname := gatewayv1beta1.Hostname(tlsRouteHostname)
 		otherHostname := gatewayv1beta1.Hostname(tlsRouteExtraHostname)
 		otherNamespace := gatewayv1beta1.Namespace(otherNs.Name)