Repository: Kong/kubernetes-ingress-controller
Pull request #5952 (2024-04-29 10:37:10)

feat(httproute): handle URLRewrite filter's hostname rewrite
**What this PR does / why we need it**:

Extends support for `URLRewrite` filter with the `Hostname` field. Enables Gateway API conformance tests covering this feature.

**Which issue this PR fixes**:

Fixes https://github.com/Kong/kubernetes-ingress-controller/issues/3685.

**Special notes for your reviewer**:

<!-- Here you can add any open questions or notes that you might have for reviewers -->

**PR Readiness Checklist**:

Complete these before marking the PR as `ready to review`:

- [x] the `CHANGELOG.md` release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong) Report
Attention: Patch coverage is `85.18519%` with `8 lines` in your changes are missing coverage. Please review.
> Project coverage is 68.3%. Comparing base [(`641c73c`)](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/commit/641c73c3897db358ac6c0693003b985e2d4e3eed?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong) to head [(`a4fcb91`)](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong).

> :exclamation: Current head a4fcb91 differs from pull request most recent head 6f0ac96. Consider uploading reports for the commit 6f0ac96 to get more accurate results

| [Files](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong) | Patch % | Lines |
|---|---|---|
| [...al/dataplane/translator/subtranslator/httproute.go](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?src=pr&el=tree&filepath=internal%2Fdataplane%2Ftranslator%2Fsubtranslator%2Fhttproute.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong#diff-aW50ZXJuYWwvZGF0YXBsYW5lL3RyYW5zbGF0b3Ivc3VidHJhbnNsYXRvci9odHRwcm91dGUuZ28=) | 85.1% | [5 Missing and 3 partials :warning: ](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong) |

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##            main   #5952     +/-   ##
=======================================
- Coverage   72.2%   68.3%   -4.0%     
=======================================
  Files        180     180             
  Lines      18425   18453     +28     
=======================================
- Hits       13314   12610    -704     
- Misses      4117    4859    +742     
+ Partials     994     984     -10     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/Kong/kubernetes-ingress-controller/pull/5952?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Kong).

czeslavo: > I'm a bit surprised this does apparently work with conformance as a pseudo-integration test. AFAIK using the request transformer to modify Host usually doesn't quite work as expected. There's no link between the header value and Kong's notion of what the actual upstream host is, and the "proper" host is used for other purposes beyond setting the upstream host header.
>
>Can you run a manual attempt to configure this on an HTTPS HTTPRoute? Does it behave oddly? I expect that the test isn't being run over HTTPS (not sure, but didn't see otherwise on a brief review of upstream code), and that we'll actually hit an issue here because the plugin doesn't change TLS SNI.

In KIC [we only support HTTPRoutes with Gateways configured with `TLSModeTerminate`](https://github.com/Kong/kubernetes-ingress-controller/blob/bd686c703031c1745d98d3848c9f515d703e0ae4/internal/controllers/gateway/route_utils.go#L386). That's the only mode Gateway API [documents as supported](https://gateway-api.sigs.k8s.io/guides/tls/#clientserver-and-tls). By saying HTTPS HTTPRoute - what do you mean exactly?

>Upstream docs don't really touch on SNI handling, but in general you shouldn't ever send requests where the TLS SNI and HTTP Host header don't match, which is what I think this will do.

I've verified that if we modify the `Host` header using the plugin as in this implementation, HTTPs Services are called by Kong with a properly modified SNI server name, matching the `Host` header. I understand that's handled by Kong's HTTP client automatically based on the prepared request.

Please note `connection.servername` matches the modified `host` header - `connection.servername` is how `mendhak/docker-http-https-echo` indicates SNI server name: https://github.com/mendhak/docker-http-https-echo/pull/2/files):
```terminal
$ http localhost:8000/host-rewrite
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 815
Content-Type: application/json; charset=utf-8
Date: Wed, 08 May 2024 13:04:30 GMT
ETag: W/"32f-3CuRJNRNzt+PJ42KBRr9SMbAfrA"
Via: kong/3.6.0
X-Kong-Proxy-Latency: 1
X-Kong-Request-Id: f24af438844d9e51f9db306ba2e98f49
X-Kong-Upstream-Latency: 47
X-Powered-By: Express

{
    "body": "",
    "clientCertificate": {},
    "connection": {
        "servername": "modified.host"
    },
    "fresh": false,
    "headers": {
        "accept": "*/*",
        "accept-encoding": "gzip, deflate",
        "connection": "keep-alive",
        "host": "modified.host",
        "user-agent": "HTTPie/3.2.2",
        "x-forwarded-for": "127.0.0.1",
        "x-forwarded-host": "localhost",
        "x-forwarded-path": "/host-rewrite",
        "x-forwarded-port": "80",
        "x-forwarded-proto": "http",
        "x-kong-request-id": "f24af438844d9e51f9db306ba2e98f49",
        "x-real-ip": "127.0.0.1"
    },
    "hostname": "localhost",
    "ip": "127.0.0.1",
    "ips": [
        "127.0.0.1"
    ],
    "method": "GET",
    "os": {
        "hostname": "echo-59875b796d-mppfc"
    },
    "path": "/host-rewrite",
    "protocol": "http",
    "query": {},
    "subdomains": [],
    "xhr": false
}
```

Manifests:
```yaml
apiVersion: v1
kind: Service
metadata:
  name: echo
  annotations:
    konghq.com/protocol: https
spec:
  ports:
    - protocol: TCP
      name: https
      port: 443
      targetPort: https
  selector:
    app: echo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: echo
  name: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: mendhak/http-https-echo
          ports:
            - containerPort: 8443
              name: https
          resources:
            requests:
              cpu: 10m
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: kong
  annotations:
    konghq.com/gatewayclass-unmanaged: "true"
spec:
  controllerName: konghq.com/kic-gateway-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kong
spec:
  gatewayClassName: kong
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: rewrite-host
spec:
  parentRefs:
    - name: kong
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /host-rewrite
      filters:
        - type: URLRewrite
          urlRewrite:
            hostname: "modified.host"
      backendRefs:
        - name: echo
          port: 443
```





Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2eb57c0bd..2a840f915 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -119,17 +119,26 @@ Adding a new version? You'll need three changes:
 - Add support in `HTTPRoute`s for `URLRewrite`:
   - `FullPathRewrite` [#5855](https://github.com/Kong/kubernetes-ingress-controller/pull/5855)
   - `ReplacePrefixMatch` for both router modes:
-    - `traditional_compatible`: [#5895](https://github.com/Kong/kubernetes-ingress-controller/pull/5895)
-    - `expressions`: [#5940](https://github.com/Kong/kubernetes-ingress-controller/pull/5940)
+    - `traditional_compatible` [#5895](https://github.com/Kong/kubernetes-ingress-controller/pull/5895)
+    - `expressions` [#5940](https://github.com/Kong/kubernetes-ingress-controller/pull/5940)
+  - `Hostname` [#5952](https://github.com/Kong/kubernetes-ingress-controller/pull/5952)
 - DB mode now supports Event reporting for resources that failed to apply.
   [#5785](https://github.com/Kong/kubernetes-ingress-controller/pull/5785)
-- Improve validation - reject `Ingresses`, `Services` or `KongConsumers` that have multiple instances
-  of the same type plugin attached.
+- Improve validation - reject `Ingresses`, `Services`, `HTTPRoutes`, `KongConsumers` or `KongConsumerGroups`
+  that have multiple instances of the same type plugin attached.
   [#5972](https://github.com/Kong/kubernetes-ingress-controller/pull/5972)
+  [#5979](https://github.com/Kong/kubernetes-ingress-controller/pull/5979)
 - Added support for `konghq.com/headers-separator` that allows to set custom separator (instead of default `,`)
   for headers specified with `konghq.com/headers.*` annotations. Moreover parsing a content of `konghq.com/headers.*`
   is more robust - leading and trailing whitespace characters are discarded.
   [#5977](https://github.com/Kong/kubernetes-ingress-controller/pull/5977)
+- Added the CRD `KongCustomEntity` to support custom Kong entities that are not
+  defined in KIC yet.
+  [#5982](https://github.com/Kong/kubernetes-ingress-controller/pull/5982)
+- Added `FallbackConfiguration` feature gate to enable the controller to generate a fallback configuration
+  for Kong when it fails to apply the configuration. The feature gate is disabled by default.
+  [#5993](https://github.com/Kong/kubernetes-ingress-controller/pull/5993)
+
 
 ### Fixed
 
@@ -148,6 +157,8 @@ Adding a new version? You'll need three changes:
   `parentRefs`'s group/kind is not `gateway.network.k8s.io/Gateway`, the item
   is seen as a parent other than the controller and ignored in parentRef check.
   [#5919](https://github.com/Kong/kubernetes-ingress-controller/pull/5919)
+- Redacted values no longer cause collisions in configuration reported to Konnect.
+  [5964](https://github.com/Kong/kubernetes-ingress-controller/pull/5964)
 
 ### Changed
 
diff --git a/Makefile b/Makefile
index b5c731608..120b27ac1 100644
--- a/Makefile
+++ b/Makefile
@@ -289,7 +289,7 @@ manifests.single: kustomize ## Compose single-file deployment manifests from bui
 # ------------------------------------------------------------------------------
 
 .PHONY: generate
-generate: generate.controllers generate.clientsets generate.gateway-api-consts generate.docs fmt
+generate: generate.controllers generate.clientsets generate.gateway-api-consts generate.docs generate.store fmt
 
 .PHONY: generate.controllers
 generate.controllers: controller-gen
@@ -328,6 +328,10 @@ generate.apidocs: crd-ref-docs
 generate.cli-arguments-docs:
 	go run ./scripts/cli-arguments-docs-gen/main.go > ./docs/cli-arguments.md
 
+.PHONY: generate.store
+generate.store:
+	go generate ./internal/store
+
 # ------------------------------------------------------------------------------
 # Build - Container Images
 # ------------------------------------------------------------------------------
diff --git a/config/crd/bases/configuration.konghq.com_kongcustomentities.yaml b/config/crd/bases/configuration.konghq.com_kongcustomentities.yaml
new file mode 100644
index 000000000..91c6da9b5
--- /dev/null
+++ b/config/crd/bases/configuration.konghq.com_kongcustomentities.yaml
@@ -0,0 +1,197 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.15.0
+  name: kongcustomentities.configuration.konghq.com
+spec:
+  group: configuration.konghq.com
+  names:
+    categories:
+    - kong-ingress-controller
+    kind: KongCustomEntity
+    listKind: KongCustomEntityList
+    plural: kongcustomentities
+    shortNames:
+    - kce
+    singular: kongcustomentity
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - description: type of the Kong entity
+      jsonPath: .spec.type
+      name: Entity Type
+      type: string
+    - description: Age
+      jsonPath: .metadata.creationTimestamp
+      name: Age
+      type: date
+    - jsonPath: .status.conditions[?(@.type=="Programmed")].status
+      name: Programmed
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        description: KongCustomEntity defines a "custom" Kong entity that KIC cannot
+          support the entity type directly.
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              controllerName:
+                description: ControllerName specifies the controller that should reconcile
+                  it, like ingress class.
+                type: string
+              fields:
+                description: Fields defines the fields of the Kong entity itself.
+                x-kubernetes-preserve-unknown-fields: true
+              parentRef:
+                description: |-
+                  ParentRef references the kubernetes resource it attached to when its scope is "attached".
+                  Currently only KongPlugin/KongClusterPlugin allowed. This will make the custom entity to be attached
+                  to the entity(service/route/consumer) where the plugin is attached.
+                properties:
+                  group:
+                    type: string
+                  kind:
+                    type: string
+                  name:
+                    type: string
+                  namespace:
+                    description: Empty namespace means the same namespace of the owning
+                      object.
+                    type: string
+                required:
+                - name
+                type: object
+              type:
+                description: EntityType is the type of the Kong entity. The type is
+                  used in generating declarative configuration.
+                type: string
+            required:
+            - controllerName
+            - fields
+            - type
+            type: object
+          status:
+            description: Status stores the reconciling status of the resource.
+            properties:
+              conditions:
+                default:
+                - lastTransitionTime: "1970-01-01T00:00:00Z"
+                  message: Waiting for controller
+                  reason: Pending
+                  status: Unknown
+                  type: Programmed
+                description: |-
+                  Conditions describe the current conditions of the KongCustomEntityStatus.
+
+
+                  Known condition types are:
+
+
+                  * "Programmed"
+                items:
+                  description: "Condition contains details for one aspect of the current
+                    state of this API Resource.\n---\nThis struct is intended for
+                    direct use as an array at the field path .status.conditions.  For
+                    example,\n\n\n\ttype FooStatus struct{\n\t    // Represents the
+                    observations of a foo's current state.\n\t    // Known .status.conditions.type
+                    are: \"Available\", \"Progressing\", and \"Degraded\"\n\t    //
+                    +patchMergeKey=type\n\t    // +patchStrategy=merge\n\t    // +listType=map\n\t
+                    \   // +listMapKey=type\n\t    Conditions []metav1.Condition `json:\"conditions,omitempty\"
+                    patchStrategy:\"merge\" patchMergeKey:\"type\" protobuf:\"bytes,1,rep,name=conditions\"`\n\n\n\t
+                    \   // other fields\n\t}"
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: |-
+                        type of condition in CamelCase or in foo.example.com/CamelCase.
+                        ---
+                        Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
+                        useful (see .node.status.conditions), the ability to deconflict is important.
+                        The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                maxItems: 8
+                type: array
+                x-kubernetes-list-map-keys:
+                - type
+                x-kubernetes-list-type: map
+            required:
+            - conditions
+            type: object
+        required:
+        - spec
+        type: object
+        x-kubernetes-validations:
+        - message: The spec.type field is immutable
+          rule: self.spec.type == oldSelf.spec.type
+        - message: The spec.type field cannot be known Kong entity types
+          rule: '!(self.spec.type in [''services'',''routes'',''upstreams'',''targets'',''plugins'',''consumers'',''consumer_groups''])'
+    served: true
+    storage: true
+    subresources:
+      status: {}
diff --git a/docs/api-reference.md b/docs/api-reference.md
index 1a9f7a1c8..28bf2b3d5 100644
--- a/docs/api-reference.md
+++ b/docs/api-reference.md
@@ -313,6 +313,7 @@ _Appears in:_
 Package v1alpha1 contains API Schema definitions for the configuration.konghq.com v1alpha1 API group.
 
 - [IngressClassParameters](#ingressclassparameters)
+- [KongCustomEntity](#kongcustomentity)
 - [KongLicense](#konglicense)
 - [KongVault](#kongvault)
 ### IngressClassParameters
@@ -331,6 +332,22 @@ IngressClassParameters is the Schema for the IngressClassParameters API.
 
 
 
+### KongCustomEntity
+
+
+KongCustomEntity defines a "custom" Kong entity that KIC cannot support the entity type directly.
+
+<!-- kong_custom_entity description placeholder -->
+
+| Field | Description |
+| --- | --- |
+| `apiVersion` _string_ | `configuration.konghq.com/v1alpha1`
+| `kind` _string_ | `KongCustomEntity`
+| `metadata` _[ObjectMeta](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.25/#objectmeta-v1-meta)_ | Refer to Kubernetes API documentation for fields of `metadata`. |
+| `spec` _[KongCustomEntitySpec](#kongcustomentityspec)_ |  |
+
+
+
 ### KongLicense
 
 
@@ -428,6 +445,28 @@ Kind refers to a Kubernetes kind.
 _Appears in:_
 - [ControllerReference](#controllerreference)
 
+#### KongCustomEntitySpec
+
+
+
+
+
+
+| Field | Description |
+| --- | --- |
+| `type` _string_ | EntityType is the type of the Kong entity. The type is used in generating declarative configuration. |
+| `fields` _[JSON](#json)_ | Fields defines the fields of the Kong entity itself. |
+| `controllerName` _string_ | ControllerName specifies the controller that should reconcile it, like ingress class. |
+| `parentRef` _[ObjectReference](#objectreference)_ | ParentRef references the kubernetes resource it attached to when its scope is "attached". Currently only KongPlugin/KongClusterPlugin allowed. This will make the custom entity to be attached to the entity(service/route/consumer) where the plugin is attached. |
+
+
+_Appears in:_
+- [KongCustomEntity](#kongcustomentity)
+
+
+
+
+
 #### KongLicenseControllerStatus
 
 
@@ -496,6 +535,24 @@ RFC 1123 labels, or RFC 1035 labels.
 _Appears in:_
 - [ControllerReference](#controllerreference)
 
+#### ObjectReference
+
+
+ObjectReference defines reference of a kubernetes object.
+
+
+
+| Field | Description |
+| --- | --- |
+| `group` _string_ |  |
+| `kind` _string_ |  |
+| `namespace` _string_ | Empty namespace means the same namespace of the owning object. |
+| `name` _string_ |  |
+
+
+_Appears in:_
+- [KongCustomEntitySpec](#kongcustomentityspec)
+
 
 ## configuration.konghq.com/v1beta1
 
diff --git a/go.mod b/go.mod
index ba673aaf2..40ea2a039 100644
--- a/go.mod
+++ b/go.mod
@@ -38,7 +38,7 @@ require (
 	github.com/moul/pb v0.0.0-20220425114252-bca18df4138c
 	github.com/oapi-codegen/runtime v1.1.1
 	github.com/phayes/freeport v0.0.0-20220201140144-74d24b5ae9f5
-	github.com/prometheus/client_golang v1.19.0
+	github.com/prometheus/client_golang v1.19.1
 	github.com/prometheus/common v0.53.0
 	github.com/samber/lo v1.39.0
 	github.com/samber/mo v1.11.0
diff --git a/go.sum b/go.sum
index 73c89d8f5..31a14a2a4 100644
--- a/go.sum
+++ b/go.sum
@@ -357,8 +357,8 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH
 github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
 github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b h1:0LFwY6Q3gMACTjAbMZBjXAqTOzOwFaj2Ld6cjeQ7Rig=
 github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
-github.com/prometheus/client_golang v1.19.0 h1:ygXvpU1AoN1MhdzckN+PyD9QJOSD4x7kmXYlnfbA6JU=
-github.com/prometheus/client_golang v1.19.0/go.mod h1:ZRM9uEAypZakd+q/x7+gmsvXdURP+DABIEIjnmDdp+k=
+github.com/prometheus/client_golang v1.19.1 h1:wZWJDwK+NameRJuPGDhlnFgx8e8HN3XHQeLaYJFJBOE=
+github.com/prometheus/client_golang v1.19.1/go.mod h1:mP78NwGzrVks5S2H6ab8+ZZGJLZUq1hoULYBAYBw1Ho=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.6.0 h1:k1v3CzpSRUTrKMppY35TLwPvxHqBu0bYgxZzqGIgaos=
 github.com/prometheus/client_model v0.6.0/go.mod h1:NTQHnmxFpouOD0DpvP4XujX3CdOAGQPoaGhyTchlyt8=
diff --git a/hack/generators/cache-stores-snapshot/main.go b/hack/generators/cache-stores-snapshot/main.go
new file mode 100644
index 000000000..834ed2d01
--- /dev/null
+++ b/hack/generators/cache-stores-snapshot/main.go
@@ -0,0 +1,85 @@
+package main
+
+import (
+	"bytes"
+	"os"
+	"reflect"
+	"text/template"
+
+	"github.com/kong/kubernetes-ingress-controller/v3/internal/store"
+)
+
+// This code generates `store.CacheStores.TakeSnapshot` method.
+
+const (
+	outputFile = "zz_generated_cache_stores_snapshot.go"
+
+	outputTemplate = `// Code generated by hack/generators/cache/snapshot.go; DO NOT EDIT.
+package store
+
+import (
+	"fmt"
+
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/tools/cache"
+)
+
+func (c CacheStores) TakeSnapshot() (CacheStores, error) {
+	// Create a fresh CacheStores instance to store the snapshot.
+	snapshot := NewCacheStores()
+
+	// Gather cache store fields for all kinds.
+	allStores := []cache.Store{
+		{{- range . }}
+		c.{{ . }},
+		{{- end }}
+	}
+
+	c.l.RLock()
+	defer c.l.RUnlock()
+
+	// Iterate over all stores and add a deep copy of each object to the snapshot.
+	for _, store := range allStores {
+		for _, item := range store.List() {
+			obj, ok := item.(runtime.Object)
+			if !ok {
+				return CacheStores{}, fmt.Errorf("expected runtime.Object, got %T", item)
+			}
+
+			copiedObj := obj.DeepCopyObject()
+			if err := snapshot.Add(copiedObj); err != nil {
+				return CacheStores{}, err
+			}
+		}
+	}
+
+	return snapshot, nil
+}
+`
+)
+
+func main() {
+	cacheStoresType := reflect.TypeOf(store.CacheStores{})
+	var storesFieldsNames []string
+	for i := 0; i < cacheStoresType.NumField(); i++ {
+		field := cacheStoresType.Field(i)
+		// Only collect fields of `Store` type as `CacheStores` may include other ones (e.g. a mutex).
+		if field.Type.Name() == "Store" {
+			storesFieldsNames = append(storesFieldsNames, field.Name)
+		}
+	}
+
+	tpl, err := template.New("output").Parse(outputTemplate)
+	if err != nil {
+		panic(err)
+	}
+
+	contents := &bytes.Buffer{}
+	if err := tpl.Execute(contents, storesFieldsNames); err != nil {
+		panic(err)
+	}
+
+	if err := os.WriteFile(outputFile, contents.Bytes(), 0o600); err != nil {
+		panic(err)
+	}
+}
diff --git a/internal/admission/validation/gateway/httproute.go b/internal/admission/validation/gateway/httproute.go
index 2faebcf3e..5c897b64d 100644
--- a/internal/admission/validation/gateway/httproute.go
+++ b/internal/admission/validation/gateway/httproute.go
@@ -10,6 +10,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/admission/validation"
+	"github.com/kong/kubernetes-ingress-controller/v3/internal/admission/validation/kongplugin"
 	gatewaycontroller "github.com/kong/kubernetes-ingress-controller/v3/internal/controllers/gateway"
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/dataplane/translator"
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/dataplane/translator/subtranslator"
@@ -46,6 +47,10 @@ func ValidateHTTPRoute(
 		return true, "", nil
 	}
 
+	if err := kongplugin.ValidatePluginUniquenessPerObject(ctx, managerClient, httproute); err != nil {
+		return false, fmt.Sprintf("HTTPRoute has invalid KongPlugin annotation: %s", err), nil
+	}
+
 	if err := validateHTTPRouteTimeoutBackendRequest(httproute); err != nil {
 		return false, fmt.Sprintf("HTTPRoute spec did not pass validation: %s", err), nil
 	}
diff --git a/internal/admission/validator.go b/internal/admission/validator.go
index 78502493c..d376bc87e 100644
--- a/internal/admission/validator.go
+++ b/internal/admission/validator.go
@@ -234,6 +234,10 @@ func (validator KongHTTPValidator) ValidateConsumerGroup(
 		return true, "", nil
 	}
 
+	if err := kongplugin.ValidatePluginUniquenessPerObject(ctx, validator.ManagerClient, &consumerGroup); err != nil {
+		return false, fmt.Sprintf("KongConsumerGroup has invalid KongPlugin annotation: %s", err), nil
+	}
+
 	infoSvc, ok := validator.AdminAPIServicesProvider.GetInfoService()
 	if !ok {
 		return true, "", nil
diff --git a/internal/dataplane/kong_client.go b/internal/dataplane/kong_client.go
index acf85e8ff..54fc5d823 100644
--- a/internal/dataplane/kong_client.go
+++ b/internal/dataplane/kong_client.go
@@ -56,6 +56,7 @@ const (
 // KongConfigBuilder builds a Kong configuration from a Kubernetes object cache.
 type KongConfigBuilder interface {
 	BuildKongConfig() translator.KongConfigBuildingResult
+	UpdateCache(store.CacheStores)
 }
 
 // KongClient is a threadsafe high level API client for the Kong data-plane(s)
@@ -156,7 +157,7 @@ func NewKongClient(
 	updateStrategyResolver sendconfig.UpdateStrategyResolver,
 	configChangeDetector sendconfig.ConfigurationChangeDetector,
 	kongConfigFetcher configfetcher.LastValidConfigFetcher,
-	parser KongConfigBuilder,
+	kongConfigBuilder KongConfigBuilder,
 	cacheStores store.CacheStores,
 ) (*KongClient, error) {
 	c := &KongClient{
@@ -172,7 +173,7 @@ func NewKongClient(
 		configStatusNotifier:   clients.NoOpConfigStatusNotifier{},
 		updateStrategyResolver: updateStrategyResolver,
 		configChangeDetector:   configChangeDetector,
-		kongConfigBuilder:      parser,
+		kongConfigBuilder:      kongConfigBuilder,
 		kongConfigFetcher:      kongConfigFetcher,
 	}
 	c.initializeControllerPodReference()
@@ -396,6 +397,17 @@ func (c *KongClient) Update(ctx context.Context) error {
 		}
 	}
 
+	// If FallbackConfiguration is enabled, we take a snapshot of the cache so that we operate on a consistent
+	// set of resources in case of failures being returned from Kong. As we're going to generate a fallback config
+	// based on the cache contents, we need to ensure it is not modified during the process.
+	if c.kongConfig.FallbackConfiguration {
+		cacheSnapshot, err := c.cache.TakeSnapshot()
+		if err != nil {
+			return fmt.Errorf("failed to take snapshot of cache: %w", err)
+		}
+		c.kongConfigBuilder.UpdateCache(cacheSnapshot)
+	}
+
 	c.logger.V(util.DebugLevel).Info("Parsing kubernetes objects into data-plane configuration")
 	parsingResult := c.kongConfigBuilder.BuildKongConfig()
 	if failuresCount := len(parsingResult.TranslationFailures); failuresCount > 0 {
@@ -424,6 +436,9 @@ func (c *KongClient) Update(ctx context.Context) error {
 
 	// In case of a failure in syncing configuration with Gateways, propagate the error.
 	if gatewaysSyncErr != nil {
+		// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/5931
+		// In case of a failure in syncing configuration with Gateways, if FallbackConfiguration is enabled,
+		// we should generate a fallback configuration and push it to the gateways first.
 		if state, found := c.kongConfigFetcher.LastValidConfig(); found {
 			_, fallbackSyncErr := c.sendOutToGatewayClients(ctx, state, c.kongConfig)
 			if fallbackSyncErr != nil {
@@ -553,7 +568,7 @@ func (c *KongClient) sendToClient(
 	// If the client is Konnect and the feature flag is turned on,
 	// we should sanitize the configuration before sending it out.
 	if client.IsKonnect() && config.SanitizeKonnectConfigDumps {
-		s = s.SanitizedCopy()
+		s = s.SanitizedCopy(util.DefaultUUIDGenerator{})
 	}
 	deckGenParams := deckgen.GenerateDeckContentParams{
 		SelectorTags:                    config.FilterTags,
@@ -641,7 +656,7 @@ func prepareSendDiagnosticFn(
 	if diagnosticConfig.DumpsIncludeSensitive {
 		redactedConfig := deckgen.ToDeckContent(ctx,
 			logger,
-			targetState.SanitizedCopy(),
+			targetState.SanitizedCopy(util.DefaultUUIDGenerator{}),
 			deckGenParams,
 		)
 		config = redactedConfig
diff --git a/internal/dataplane/kongstate/consumer.go b/internal/dataplane/kongstate/consumer.go
index 9c36945ae..21d9621e3 100644
--- a/internal/dataplane/kongstate/consumer.go
+++ b/internal/dataplane/kongstate/consumer.go
@@ -5,6 +5,7 @@ import (
 
 	"github.com/kong/go-kong/kong"
 
+	"github.com/kong/kubernetes-ingress-controller/v3/internal/util"
 	kongv1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1"
 )
 
@@ -27,13 +28,13 @@ type Consumer struct {
 }
 
 // SanitizedCopy returns a shallow copy with sensitive values redacted best-effort.
-func (c *Consumer) SanitizedCopy() *Consumer {
+func (c *Consumer) SanitizedCopy(uuidGenerator util.UUIDGenerator) *Consumer {
 	return &Consumer{
 		Consumer: c.Consumer,
 		Plugins:  c.Plugins,
 		KeyAuths: func() (res []*KeyAuth) {
 			for _, v := range c.KeyAuths {
-				res = append(res, v.SanitizedCopy())
+				res = append(res, v.SanitizedCopy(uuidGenerator))
 			}
 			return
 		}(),
diff --git a/internal/dataplane/kongstate/credentials.go b/internal/dataplane/kongstate/credentials.go
index 48d9b4d60..e3db1b3a4 100644
--- a/internal/dataplane/kongstate/credentials.go
+++ b/internal/dataplane/kongstate/credentials.go
@@ -5,6 +5,8 @@ import (
 
 	"github.com/kong/go-kong/kong"
 	"github.com/mitchellh/mapstructure"
+
+	"github.com/kong/kubernetes-ingress-controller/v3/internal/util"
 )
 
 // redactedString is used to redact sensitive values in the KongState.
@@ -12,6 +14,13 @@ import (
 // to pass a valid key or a vault URI).
 var redactedString = kong.String("{vault://redacted-value}")
 
+// randRedactedString is used to redact sensitive values in the KongState when the value must be random to avoid
+// collisions.
+func randRedactedString(uuidGenerator util.UUIDGenerator) *string {
+	s := fmt.Sprintf("{vault://%s}", uuidGenerator.NewString())
+	return &s
+}
+
 // KeyAuth represents a key-auth credential.
 type KeyAuth struct {
 	kong.KeyAuth
@@ -152,13 +161,13 @@ func NewMTLSAuth(config interface{}) (*MTLSAuth, error) {
 }
 
 // SanitizedCopy returns a shallow copy with sensitive values redacted best-effort.
-func (c *KeyAuth) SanitizedCopy() *KeyAuth {
+func (c *KeyAuth) SanitizedCopy(uuidGenerator util.UUIDGenerator) *KeyAuth {
 	return &KeyAuth{
 		kong.KeyAuth{
 			// Consumer field omitted
 			CreatedAt: c.CreatedAt,
 			ID:        c.ID,
-			Key:       redactedString,
+			Key:       randRedactedString(uuidGenerator),
 			Tags:      c.Tags,
 		},
 	}
diff --git a/internal/dataplane/kongstate/kongstate.go b/internal/dataplane/kongstate/kongstate.go
index c30c9c1fe..a2b7d5be6 100644
--- a/internal/dataplane/kongstate/kongstate.go
+++ b/internal/dataplane/kongstate/kongstate.go
@@ -36,7 +36,7 @@ type KongState struct {
 }
 
 // SanitizedCopy returns a shallow copy with sensitive values redacted best-effort.
-func (ks *KongState) SanitizedCopy() *KongState {
+func (ks *KongState) SanitizedCopy(uuidGenerator util.UUIDGenerator) *KongState {
 	return &KongState{
 		Services:  ks.Services,
 		Upstreams: ks.Upstreams,
@@ -52,7 +52,7 @@ func (ks *KongState) SanitizedCopy() *KongState {
 		}),
 		Consumers: func() (res []Consumer) {
 			for _, v := range ks.Consumers {
-				res = append(res, *v.SanitizedCopy())
+				res = append(res, *v.SanitizedCopy(uuidGenerator))
 			}
 			return
 		}(),
diff --git a/internal/dataplane/sendconfig/kong.go b/internal/dataplane/sendconfig/kong.go
index 9ce82122d..59bfb0833 100644
--- a/internal/dataplane/sendconfig/kong.go
+++ b/internal/dataplane/sendconfig/kong.go
@@ -33,4 +33,8 @@ type Config struct {
 
 	// SanitizeKonnectConfigDumps indicates whether to sanitize Konnect config dumps.
 	SanitizeKonnectConfigDumps bool
+
+	// FallbackConfiguration indicates whether to generate fallback configuration in the case of entity
+	// errors returned by the Kong Admin API.
+	FallbackConfiguration bool
 }
diff --git a/internal/dataplane/translator/subtranslator/httproute.go b/internal/dataplane/translator/subtranslator/httproute.go
index 7b59b3cda..892d47492 100644
--- a/internal/dataplane/translator/subtranslator/httproute.go
+++ b/internal/dataplane/translator/subtranslator/httproute.go
@@ -8,6 +8,7 @@ import (
 	"strings"
 
 	"dario.cat/mergo"
+	"github.com/google/go-cmp/cmp"
 	"github.com/kong/go-kong/kong"
 	"github.com/samber/lo"
 
@@ -407,14 +408,13 @@ func generatePluginsFromHTTPRouteFilters(
 	filters []gatewayapi.HTTPRouteFilter,
 	path string,
 	tags []*string,
-	// As of now, expressions router is not supported for URLRewrite with PrefixMatchHTTPPathModifier.
-	// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/3686
 	expressionsRouterEnabled bool,
 ) (httpRouteFiltersOriginatedPlugins, error) {
 	if len(filters) == 0 {
 		return httpRouteFiltersOriginatedPlugins{}, nil
 	}
 	var (
+		transformerPlugins          []transformerPlugin
 		kongPlugins                 []kong.Plugin
 		pluginNamesFromExtensionRef []string
 		kongRouteModifiers          []kongRouteModifier
@@ -423,13 +423,15 @@ func generatePluginsFromHTTPRouteFilters(
 	for _, filter := range filters {
 		switch filter.Type {
 		case gatewayapi.HTTPRouteFilterRequestHeaderModifier:
-			kongPlugins = append(kongPlugins, generateRequestHeaderModifierKongPlugin(filter.RequestHeaderModifier))
+			transformerPlugins = append(transformerPlugins, generateRequestHeaderModifierKongPlugin(filter.RequestHeaderModifier))
 
 		case gatewayapi.HTTPRouteFilterRequestRedirect:
-			kongPlugins = append(kongPlugins, generateRequestRedirectKongPlugin(filter.RequestRedirect, path)...)
+			kongPlugin, transformerPlugin := generateRequestRedirectKongPlugin(filter.RequestRedirect, path)
+			kongPlugins = append(kongPlugins, kongPlugin)
+			transformerPlugins = append(transformerPlugins, transformerPlugin)
 
 		case gatewayapi.HTTPRouteFilterResponseHeaderModifier:
-			kongPlugins = append(kongPlugins, generateResponseHeaderModifierKongPlugin(filter.ResponseHeaderModifier))
+			transformerPlugins = append(transformerPlugins, generateResponseHeaderModifierKongPlugin(filter.ResponseHeaderModifier))
 
 		case gatewayapi.HTTPRouteFilterExtensionRef:
 			plugin, err := generateExtensionRefKongPlugin(filter.ExtensionRef)
@@ -439,31 +441,31 @@ func generatePluginsFromHTTPRouteFilters(
 			pluginNamesFromExtensionRef = append(pluginNamesFromExtensionRef, plugin)
 
 		case gatewayapi.HTTPRouteFilterURLRewrite:
-			plugin, routeModifier, err := generateRequestTransformerForURLRewrite(filter.URLRewrite, path, expressionsRouterEnabled)
+			plugins, routeModifiers, err := generateRequestTransformerForURLRewrite(filter.URLRewrite, path, expressionsRouterEnabled)
 			if err != nil {
 				return httpRouteFiltersOriginatedPlugins{}, err
 			}
-			kongPlugins = append(kongPlugins, plugin)
-			if routeModifier != nil {
-				kongRouteModifiers = append(kongRouteModifiers, routeModifier)
-			}
+			transformerPlugins = append(transformerPlugins, plugins...)
+			kongRouteModifiers = append(kongRouteModifiers, routeModifiers...)
 
 		case gatewayapi.HTTPRouteFilterRequestMirror:
 			// not supported
 			return httpRouteFiltersOriginatedPlugins{}, fmt.Errorf("httpFilter %s unsupported", filter.Type)
 		}
 	}
-	for _, p := range kongPlugins {
-		// This plugin is derived from an HTTPRoute filter, not a KongPlugin, so we apply tags indicating that
-		// HTTPRoute as the parent Kubernetes resource for these generated plugins.
-		p.Tags = tags
-	}
 
-	// It's possible the above loop generates multiple plugins of the same type, so we need to merge them.
+	// It's possible the above loop generates multiple transformerPlugins of the same type, so we need to merge them.
 	// It can happen for example when both RequestHeaderModifier and HTTPRouteFilterURLRewrite filters are present.
-	kongPlugins, err := mergePluginsOfTheSameType(kongPlugins)
+	transformerPlugins, err := mergePluginsOfTheSameType(transformerPlugins)
 	if err != nil {
-		return httpRouteFiltersOriginatedPlugins{}, fmt.Errorf("failed to merge plugins of the same type: %w", err)
+		return httpRouteFiltersOriginatedPlugins{}, fmt.Errorf("failed to merge transformerPlugins of the same type: %w", err)
+	}
+	kongPlugins = append(kongPlugins, transformerPluginsToKongPlugins(transformerPlugins)...)
+
+	for _, p := range kongPlugins {
+		// This plugin is derived from an HTTPRoute filter, not a KongPlugin, so we apply tags indicating that
+		// HTTPRoute as the parent Kubernetes resource for these generated transformerPlugins.
+		p.Tags = tags
 	}
 
 	if len(pluginNamesFromExtensionRef) > 0 {
@@ -477,32 +479,61 @@ func generatePluginsFromHTTPRouteFilters(
 }
 
 // mergePluginsOfTheSameType merges plugins of the same type into a single plugin with merged configurations.
-func mergePluginsOfTheSameType(plugins []kong.Plugin) ([]kong.Plugin, error) {
-	pluginsByName := lo.GroupBy(plugins, func(p kong.Plugin) string {
-		return *p.Name // Name is effectively a plugin type.
+func mergePluginsOfTheSameType(plugins []transformerPlugin) ([]transformerPlugin, error) {
+	pluginsByName := lo.GroupBy(plugins, func(p transformerPlugin) transformerPluginType {
+		return p.Type // Name is effectively a plugin type.
 	})
 	for pluginName, plugins := range pluginsByName {
+		plugins := plugins
 		// If we produced multiple plugins of the same type, we need to merge their configurations now.
 		if len(plugins) > 1 {
-			mergedPlugin := *plugins[0].DeepCopy()
+			mergedPlugin := transformerPlugin{}
 			for _, plugin := range plugins {
-				if err := mergo.Merge(&mergedPlugin.Config, plugin.Config); err != nil {
+				if err := mergo.Merge(&mergedPlugin, plugin, mergo.WithAppendSlice); err != nil {
 					// Should never happen as we're passing the same type of objects.
 					return nil, fmt.Errorf("failed to merge %q plugin configurations: %w", pluginName, err)
 				}
 			}
-			pluginsByName[pluginName] = []kong.Plugin{mergedPlugin}
+			pluginsByName[pluginName] = []transformerPlugin{mergedPlugin}
 		}
 	}
 
 	// Sort the plugins by name to ensure that the order is deterministic.
 	mergedPlugins := lo.Flatten(lo.Values(pluginsByName))
 	sort.Slice(mergedPlugins, func(i, j int) bool {
-		return *mergedPlugins[i].Name < *mergedPlugins[j].Name
+		return mergedPlugins[i].Type < mergedPlugins[j].Type
 	})
 	return mergedPlugins, nil
 }
 
+func transformerPluginsToKongPlugins(plugins []transformerPlugin) []kong.Plugin {
+	kongPlugins := make([]kong.Plugin, 0, len(plugins))
+	for _, plugin := range plugins {
+		kongPlugins = append(kongPlugins, transformerPluginToKongPlugin(plugin))
+	}
+	return kongPlugins
+}
+
+func transformerPluginToKongPlugin(plugin transformerPlugin) kong.Plugin {
+	res := kong.Plugin{
+		Name:   kong.String(string(plugin.Type)),
+		Config: kong.Configuration{},
+	}
+	if !cmp.Equal(plugin.Replace, TransformerPluginReplaceConfig{}) {
+		res.Config["replace"] = plugin.Replace
+	}
+	if !cmp.Equal(plugin.Add, TransformerPluginConfig{}) {
+		res.Config["add"] = plugin.Add
+	}
+	if !cmp.Equal(plugin.Append, TransformerPluginConfig{}) {
+		res.Config["append"] = plugin.Append
+	}
+	if !cmp.Equal(plugin.Remove, TransformerPluginConfig{}) {
+		res.Config["remove"] = plugin.Remove
+	}
+	return res
+}
+
 func generateKongRouteModifierFromExtensionRef(pluginNamesFromExtensionRef []string) kongRouteModifier {
 	return func(route *kongstate.Route) {
 		if route.Ingress.Annotations == nil {
@@ -522,9 +553,8 @@ func generateKongRouteModifierFromExtensionRef(pluginNamesFromExtensionRef []str
 
 // generateRequestRedirectKongPlugin generates configurations of plugins to satisfy the specification
 // of request redirect filter.
-func generateRequestRedirectKongPlugin(modifier *gatewayapi.HTTPRequestRedirectFilter, path string) []kong.Plugin {
-	plugins := make([]kong.Plugin, 2)
-	plugins[0] = kong.Plugin{
+func generateRequestRedirectKongPlugin(modifier *gatewayapi.HTTPRequestRedirectFilter, path string) (kong.Plugin, transformerPlugin) {
+	requestTerminationPlugin := kong.Plugin{
 		Name: kong.String("request-termination"),
 		Config: kong.Configuration{
 			"status_code": modifier.StatusCode,
@@ -553,16 +583,14 @@ func generateRequestRedirectKongPlugin(modifier *gatewayapi.HTTPRequestRedirectF
 		locationHeader = fmt.Sprintf("Location: %s", path)
 	}
 
-	plugins[1] = kong.Plugin{
-		Name: kong.String("response-transformer"),
-		Config: kong.Configuration{
-			"add": map[string][]string{
-				"headers": {locationHeader},
-			},
+	transformerPlugin := transformerPlugin{
+		Type: transformerPluginTypeResponse,
+		Add: TransformerPluginConfig{
+			Headers: []string{locationHeader},
 		},
 	}
 
-	return plugins
+	return requestTerminationPlugin, transformerPlugin
 }
 
 func generateExtensionRefKongPlugin(modifier *gatewayapi.LocalObjectReference) (string, error) {
@@ -574,20 +602,50 @@ func generateExtensionRefKongPlugin(modifier *gatewayapi.LocalObjectReference) (
 
 // generateRequestHeaderModifierKongPlugin converts a gatewayapi.HTTPRequestHeaderFilter into a
 // kong.Plugin of type request-transformer.
-func generateRequestHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter) kong.Plugin {
-	return generateHeaderModifierKongPlugin(modifier, "request-transformer")
+func generateRequestHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter) transformerPlugin {
+	return generateHeaderModifierKongPlugin(modifier, transformerPluginTypeRequest)
 }
 
 // generateResponseHeaderModifierKongPlugin converts a gatewayapi.HTTPResponseHeaderFilter into a
 // kong.Plugin of type response-transformer.
-func generateResponseHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter) kong.Plugin {
-	return generateHeaderModifierKongPlugin(modifier, "response-transformer")
+func generateResponseHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter) transformerPlugin {
+	return generateHeaderModifierKongPlugin(modifier, transformerPluginTypeResponse)
+}
+
+type transformerPluginType string
+
+const (
+	transformerPluginTypeRequest  transformerPluginType = "request-transformer"
+	transformerPluginTypeResponse transformerPluginType = "response-transformer"
+)
+
+// transformerPlugin is a configuration for request-transformer and response-transformer plugins.
+type transformerPlugin struct {
+	// Type is the type of the transformer plugin (request-transformer or response-transformer).
+	Type transformerPluginType `json:"-"`
+
+	Replace TransformerPluginReplaceConfig `json:"replace,omitempty"`
+	Add     TransformerPluginConfig        `json:"add,omitempty"`
+	Append  TransformerPluginConfig        `json:"append,omitempty"`
+	Remove  TransformerPluginConfig        `json:"remove,omitempty"`
+}
+
+// TransformerPluginConfig is a configuration for request-transformer and response-transformer plugins'
+// "add", "append", and "remove" fields.
+type TransformerPluginConfig struct {
+	Headers []string `json:"headers,omitempty"`
+}
+
+// TransformerPluginReplaceConfig is a configuration for request-transformer and response-transformer plugins'
+// "replace" field.
+type TransformerPluginReplaceConfig struct {
+	Headers []string `json:"headers,omitempty"`
+	URI     string   `json:"uri,omitempty"`
 }
 
-func generateHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter, pluginName string) kong.Plugin {
-	plugin := kong.Plugin{
-		Name:   kong.String(pluginName),
-		Config: make(kong.Configuration),
+func generateHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter, pluginType transformerPluginType) transformerPlugin {
+	plugin := transformerPlugin{
+		Type: pluginType,
 	}
 
 	// modifier.Set is converted to a pair composed of "replace" and "add"
@@ -596,11 +654,11 @@ func generateHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter, plu
 		for _, s := range modifier.Set {
 			setModifiers = append(setModifiers, kongHeaderFormatter(s))
 		}
-		plugin.Config["replace"] = map[string][]string{
-			"headers": setModifiers,
+		plugin.Replace = TransformerPluginReplaceConfig{
+			Headers: setModifiers,
 		}
-		plugin.Config["add"] = map[string][]string{
-			"headers": setModifiers,
+		plugin.Add = TransformerPluginConfig{
+			Headers: setModifiers,
 		}
 	}
 
@@ -610,14 +668,14 @@ func generateHeaderModifierKongPlugin(modifier *gatewayapi.HTTPHeaderFilter, plu
 		for _, a := range modifier.Add {
 			appendModifiers = append(appendModifiers, kongHeaderFormatter(a))
 		}
-		plugin.Config["append"] = map[string][]string{
-			"headers": appendModifiers,
+		plugin.Append = TransformerPluginConfig{
+			Headers: appendModifiers,
 		}
 	}
 
 	if modifier.Remove != nil {
-		plugin.Config["remove"] = map[string][]string{
-			"headers": modifier.Remove,
+		plugin.Remove = TransformerPluginConfig{
+			Headers: modifier.Remove,
 		}
 	}
 
@@ -632,51 +690,85 @@ func generateRequestTransformerForURLRewrite(
 	filter *gatewayapi.HTTPURLRewriteFilter,
 	path string,
 	expressionsRouterEnabled bool,
-) (kong.Plugin, kongRouteModifier, error) {
+) ([]transformerPlugin, []kongRouteModifier, error) {
 	if filter == nil {
-		return kong.Plugin{}, nil, fmt.Errorf("%s is not provided", gatewayapi.HTTPRouteFilterURLRewrite)
+		return nil, nil, fmt.Errorf("%s is not provided", gatewayapi.HTTPRouteFilterURLRewrite)
 	}
-
 	if filter.Path == nil && filter.Hostname == nil {
-		return kong.Plugin{}, nil, fmt.Errorf("%s missing Path and Hostname", gatewayapi.HTTPRouteFilterURLRewrite)
+		return nil, nil, fmt.Errorf("%s missing Path and Hostname", gatewayapi.HTTPRouteFilterURLRewrite)
 	}
 
+	var (
+		plugins        []transformerPlugin
+		routeModifiers []kongRouteModifier
+	)
 	if filter.Path != nil {
-		switch filter.Path.Type {
-		case gatewayapi.FullPathHTTPPathModifier:
-			plugin, err := generateRequestTransformerForURLRewriteFullPath(filter)
-			if err != nil {
-				return kong.Plugin{}, nil, fmt.Errorf("failed to generate request-transformer plugin for %s: %w", gatewayapi.HTTPRouteFilterURLRewrite, err)
-			}
-			return plugin, nil, nil
-
-		case gatewayapi.PrefixMatchHTTPPathModifier:
-			plugin, routeModifier := generateRequestTransformerForURLRewritePrefixMatch(filter, path, expressionsRouterEnabled)
-			return plugin, routeModifier, nil
+		plugin, modifier, err := generateRequestTransformerForURLRewritePath(filter, path, expressionsRouterEnabled)
+		if err != nil {
+			return nil, nil, fmt.Errorf("failed to generate request-transformer plugin for Path: %w", err)
+		}
+		plugins = append(plugins, plugin)
+		if modifier != nil {
+			routeModifiers = append(routeModifiers, modifier)
 		}
 	}
-
-	// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/3685
 	if filter.Hostname != nil {
-		return kong.Plugin{}, nil, fmt.Errorf("unsupported hostname replace for %s", gatewayapi.HTTPRouteFilterURLRewrite)
+		plugins = append(plugins, generateRequestTransformerForURLRewriteHostname(filter))
 	}
 
-	return kong.Plugin{}, nil, fmt.Errorf("invalid %s config", gatewayapi.HTTPRouteFilterURLRewrite)
+	return plugins, routeModifiers, nil
+}
+
+func generateRequestTransformerForURLRewritePath(
+	filter *gatewayapi.HTTPURLRewriteFilter,
+	path string,
+	expressionsRouterEnabled bool,
+) (transformerPlugin, kongRouteModifier, error) {
+	switch filter.Path.Type {
+	case gatewayapi.FullPathHTTPPathModifier:
+		plugin, err := generateRequestTransformerForURLRewriteFullPath(filter)
+		if err != nil {
+			return transformerPlugin{}, nil, fmt.Errorf("failed to generate request-transformer plugin for %s: %w", gatewayapi.HTTPRouteFilterURLRewrite, err)
+		}
+		return plugin, nil, nil
+
+	case gatewayapi.PrefixMatchHTTPPathModifier:
+		plugin, routeModifier := generateRequestTransformerForURLRewritePrefixMatch(filter, path, expressionsRouterEnabled)
+		return plugin, routeModifier, nil
+	default:
+		return transformerPlugin{}, nil, fmt.Errorf("unsupported path type %s for %s", filter.Path.Type, gatewayapi.HTTPRouteFilterURLRewrite)
+	}
+}
+
+func generateRequestTransformerForURLRewriteHostname(
+	filter *gatewayapi.HTTPURLRewriteFilter,
+) transformerPlugin {
+	return transformerPlugin{
+		Type: transformerPluginTypeRequest,
+		Replace: TransformerPluginReplaceConfig{
+			Headers: []string{
+				fmt.Sprintf("host:%s", string(*filter.Hostname)),
+			},
+		},
+		Add: TransformerPluginConfig{
+			Headers: []string{
+				fmt.Sprintf("host:%s", string(*filter.Hostname)),
+			},
+		},
+	}
 }
 
 // generateRequestTransformerForURLRewriteFullPath generates a request-transformer plugin for the URLRewrite filter
 // with a FullPathHTTPPathModifier.
-func generateRequestTransformerForURLRewriteFullPath(filter *gatewayapi.HTTPURLRewriteFilter) (kong.Plugin, error) {
+func generateRequestTransformerForURLRewriteFullPath(filter *gatewayapi.HTTPURLRewriteFilter) (transformerPlugin, error) {
 	if filter.Path.ReplaceFullPath == nil {
-		return kong.Plugin{}, fmt.Errorf("%s missing ReplaceFullPath", gatewayapi.HTTPRouteFilterURLRewrite)
+		return transformerPlugin{}, fmt.Errorf("%s missing ReplaceFullPath", gatewayapi.HTTPRouteFilterURLRewrite)
 	}
 
-	return kong.Plugin{
-		Name: kong.String("request-transformer"),
-		Config: kong.Configuration{
-			"replace": map[string]string{
-				"uri": *filter.Path.ReplaceFullPath,
-			},
+	return transformerPlugin{
+		Type: transformerPluginTypeRequest,
+		Replace: TransformerPluginReplaceConfig{
+			URI: *filter.Path.ReplaceFullPath,
 		},
 	}, nil
 }
@@ -687,19 +779,17 @@ func generateRequestTransformerForURLRewritePrefixMatch(
 	filter *gatewayapi.HTTPURLRewriteFilter,
 	path string,
 	expressionsRouterEnabled bool,
-) (kong.Plugin, kongRouteModifier) {
+) (transformerPlugin, kongRouteModifier) {
 	// Normalize the path before passing it down.
 	path = normalizePath(path)
 
-	return kong.Plugin{
-		Name: kong.String("request-transformer"),
-		Config: kong.Configuration{
-			"replace": map[string]string{
-				"uri": generateRequestTransformerReplaceURIForURLRewritePrefixMatch(
-					filter.Path.ReplacePrefixMatch,
-					path,
-				),
-			},
+	return transformerPlugin{
+		Type: transformerPluginTypeRequest,
+		Replace: TransformerPluginReplaceConfig{
+			URI: generateRequestTransformerReplaceURIForURLRewritePrefixMatch(
+				filter.Path.ReplacePrefixMatch,
+				path,
+			),
 		},
 	}, generateKongRouteModifierForURLRewritePrefixMatch(path, expressionsRouterEnabled)
 }
diff --git a/internal/dataplane/translator/translator.go b/internal/dataplane/translator/translator.go
index c817c0222..d1abad4a8 100644
--- a/internal/dataplane/translator/translator.go
+++ b/internal/dataplane/translator/translator.go
@@ -127,6 +127,12 @@ type KongConfigBuildingResult struct {
 	ConfiguredKubernetesObjects []client.Object
 }
 
+// UpdateCache updates the store cache used by the translator.
+// This method can be used to swap the cache with another one (e.g. the last valid snapshot).
+func (t *Translator) UpdateCache(c store.CacheStores) {
+	t.storer.UpdateCache(c)
+}
+
 // BuildKongConfig creates a Kong configuration from Ingress and Custom resources
 // defined in Kubernetes.
 func (t *Translator) BuildKongConfig() KongConfigBuildingResult {
diff --git a/internal/manager/featuregates/feature_gates.go b/internal/manager/featuregates/feature_gates.go
index a1a0f7c5f..237bfcc8c 100644
--- a/internal/manager/featuregates/feature_gates.go
+++ b/internal/manager/featuregates/feature_gates.go
@@ -28,6 +28,10 @@ const (
 	// SanitizeKonnectConfigDumps is the name of the feature-gate that enables sanitization of Konnect config dumps.
 	SanitizeKonnectConfigDumps = "SanitizeKonnectConfigDumps"
 
+	// FallbackConfiguration is the name of the featuer-gate that enables generating fallback configuration in the case
+	// of entity errors returned by the Kong Admin API.
+	FallbackConfiguration = "FallbackConfiguration"
+
 	// DocsURL provides a link to the documentation for feature gates in the KIC repository.
 	DocsURL = "https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md"
 )
@@ -68,5 +72,6 @@ func GetFeatureGatesDefaults() FeatureGates {
 		RewriteURIsFeature:         false,
 		KongServiceFacade:          false,
 		SanitizeKonnectConfigDumps: true,
+		FallbackConfiguration:      false,
 	}
 }
diff --git a/internal/manager/run.go b/internal/manager/run.go
index 32c4cfd9b..ccc8a59d8 100644
--- a/internal/manager/run.go
+++ b/internal/manager/run.go
@@ -122,6 +122,7 @@ func Run(
 		EnableReverseSync:          c.EnableReverseSync,
 		ExpressionRoutes:           dpconf.ShouldEnableExpressionRoutes(routerFlavor),
 		SanitizeKonnectConfigDumps: featureGates.Enabled(featuregates.SanitizeKonnectConfigDumps),
+		FallbackConfiguration:      featureGates.Enabled(featuregates.FallbackConfiguration),
 	}
 
 	setupLog.Info("Configuring and building the controller manager")
@@ -177,12 +178,7 @@ func Run(
 	referenceIndexers := ctrlref.NewCacheIndexers(setupLog.WithName("reference-indexers"))
 	cache := store.NewCacheStores()
 	storer := store.New(cache, c.IngressClassName, logger)
-	configTranslator, err := translator.NewTranslator(
-		logger,
-		storer,
-		c.KongWorkspace,
-		translatorFeatureFlags,
-	)
+	configTranslator, err := translator.NewTranslator(logger, storer, c.KongWorkspace, translatorFeatureFlags)
 	if err != nil {
 		return fmt.Errorf("failed to create translator: %w", err)
 	}
diff --git a/internal/store/cache_stores.go b/internal/store/cache_stores.go
index a4483fb2c..79aa7b3db 100644
--- a/internal/store/cache_stores.go
+++ b/internal/store/cache_stores.go
@@ -20,6 +20,8 @@ import (
 	incubatorv1alpha1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/incubator/v1alpha1"
 )
 
+//go:generate go run ../../hack/generators/cache-stores-snapshot
+
 // CacheStores stores cache.Store for all Kinds of k8s objects that
 // the Ingress Controller reads.
 type CacheStores struct {
diff --git a/internal/store/fake_store.go b/internal/store/fake_store.go
index 8df30fd15..2edcaa2bd 100644
--- a/internal/store/fake_store.go
+++ b/internal/store/fake_store.go
@@ -227,7 +227,7 @@ func NewFakeStore(
 		}
 	}
 
-	s = Store{
+	s = &Store{
 		stores: CacheStores{
 			IngressV1:                      ingressV1Store,
 			IngressClassV1:                 ingressClassV1Store,
diff --git a/internal/store/store.go b/internal/store/store.go
index de2b0c987..17c237a49 100644
--- a/internal/store/store.go
+++ b/internal/store/store.go
@@ -56,6 +56,8 @@ const (
 // Storer is the interface that wraps the required methods to gather information
 // about ingresses, services, secrets and ingress annotations.
 type Storer interface {
+	UpdateCache(cs CacheStores)
+
 	GetSecret(namespace, name string) (*corev1.Secret, error)
 	GetService(namespace, name string) (*corev1.Service, error)
 	GetEndpointSlicesForService(namespace, name string) ([]*discoveryv1.EndpointSlice, error)
@@ -109,11 +111,11 @@ type Store struct {
 	logger logr.Logger
 }
 
-var _ Storer = Store{}
+var _ Storer = &Store{}
 
 // New creates a new object store to be used in the ingress controller.
-func New(cs CacheStores, ingressClass string, logger logr.Logger) Storer {
-	return Store{
+func New(cs CacheStores, ingressClass string, logger logr.Logger) *Store {
+	return &Store{
 		stores:                cs,
 		ingressClass:          ingressClass,
 		ingressClassMatching:  annotations.ExactClassMatch,
@@ -123,6 +125,11 @@ func New(cs CacheStores, ingressClass string, logger logr.Logger) Storer {
 	}
 }
 
+// UpdateCache updates the cache stores in the Store.
+func (s *Store) UpdateCache(cs CacheStores) {
+	s.stores = cs
+}
+
 // GetSecret returns a Secret using the namespace and name as key.
 func (s Store) GetSecret(namespace, name string) (*corev1.Secret, error) {
 	key := fmt.Sprintf("%v/%v", namespace, name)
diff --git a/internal/store/zz_generated_cache_stores_snapshot.go b/internal/store/zz_generated_cache_stores_snapshot.go
new file mode 100644
index 000000000..300f4f22a
--- /dev/null
+++ b/internal/store/zz_generated_cache_stores_snapshot.go
@@ -0,0 +1,61 @@
+// Code generated by hack/generators/cache/snapshot.go; DO NOT EDIT.
+package store
+
+import (
+	"fmt"
+
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/tools/cache"
+)
+
+func (c CacheStores) TakeSnapshot() (CacheStores, error) {
+	// Create a fresh CacheStores instance to store the snapshot.
+	snapshot := NewCacheStores()
+
+	// Gather cache store fields for all kinds.
+	allStores := []cache.Store{
+		c.IngressV1,
+		c.IngressClassV1,
+		c.Service,
+		c.Secret,
+		c.EndpointSlice,
+		c.HTTPRoute,
+		c.UDPRoute,
+		c.TCPRoute,
+		c.TLSRoute,
+		c.GRPCRoute,
+		c.ReferenceGrant,
+		c.Gateway,
+		c.Plugin,
+		c.ClusterPlugin,
+		c.Consumer,
+		c.ConsumerGroup,
+		c.KongIngress,
+		c.TCPIngress,
+		c.UDPIngress,
+		c.KongUpstreamPolicy,
+		c.IngressClassParametersV1alpha1,
+		c.KongServiceFacade,
+		c.KongVault,
+	}
+
+	c.l.RLock()
+	defer c.l.RUnlock()
+
+	// Iterate over all stores and add a deep copy of each object to the snapshot.
+	for _, store := range allStores {
+		for _, item := range store.List() {
+			obj, ok := item.(runtime.Object)
+			if !ok {
+				return CacheStores{}, fmt.Errorf("expected runtime.Object, got %T", item)
+			}
+
+			copiedObj := obj.DeepCopyObject()
+			if err := snapshot.Add(copiedObj); err != nil {
+				return CacheStores{}, err
+			}
+		}
+	}
+
+	return snapshot, nil
+}
diff --git a/internal/util/uuid.go b/internal/util/uuid.go
new file mode 100644
index 000000000..03374b4ba
--- /dev/null
+++ b/internal/util/uuid.go
@@ -0,0 +1,15 @@
+package util
+
+import "github.com/google/uuid"
+
+// UUIDGenerator is an interface to generate UUIDs.
+type UUIDGenerator interface {
+	NewString() string
+}
+
+// DefaultUUIDGenerator is the default implementation of UUIDGenerator.
+type DefaultUUIDGenerator struct{}
+
+func (DefaultUUIDGenerator) NewString() string {
+	return uuid.NewString()
+}
diff --git a/pkg/apis/configuration/v1alpha1/kong_custom_entity_types.go b/pkg/apis/configuration/v1alpha1/kong_custom_entity_types.go
new file mode 100644
index 000000000..1f672091b
--- /dev/null
+++ b/pkg/apis/configuration/v1alpha1/kong_custom_entity_types.go
@@ -0,0 +1,85 @@
+package v1alpha1
+
+import (
+	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+const (
+	KongCustomEntityKind = "KongCustomEntity"
+)
+
+type KongEntityScope string
+
+// +genclient
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+// +kubebuilder:object:root=true
+// +kubebuilder:resource:scope=Namespaced,shortName=kce,categories=kong-ingress-controller,path=kongcustomentities,singular=kongcustomentity
+// +kubebuilder:storageversion
+// +kubebuilder:subresource:status
+// +kubebuilder:printcolumn:name="Entity Type",type=string,JSONPath=`.spec.type`,description="type of the Kong entity"
+// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`,description="Age"
+// +kubebuilder:printcolumn:name="Programmed",type=string,JSONPath=`.status.conditions[?(@.type=="Programmed")].status`
+// +kubebuilder:validation:XValidation:rule="self.spec.type == oldSelf.spec.type",message="The spec.type field is immutable"
+// +kubebuilder:validation:XValidation:rule="!(self.spec.type in ['services','routes','upstreams','targets','plugins','consumers','consumer_groups'])",message="The spec.type field cannot be known Kong entity types"
+
+// KongCustomEntity defines a "custom" Kong entity that KIC cannot support the entity type directly.
+type KongCustomEntity struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec KongCustomEntitySpec `json:"spec"`
+
+	// Status stores the reconciling status of the resource.
+	Status KongCustomEntityStatus `json:"status,omitempty"`
+}
+
+type KongCustomEntitySpec struct {
+	// EntityType is the type of the Kong entity. The type is used in generating declarative configuration.
+	EntityType string `json:"type"`
+	// Fields defines the fields of the Kong entity itself.
+	Fields apiextensionsv1.JSON `json:"fields"`
+	// ControllerName specifies the controller that should reconcile it, like ingress class.
+	ControllerName string `json:"controllerName"`
+
+	// ParentRef references the kubernetes resource it attached to when its scope is "attached".
+	// Currently only KongPlugin/KongClusterPlugin allowed. This will make the custom entity to be attached
+	// to the entity(service/route/consumer) where the plugin is attached.
+	ParentRef *ObjectReference `json:"parentRef,omitempty"`
+}
+
+// ObjectReference defines reference of a kubernetes object.
+type ObjectReference struct {
+	Group *string `json:"group,omitempty"`
+	Kind  *string `json:"kind,omitempty"`
+	// Empty namespace means the same namespace of the owning object.
+	Namespace *string `json:"namespace,omitempty"`
+	Name      string  `json:"name"`
+}
+
+type KongCustomEntityStatus struct {
+	// Conditions describe the current conditions of the KongCustomEntityStatus.
+	//
+	// Known condition types are:
+	//
+	// * "Programmed"
+	//
+	// +listType=map
+	// +listMapKey=type
+	// +kubebuilder:validation:MaxItems=8
+	// +kubebuilder:default={{type: "Programmed", status: "Unknown", reason:"Pending", message:"Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}
+	Conditions []metav1.Condition `json:"conditions"`
+}
+
+// +kubebuilder:object:root=true
+
+// KongCustomEntityList contains a list of KongCustomEntity.
+type KongCustomEntityList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty"`
+	Items           []KongCustomEntity `json:"items"`
+}
+
+func init() {
+	SchemeBuilder.Register(&KongCustomEntity{}, &KongCustomEntityList{})
+}
diff --git a/pkg/apis/configuration/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/configuration/v1alpha1/zz_generated.deepcopy.go
index 05997c29c..a3ec7c9de 100644
--- a/pkg/apis/configuration/v1alpha1/zz_generated.deepcopy.go
+++ b/pkg/apis/configuration/v1alpha1/zz_generated.deepcopy.go
@@ -128,6 +128,108 @@ func (in *IngressClassParametersSpec) DeepCopy() *IngressClassParametersSpec {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *KongCustomEntity) DeepCopyInto(out *KongCustomEntity) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	in.Status.DeepCopyInto(&out.Status)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KongCustomEntity.
+func (in *KongCustomEntity) DeepCopy() *KongCustomEntity {
+	if in == nil {
+		return nil
+	}
+	out := new(KongCustomEntity)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *KongCustomEntity) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *KongCustomEntityList) DeepCopyInto(out *KongCustomEntityList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]KongCustomEntity, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KongCustomEntityList.
+func (in *KongCustomEntityList) DeepCopy() *KongCustomEntityList {
+	if in == nil {
+		return nil
+	}
+	out := new(KongCustomEntityList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *KongCustomEntityList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *KongCustomEntitySpec) DeepCopyInto(out *KongCustomEntitySpec) {
+	*out = *in
+	in.Fields.DeepCopyInto(&out.Fields)
+	if in.ParentRef != nil {
+		in, out := &in.ParentRef, &out.ParentRef
+		*out = new(ObjectReference)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KongCustomEntitySpec.
+func (in *KongCustomEntitySpec) DeepCopy() *KongCustomEntitySpec {
+	if in == nil {
+		return nil
+	}
+	out := new(KongCustomEntitySpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *KongCustomEntityStatus) DeepCopyInto(out *KongCustomEntityStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KongCustomEntityStatus.
+func (in *KongCustomEntityStatus) DeepCopy() *KongCustomEntityStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(KongCustomEntityStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *KongLicense) DeepCopyInto(out *KongLicense) {
 	*out = *in
@@ -331,3 +433,33 @@ func (in *KongVaultStatus) DeepCopy() *KongVaultStatus {
 	in.DeepCopyInto(out)
 	return out
 }
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *ObjectReference) DeepCopyInto(out *ObjectReference) {
+	*out = *in
+	if in.Group != nil {
+		in, out := &in.Group, &out.Group
+		*out = new(string)
+		**out = **in
+	}
+	if in.Kind != nil {
+		in, out := &in.Kind, &out.Kind
+		*out = new(string)
+		**out = **in
+	}
+	if in.Namespace != nil {
+		in, out := &in.Namespace, &out.Namespace
+		*out = new(string)
+		**out = **in
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ObjectReference.
+func (in *ObjectReference) DeepCopy() *ObjectReference {
+	if in == nil {
+		return nil
+	}
+	out := new(ObjectReference)
+	in.DeepCopyInto(out)
+	return out
+}
diff --git a/pkg/clientset/typed/configuration/v1alpha1/configuration_client.go b/pkg/clientset/typed/configuration/v1alpha1/configuration_client.go
index 63b28f3aa..440b7a55e 100644
--- a/pkg/clientset/typed/configuration/v1alpha1/configuration_client.go
+++ b/pkg/clientset/typed/configuration/v1alpha1/configuration_client.go
@@ -29,6 +29,7 @@ import (
 type ConfigurationV1alpha1Interface interface {
 	RESTClient() rest.Interface
 	IngressClassParametersesGetter
+	KongCustomEntitiesGetter
 	KongLicensesGetter
 	KongVaultsGetter
 }
@@ -42,6 +43,10 @@ func (c *ConfigurationV1alpha1Client) IngressClassParameterses(namespace string)
 	return newIngressClassParameterses(c, namespace)
 }
 
+func (c *ConfigurationV1alpha1Client) KongCustomEntities(namespace string) KongCustomEntityInterface {
+	return newKongCustomEntities(c, namespace)
+}
+
 func (c *ConfigurationV1alpha1Client) KongLicenses() KongLicenseInterface {
 	return newKongLicenses(c)
 }
diff --git a/pkg/clientset/typed/configuration/v1alpha1/fake/fake_configuration_client.go b/pkg/clientset/typed/configuration/v1alpha1/fake/fake_configuration_client.go
index 6a7de0703..be08b5cd2 100644
--- a/pkg/clientset/typed/configuration/v1alpha1/fake/fake_configuration_client.go
+++ b/pkg/clientset/typed/configuration/v1alpha1/fake/fake_configuration_client.go
@@ -32,6 +32,10 @@ func (c *FakeConfigurationV1alpha1) IngressClassParameterses(namespace string) v
 	return &FakeIngressClassParameterses{c, namespace}
 }
 
+func (c *FakeConfigurationV1alpha1) KongCustomEntities(namespace string) v1alpha1.KongCustomEntityInterface {
+	return &FakeKongCustomEntities{c, namespace}
+}
+
 func (c *FakeConfigurationV1alpha1) KongLicenses() v1alpha1.KongLicenseInterface {
 	return &FakeKongLicenses{c}
 }
diff --git a/pkg/clientset/typed/configuration/v1alpha1/fake/fake_kongcustomentity.go b/pkg/clientset/typed/configuration/v1alpha1/fake/fake_kongcustomentity.go
new file mode 100644
index 000000000..61b69d7fb
--- /dev/null
+++ b/pkg/clientset/typed/configuration/v1alpha1/fake/fake_kongcustomentity.go
@@ -0,0 +1,141 @@
+/*
+Copyright 2021 Kong, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	"context"
+
+	v1alpha1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	testing "k8s.io/client-go/testing"
+)
+
+// FakeKongCustomEntities implements KongCustomEntityInterface
+type FakeKongCustomEntities struct {
+	Fake *FakeConfigurationV1alpha1
+	ns   string
+}
+
+var kongcustomentitiesResource = v1alpha1.SchemeGroupVersion.WithResource("kongcustomentities")
+
+var kongcustomentitiesKind = v1alpha1.SchemeGroupVersion.WithKind("KongCustomEntity")
+
+// Get takes name of the kongCustomEntity, and returns the corresponding kongCustomEntity object, and an error if there is any.
+func (c *FakeKongCustomEntities) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewGetAction(kongcustomentitiesResource, c.ns, name), &v1alpha1.KongCustomEntity{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.KongCustomEntity), err
+}
+
+// List takes label and field selectors, and returns the list of KongCustomEntities that match those selectors.
+func (c *FakeKongCustomEntities) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.KongCustomEntityList, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewListAction(kongcustomentitiesResource, kongcustomentitiesKind, c.ns, opts), &v1alpha1.KongCustomEntityList{})
+
+	if obj == nil {
+		return nil, err
+	}
+
+	label, _, _ := testing.ExtractFromListOptions(opts)
+	if label == nil {
+		label = labels.Everything()
+	}
+	list := &v1alpha1.KongCustomEntityList{ListMeta: obj.(*v1alpha1.KongCustomEntityList).ListMeta}
+	for _, item := range obj.(*v1alpha1.KongCustomEntityList).Items {
+		if label.Matches(labels.Set(item.Labels)) {
+			list.Items = append(list.Items, item)
+		}
+	}
+	return list, err
+}
+
+// Watch returns a watch.Interface that watches the requested kongCustomEntities.
+func (c *FakeKongCustomEntities) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	return c.Fake.
+		InvokesWatch(testing.NewWatchAction(kongcustomentitiesResource, c.ns, opts))
+
+}
+
+// Create takes the representation of a kongCustomEntity and creates it.  Returns the server's representation of the kongCustomEntity, and an error, if there is any.
+func (c *FakeKongCustomEntities) Create(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.CreateOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewCreateAction(kongcustomentitiesResource, c.ns, kongCustomEntity), &v1alpha1.KongCustomEntity{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.KongCustomEntity), err
+}
+
+// Update takes the representation of a kongCustomEntity and updates it. Returns the server's representation of the kongCustomEntity, and an error, if there is any.
+func (c *FakeKongCustomEntities) Update(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewUpdateAction(kongcustomentitiesResource, c.ns, kongCustomEntity), &v1alpha1.KongCustomEntity{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.KongCustomEntity), err
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *FakeKongCustomEntities) UpdateStatus(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (*v1alpha1.KongCustomEntity, error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewUpdateSubresourceAction(kongcustomentitiesResource, "status", c.ns, kongCustomEntity), &v1alpha1.KongCustomEntity{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.KongCustomEntity), err
+}
+
+// Delete takes name of the kongCustomEntity and deletes it. Returns an error if one occurs.
+func (c *FakeKongCustomEntities) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	_, err := c.Fake.
+		Invokes(testing.NewDeleteActionWithOptions(kongcustomentitiesResource, c.ns, name, opts), &v1alpha1.KongCustomEntity{})
+
+	return err
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *FakeKongCustomEntities) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	action := testing.NewDeleteCollectionAction(kongcustomentitiesResource, c.ns, listOpts)
+
+	_, err := c.Fake.Invokes(action, &v1alpha1.KongCustomEntityList{})
+	return err
+}
+
+// Patch applies the patch and returns the patched kongCustomEntity.
+func (c *FakeKongCustomEntities) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.KongCustomEntity, err error) {
+	obj, err := c.Fake.
+		Invokes(testing.NewPatchSubresourceAction(kongcustomentitiesResource, c.ns, name, pt, data, subresources...), &v1alpha1.KongCustomEntity{})
+
+	if obj == nil {
+		return nil, err
+	}
+	return obj.(*v1alpha1.KongCustomEntity), err
+}
diff --git a/pkg/clientset/typed/configuration/v1alpha1/generated_expansion.go b/pkg/clientset/typed/configuration/v1alpha1/generated_expansion.go
index bab2185b7..a05eae8a7 100644
--- a/pkg/clientset/typed/configuration/v1alpha1/generated_expansion.go
+++ b/pkg/clientset/typed/configuration/v1alpha1/generated_expansion.go
@@ -20,6 +20,8 @@ package v1alpha1
 
 type IngressClassParametersExpansion interface{}
 
+type KongCustomEntityExpansion interface{}
+
 type KongLicenseExpansion interface{}
 
 type KongVaultExpansion interface{}
diff --git a/pkg/clientset/typed/configuration/v1alpha1/kongcustomentity.go b/pkg/clientset/typed/configuration/v1alpha1/kongcustomentity.go
new file mode 100644
index 000000000..6401393d5
--- /dev/null
+++ b/pkg/clientset/typed/configuration/v1alpha1/kongcustomentity.go
@@ -0,0 +1,195 @@
+/*
+Copyright 2021 Kong, Inc.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"context"
+	"time"
+
+	v1alpha1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1alpha1"
+	scheme "github.com/kong/kubernetes-ingress-controller/v3/pkg/clientset/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	rest "k8s.io/client-go/rest"
+)
+
+// KongCustomEntitiesGetter has a method to return a KongCustomEntityInterface.
+// A group's client should implement this interface.
+type KongCustomEntitiesGetter interface {
+	KongCustomEntities(namespace string) KongCustomEntityInterface
+}
+
+// KongCustomEntityInterface has methods to work with KongCustomEntity resources.
+type KongCustomEntityInterface interface {
+	Create(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.CreateOptions) (*v1alpha1.KongCustomEntity, error)
+	Update(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (*v1alpha1.KongCustomEntity, error)
+	UpdateStatus(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (*v1alpha1.KongCustomEntity, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*v1alpha1.KongCustomEntity, error)
+	List(ctx context.Context, opts v1.ListOptions) (*v1alpha1.KongCustomEntityList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.KongCustomEntity, err error)
+	KongCustomEntityExpansion
+}
+
+// kongCustomEntities implements KongCustomEntityInterface
+type kongCustomEntities struct {
+	client rest.Interface
+	ns     string
+}
+
+// newKongCustomEntities returns a KongCustomEntities
+func newKongCustomEntities(c *ConfigurationV1alpha1Client, namespace string) *kongCustomEntities {
+	return &kongCustomEntities{
+		client: c.RESTClient(),
+		ns:     namespace,
+	}
+}
+
+// Get takes name of the kongCustomEntity, and returns the corresponding kongCustomEntity object, and an error if there is any.
+func (c *kongCustomEntities) Get(ctx context.Context, name string, options v1.GetOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	result = &v1alpha1.KongCustomEntity{}
+	err = c.client.Get().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		Name(name).
+		VersionedParams(&options, scheme.ParameterCodec).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// List takes label and field selectors, and returns the list of KongCustomEntities that match those selectors.
+func (c *kongCustomEntities) List(ctx context.Context, opts v1.ListOptions) (result *v1alpha1.KongCustomEntityList, err error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	result = &v1alpha1.KongCustomEntityList{}
+	err = c.client.Get().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Watch returns a watch.Interface that watches the requested kongCustomEntities.
+func (c *kongCustomEntities) Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error) {
+	var timeout time.Duration
+	if opts.TimeoutSeconds != nil {
+		timeout = time.Duration(*opts.TimeoutSeconds) * time.Second
+	}
+	opts.Watch = true
+	return c.client.Get().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Watch(ctx)
+}
+
+// Create takes the representation of a kongCustomEntity and creates it.  Returns the server's representation of the kongCustomEntity, and an error, if there is any.
+func (c *kongCustomEntities) Create(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.CreateOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	result = &v1alpha1.KongCustomEntity{}
+	err = c.client.Post().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(kongCustomEntity).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Update takes the representation of a kongCustomEntity and updates it. Returns the server's representation of the kongCustomEntity, and an error, if there is any.
+func (c *kongCustomEntities) Update(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	result = &v1alpha1.KongCustomEntity{}
+	err = c.client.Put().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		Name(kongCustomEntity.Name).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(kongCustomEntity).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// UpdateStatus was generated because the type contains a Status member.
+// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+func (c *kongCustomEntities) UpdateStatus(ctx context.Context, kongCustomEntity *v1alpha1.KongCustomEntity, opts v1.UpdateOptions) (result *v1alpha1.KongCustomEntity, err error) {
+	result = &v1alpha1.KongCustomEntity{}
+	err = c.client.Put().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		Name(kongCustomEntity.Name).
+		SubResource("status").
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(kongCustomEntity).
+		Do(ctx).
+		Into(result)
+	return
+}
+
+// Delete takes name of the kongCustomEntity and deletes it. Returns an error if one occurs.
+func (c *kongCustomEntities) Delete(ctx context.Context, name string, opts v1.DeleteOptions) error {
+	return c.client.Delete().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		Name(name).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// DeleteCollection deletes a collection of objects.
+func (c *kongCustomEntities) DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error {
+	var timeout time.Duration
+	if listOpts.TimeoutSeconds != nil {
+		timeout = time.Duration(*listOpts.TimeoutSeconds) * time.Second
+	}
+	return c.client.Delete().
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		VersionedParams(&listOpts, scheme.ParameterCodec).
+		Timeout(timeout).
+		Body(&opts).
+		Do(ctx).
+		Error()
+}
+
+// Patch applies the patch and returns the patched kongCustomEntity.
+func (c *kongCustomEntities) Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *v1alpha1.KongCustomEntity, err error) {
+	result = &v1alpha1.KongCustomEntity{}
+	err = c.client.Patch(pt).
+		Namespace(c.ns).
+		Resource("kongcustomentities").
+		Name(name).
+		SubResource(subresources...).
+		VersionedParams(&opts, scheme.ParameterCodec).
+		Body(data).
+		Do(ctx).
+		Into(result)
+	return
+}


Test patch:
diff --git a/internal/admission/validation/gateway/httproute_test.go b/internal/admission/validation/gateway/httproute_test.go
index b7971a3b5..a52342c4b 100644
--- a/internal/admission/validation/gateway/httproute_test.go
+++ b/internal/admission/validation/gateway/httproute_test.go
@@ -17,6 +17,7 @@ import (
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/dataplane/translator"
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/gatewayapi"
 	"github.com/kong/kubernetes-ingress-controller/v3/internal/manager/scheme"
+	kongv1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1"
 )
 
 func TestValidateHTTPRoute(t *testing.T) {
@@ -78,7 +79,7 @@ func TestValidateHTTPRoute(t *testing.T) {
 			valid: true,
 		},
 		{
-			msg: "route with parentRef to non-gateway object is accepted with no vlaidation",
+			msg: "route with parentRef to non-gateway object is accepted with no validation",
 			route: &gatewayapi.HTTPRoute{
 				ObjectMeta: metav1.ObjectMeta{
 					Namespace: corev1.NamespaceDefault,
@@ -161,6 +162,128 @@ func TestValidateHTTPRoute(t *testing.T) {
 			},
 			valid: true,
 		},
+		{
+			msg: "valid HTTPRoute with multiple plugins of the same type attached - does not pass validation",
+			route: &gatewayapi.HTTPRoute{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace: corev1.NamespaceDefault,
+					Name:      "testing-httproute",
+					Annotations: map[string]string{
+						"konghq.com/plugins": "plugin1,plugin2,plugin3",
+					},
+				},
+				Spec: gatewayapi.HTTPRouteSpec{
+					CommonRouteSpec: gatewayapi.CommonRouteSpec{
+						ParentRefs: []gatewayapi.ParentReference{{
+							Name: "testing-gateway",
+						}},
+					},
+				},
+			},
+			cachedObjects: []client.Object{
+				gatewayClass,
+				&gatewayapi.Gateway{
+					ObjectMeta: metav1.ObjectMeta{
+						Namespace: corev1.NamespaceDefault,
+						Name:      "testing-gateway",
+					},
+					Spec: gatewayapi.GatewaySpec{
+						GatewayClassName: gatewayClassName,
+						Listeners: []gatewayapi.Listener{{
+							Name:     "http",
+							Port:     80,
+							Protocol: (gatewayapi.HTTPProtocolType),
+							AllowedRoutes: &gatewayapi.AllowedRoutes{
+								Kinds: []gatewayapi.RouteGroupKind{{
+									Group: &group,
+									Kind:  "HTTPRoute",
+								}},
+							},
+						}},
+					},
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin1",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin2",
+						Namespace: "default",
+					},
+					PluginName: "bar",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin3",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+			},
+			valid:         false,
+			validationMsg: "HTTPRoute has invalid KongPlugin annotation: cannot attach multiple plugins: plugin1, plugin3 of the same type foo",
+		},
+		{
+			msg: "valid HTTPRoute with multiple plugins of different types and one no-existing attached - passes validation",
+			route: &gatewayapi.HTTPRoute{
+				ObjectMeta: metav1.ObjectMeta{
+					Namespace: corev1.NamespaceDefault,
+					Name:      "testing-httproute",
+					Annotations: map[string]string{
+						"konghq.com/plugins": "plugin1,plugin2,plugin3",
+					},
+				},
+				Spec: gatewayapi.HTTPRouteSpec{
+					CommonRouteSpec: gatewayapi.CommonRouteSpec{
+						ParentRefs: []gatewayapi.ParentReference{{
+							Name: "testing-gateway",
+						}},
+					},
+				},
+			},
+			cachedObjects: []client.Object{
+				gatewayClass,
+				&gatewayapi.Gateway{
+					ObjectMeta: metav1.ObjectMeta{
+						Namespace: corev1.NamespaceDefault,
+						Name:      "testing-gateway",
+					},
+					Spec: gatewayapi.GatewaySpec{
+						GatewayClassName: gatewayClassName,
+						Listeners: []gatewayapi.Listener{{
+							Name:     "http",
+							Port:     80,
+							Protocol: (gatewayapi.HTTPProtocolType),
+							AllowedRoutes: &gatewayapi.AllowedRoutes{
+								Kinds: []gatewayapi.RouteGroupKind{{
+									Group: &group,
+									Kind:  "HTTPRoute",
+								}},
+							},
+						}},
+					},
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin1",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin2",
+						Namespace: "default",
+					},
+					PluginName: "bar",
+				},
+			},
+			valid: true,
+		},
 		{
 			msg: "if an HTTPRoute is using queryparams matching it fails validation due to only supporting expression router",
 			route: &gatewayapi.HTTPRoute{
diff --git a/internal/admission/validation/ingress/ingress_test.go b/internal/admission/validation/ingress/ingress_test.go
index 27dc1d16d..aee88a2b1 100644
--- a/internal/admission/validation/ingress/ingress_test.go
+++ b/internal/admission/validation/ingress/ingress_test.go
@@ -57,7 +57,7 @@ func TestValidateIngress(t *testing.T) {
 				tt.ingress,
 				logger,
 				fakestore,
-				fake.NewClientBuilder().Build(),
+				fake.NewFakeClient(),
 			)
 			assert.Equal(t, tt.valid, valid, tt.msg)
 			assert.Equal(t, tt.validationMsg, validMsg, tt.msg)
diff --git a/internal/admission/validator_test.go b/internal/admission/validator_test.go
index 6a51a8519..95aac2362 100644
--- a/internal/admission/validator_test.go
+++ b/internal/admission/validator_test.go
@@ -701,13 +701,14 @@ func TestKongHTTPValidator_ValidateConsumerGroup(t *testing.T) {
 		cg kongv1beta1.KongConsumerGroup
 	}
 	tests := []struct {
-		name             string
-		ConsumerGroupSvc kong.AbstractConsumerGroupService
-		InfoSvc          kong.AbstractInfoService
-		args             args
-		wantOK           bool
-		wantMessage      string
-		wantErr          bool
+		name                 string
+		ConsumerGroupSvc     kong.AbstractConsumerGroupService
+		InfoSvc              kong.AbstractInfoService
+		ManagerClientObjects []client.Object
+		args                 args
+		wantOK               bool
+		wantMessage          string
+		wantErr              bool
 	}{
 		{
 			name:             "Enterprise version",
@@ -720,6 +721,79 @@ func TestKongHTTPValidator_ValidateConsumerGroup(t *testing.T) {
 			wantMessage: "",
 			wantErr:     false,
 		},
+		{
+			name:             "Enterprise Kong Gateway and KongConsumerGroup with multiple plugins of the same type attached",
+			ConsumerGroupSvc: &fakeConsumerGroupSvc{err: nil},
+			InfoSvc:          &fakeInfoSvc{version: "3.4.1.0"},
+			ManagerClientObjects: []client.Object{
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin1",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin2",
+						Namespace: "default",
+					},
+					PluginName: "bar",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin3",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+			},
+			args: args{
+				cg: kongv1beta1.KongConsumerGroup{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{
+							annotations.AnnotationPrefix + annotations.PluginsKey: "plugin1,plugin2,plugin3",
+						},
+					},
+				},
+			},
+			wantOK:      false,
+			wantMessage: "KongConsumerGroup has invalid KongPlugin annotation: cannot attach multiple plugins: plugin1, plugin3 of the same type foo",
+			wantErr:     false,
+		},
+		{
+			name:             "Enterprise Kong Gateway and KongConsumerGroup with plugins of the different types and non-existing one attached",
+			ConsumerGroupSvc: &fakeConsumerGroupSvc{err: nil},
+			InfoSvc:          &fakeInfoSvc{version: "3.4.1.0"},
+			ManagerClientObjects: []client.Object{
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin1",
+						Namespace: "default",
+					},
+					PluginName: "foo",
+				},
+				&kongv1.KongPlugin{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "plugin2",
+						Namespace: "default",
+					},
+					PluginName: "bar",
+				},
+			},
+			args: args{
+				cg: kongv1beta1.KongConsumerGroup{
+					ObjectMeta: metav1.ObjectMeta{
+						Annotations: map[string]string{
+							annotations.AnnotationPrefix + annotations.PluginsKey: "plugin1,plugin2,plugin3",
+						},
+					},
+				},
+			},
+			wantOK:      true,
+			wantMessage: "",
+			wantErr:     false,
+		},
 		{
 			name:             "OSS version",
 			ConsumerGroupSvc: &fakeConsumerGroupSvc{err: nil},
@@ -787,10 +861,14 @@ func TestKongHTTPValidator_ValidateConsumerGroup(t *testing.T) {
 			wantErr:     false,
 		},
 	}
+
+	scheme := runtime.NewScheme()
+	require.NoError(t, kongv1.AddToScheme(scheme))
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
 			validator := KongHTTPValidator{
-				SecretGetter: store,
+				ManagerClient: fake.NewClientBuilder().WithScheme(scheme).WithObjects(tt.ManagerClientObjects...).Build(),
+				SecretGetter:  store,
 				AdminAPIServicesProvider: fakeServicesProvider{
 					infoSvc:          tt.InfoSvc,
 					consumerGroupSvc: tt.ConsumerGroupSvc,
diff --git a/internal/dataplane/kong_client_test.go b/internal/dataplane/kong_client_test.go
index 391616d3b..de1fda47f 100644
--- a/internal/dataplane/kong_client_test.go
+++ b/internal/dataplane/kong_client_test.go
@@ -427,6 +427,7 @@ func (m *mockConfigStatusQueue) Notifications() []clients.ConfigStatus {
 type mockKongConfigBuilder struct {
 	translationFailuresToReturn []failures.ResourceFailure
 	kongState                   *kongstate.KongState
+	updateCacheCalled           bool
 }
 
 func newMockKongConfigBuilder() *mockKongConfigBuilder {
@@ -442,6 +443,14 @@ func (p *mockKongConfigBuilder) BuildKongConfig() translator.KongConfigBuildingR
 	}
 }
 
+func (p *mockKongConfigBuilder) UpdateCache(store.CacheStores) {
+	p.updateCacheCalled = true
+}
+
+func (p *mockKongConfigBuilder) IngressClassName() string {
+	return "kong"
+}
+
 func (p *mockKongConfigBuilder) returnTranslationFailures(enabled bool) {
 	if enabled {
 		// Return some mocked translation failures.
@@ -928,3 +937,34 @@ func TestKongClientUpdate_KonnectUpdatesAreSanitized(t *testing.T) {
 	require.NotNil(t, cert.Key, "expected Konnect to have certificate key")
 	require.Equal(t, "{vault://redacted-value}", *cert.Key, "expected Konnect to have redacted certificate key")
 }
+
+func TestKongClient_FallbackConfiguration(t *testing.T) {
+	ctx := context.Background()
+	clientsProvider := mockGatewayClientsProvider{
+		gatewayClients: []*adminapi.Client{mustSampleGatewayClient(t)},
+	}
+	updateStrategyResolver := newMockUpdateStrategyResolver(t)
+	configChangeDetector := mockConfigurationChangeDetector{hasConfigurationChanged: true}
+	configBuilder := newMockKongConfigBuilder()
+	kongRawStateGetter := &mockKongLastValidConfigFetcher{}
+	kongClient, err := NewKongClient(
+		zapr.NewLogger(zap.NewNop()),
+		time.Second,
+		util.ConfigDumpDiagnostic{},
+		sendconfig.Config{
+			FallbackConfiguration: true,
+		},
+		mocks.NewEventRecorder(),
+		dpconf.DBModeOff,
+		clientsProvider,
+		updateStrategyResolver,
+		configChangeDetector,
+		kongRawStateGetter,
+		configBuilder,
+		store.NewCacheStores(),
+	)
+	require.NoError(t, err)
+
+	require.NoError(t, kongClient.Update(ctx))
+	require.True(t, configBuilder.updateCacheCalled, "expected store to be updated with a snapshot")
+}
diff --git a/internal/dataplane/kongstate/consumer_test.go b/internal/dataplane/kongstate/consumer_test.go
index 644e79bcc..8933b9de3 100644
--- a/internal/dataplane/kongstate/consumer_test.go
+++ b/internal/dataplane/kongstate/consumer_test.go
@@ -7,6 +7,7 @@ import (
 	"github.com/stretchr/testify/assert"
 
 	kongv1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1"
+	"github.com/kong/kubernetes-ingress-controller/v3/test/mocks"
 )
 
 func int64Ptr(i int64) *int64 {
@@ -50,7 +51,7 @@ func TestConsumer_SanitizedCopy(t *testing.T) {
 					Tags:      []*string{kong.String("5.1"), kong.String("5.2")},
 				},
 				Plugins:    []kong.Plugin{{ID: kong.String("1")}},
-				KeyAuths:   []*KeyAuth{{kong.KeyAuth{ID: kong.String("1"), Key: redactedString}}},
+				KeyAuths:   []*KeyAuth{{kong.KeyAuth{ID: kong.String("1"), Key: kong.String("{vault://52fdfc07-2182-454f-963f-5f0f9a621d72}")}}},
 				HMACAuths:  []*HMACAuth{{kong.HMACAuth{ID: kong.String("1"), Secret: redactedString}}},
 				JWTAuths:   []*JWTAuth{{kong.JWTAuth{ID: kong.String("1"), Secret: redactedString}}},
 				BasicAuths: []*BasicAuth{{kong.BasicAuth{ID: kong.String("1"), Password: redactedString}}},
@@ -64,7 +65,7 @@ func TestConsumer_SanitizedCopy(t *testing.T) {
 		},
 	} {
 		t.Run(tt.name, func(t *testing.T) {
-			got := *tt.in.SanitizedCopy()
+			got := *tt.in.SanitizedCopy(mocks.StaticUUIDGenerator{UUID: "52fdfc07-2182-454f-963f-5f0f9a621d72"})
 			assert.Equal(t, tt.want, got)
 		})
 	}
diff --git a/internal/dataplane/kongstate/credentials_test.go b/internal/dataplane/kongstate/credentials_test.go
index 23d7bf27d..2c090a74e 100644
--- a/internal/dataplane/kongstate/credentials_test.go
+++ b/internal/dataplane/kongstate/credentials_test.go
@@ -5,6 +5,8 @@ import (
 
 	"github.com/kong/go-kong/kong"
 	"github.com/stretchr/testify/assert"
+
+	"github.com/kong/kubernetes-ingress-controller/v3/test/mocks"
 )
 
 func TestKeyAuth_SanitizedCopy(t *testing.T) {
@@ -25,13 +27,13 @@ func TestKeyAuth_SanitizedCopy(t *testing.T) {
 			want: KeyAuth{kong.KeyAuth{
 				CreatedAt: kong.Int(1),
 				ID:        kong.String("2"),
-				Key:       redactedString,
+				Key:       kong.String("{vault://52fdfc07-2182-454f-963f-5f0f9a621d72}"),
 				Tags:      []*string{kong.String("4.1"), kong.String("4.2")},
 			}},
 		},
 	} {
 		t.Run(tt.name, func(t *testing.T) {
-			got := *tt.in.SanitizedCopy()
+			got := *tt.in.SanitizedCopy(mocks.StaticUUIDGenerator{UUID: "52fdfc07-2182-454f-963f-5f0f9a621d72"})
 			assert.Equal(t, tt.want, got)
 		})
 	}
diff --git a/internal/dataplane/kongstate/kongstate_test.go b/internal/dataplane/kongstate/kongstate_test.go
index 522707a54..e87399b3b 100644
--- a/internal/dataplane/kongstate/kongstate_test.go
+++ b/internal/dataplane/kongstate/kongstate_test.go
@@ -31,6 +31,7 @@ import (
 	kongv1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1"
 	kongv1alpha1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1alpha1"
 	kongv1beta1 "github.com/kong/kubernetes-ingress-controller/v3/pkg/apis/configuration/v1beta1"
+	"github.com/kong/kubernetes-ingress-controller/v3/test/mocks"
 )
 
 var kongConsumerTypeMeta = metav1.TypeMeta{
@@ -86,7 +87,7 @@ func TestKongState_SanitizedCopy(t *testing.T) {
 					Plugin:              kong.Plugin{ID: kong.String("1"), Config: kong.Configuration{"secret": *redactedString}},
 				}},
 				Consumers: []Consumer{{
-					KeyAuths: []*KeyAuth{{kong.KeyAuth{ID: kong.String("1"), Key: redactedString}}},
+					KeyAuths: []*KeyAuth{{kong.KeyAuth{ID: kong.String("1"), Key: kong.String("{vault://52fdfc07-2182-454f-963f-5f0f9a621d72}")}}},
 				}},
 				Licenses: []License{{kong.License{ID: kong.String("1"), Payload: redactedString}}},
 				ConsumerGroups: []ConsumerGroup{{
@@ -104,7 +105,7 @@ func TestKongState_SanitizedCopy(t *testing.T) {
 	} {
 		t.Run(tt.name, func(t *testing.T) {
 			testedFields.Insert(extractNotEmptyFieldNames(tt.in)...)
-			got := *tt.in.SanitizedCopy()
+			got := *tt.in.SanitizedCopy(mocks.StaticUUIDGenerator{UUID: "52fdfc07-2182-454f-963f-5f0f9a621d72"})
 			assert.Equal(t, tt.want, got)
 		})
 	}
diff --git a/internal/dataplane/translator/subtranslator/httproute_atc_test.go b/internal/dataplane/translator/subtranslator/httproute_atc_test.go
index be070b7c4..401cb42ef 100644
--- a/internal/dataplane/translator/subtranslator/httproute_atc_test.go
+++ b/internal/dataplane/translator/subtranslator/httproute_atc_test.go
@@ -138,8 +138,8 @@ func TestGenerateKongExpressionRoutesFromHTTPRouteMatches(t *testing.T) {
 						{
 							Name: kong.String("response-transformer"),
 							Config: kong.Configuration{
-								"add": map[string][]string{
-									"headers": {`Location: http://a.foo.com:80/exact/0`},
+								"add": TransformerPluginConfig{
+									Headers: []string{`Location: http://a.foo.com:80/exact/0`},
 								},
 							},
 						},
@@ -164,8 +164,8 @@ func TestGenerateKongExpressionRoutesFromHTTPRouteMatches(t *testing.T) {
 						{
 							Name: kong.String("response-transformer"),
 							Config: kong.Configuration{
-								"add": map[string][]string{
-									"headers": {`Location: http://a.foo.com:80/exact/1`},
+								"add": TransformerPluginConfig{
+									Headers: []string{`Location: http://a.foo.com:80/exact/1`},
 								},
 							},
 						},
@@ -200,8 +200,8 @@ func TestGenerateKongExpressionRoutesFromHTTPRouteMatches(t *testing.T) {
 						{
 							Name: kong.String("request-transformer"),
 							Config: kong.Configuration{
-								"append": map[string][]string{
-									"headers": {"foo:bar"},
+								"append": TransformerPluginConfig{
+									Headers: []string{"foo:bar"},
 								},
 							},
 						},
diff --git a/internal/dataplane/translator/subtranslator/httproute_test.go b/internal/dataplane/translator/subtranslator/httproute_test.go
index d47288cbc..8e270f3ec 100644
--- a/internal/dataplane/translator/subtranslator/httproute_test.go
+++ b/internal/dataplane/translator/subtranslator/httproute_test.go
@@ -2,7 +2,6 @@ package subtranslator
 
 import (
 	"errors"
-	"fmt"
 	"testing"
 
 	"github.com/kong/go-kong/kong"
@@ -54,23 +53,23 @@ func TestGeneratePluginsFromHTTPRouteFilters(t *testing.T) {
 				{
 					Name: kong.String("request-transformer"),
 					Config: kong.Configuration{
-						"add": map[string][]string{
-							"headers": {
+						"add": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
-						"append": map[string][]string{
-							"headers": {
+						"append": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-add:foo",
 							},
 						},
-						"remove": map[string][]string{
-							"headers": {
+						"remove": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-remove",
 							},
 						},
-						"replace": map[string][]string{
-							"headers": {
+						"replace": TransformerPluginReplaceConfig{
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
@@ -100,8 +99,8 @@ func TestGeneratePluginsFromHTTPRouteFilters(t *testing.T) {
 				{
 					Name: kong.String("response-transformer"),
 					Config: kong.Configuration{
-						"add": map[string][]string{
-							"headers": {
+						"add": TransformerPluginConfig{
+							Headers: []string{
 								"Location: http://example.org:80/test",
 							},
 						},
@@ -135,23 +134,23 @@ func TestGeneratePluginsFromHTTPRouteFilters(t *testing.T) {
 				{
 					Name: kong.String("response-transformer"),
 					Config: kong.Configuration{
-						"add": map[string][]string{
-							"headers": {
+						"add": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
-						"append": map[string][]string{
-							"headers": {
+						"append": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-add:foo",
 							},
 						},
-						"remove": map[string][]string{
-							"headers": {
+						"remove": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-remove",
 							},
 						},
-						"replace": map[string][]string{
-							"headers": {
+						"replace": TransformerPluginReplaceConfig{
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
@@ -186,7 +185,7 @@ func TestGeneratePluginsFromHTTPRouteFilters(t *testing.T) {
 					},
 				},
 			},
-			expectedPlugins: []kong.Plugin{},
+			expectedPlugins: nil,
 		},
 		{
 			name: "invalid extensionrefs filter group",
@@ -245,14 +244,14 @@ func TestGeneratePluginsFromHTTPRouteFilters(t *testing.T) {
 				{
 					Name: kong.String("request-transformer"),
 					Config: kong.Configuration{
-						"add": map[string]interface{}{
-							"headers": []interface{}{
+						"add": TransformerPluginConfig{
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
-						"replace": map[string]interface{}{
-							"uri": "/new$(uri_captures[1])",
-							"headers": []interface{}{
+						"replace": TransformerPluginReplaceConfig{
+							URI: "/new$(uri_captures[1])",
+							Headers: []string{
 								"header-to-set:bar",
 							},
 						},
@@ -292,7 +291,7 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 		modifier                      *gatewayapi.HTTPURLRewriteFilter
 		firstMatchPath                string
 		expectedKongRouteModification kongstate.Route
-		expected                      kong.Plugin
+		expected                      []transformerPlugin
 		expectedErr                   error
 	}{
 		{
@@ -303,14 +302,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 					ReplaceFullPath: lo.ToPtr("/new-path"),
 				},
 			},
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": "/new-path",
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: "/new-path",
 				},
-			},
+			}},
 			expectedErr: nil,
 		},
 		{
@@ -322,14 +319,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/prefix",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": "/new$(uri_captures[1])",
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: "/new$(uri_captures[1])",
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -348,14 +343,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/prefix",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `$(uri_captures[1] == nil and "/" or uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `$(uri_captures[1] == nil and "/" or uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -374,14 +367,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `/prefix$(uri_captures[1] == nil and "" or "/" .. uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `/prefix$(uri_captures[1] == nil and "" or "/" .. uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -400,14 +391,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/prefix",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `$(uri_captures[1] == nil and "/" or uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `$(uri_captures[1] == nil and "/" or uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -426,14 +415,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `$(uri_captures[1] == nil and "/" or "/" .. uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `$(uri_captures[1] == nil and "/" or "/" .. uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -452,14 +439,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `/new-prefix$(uri_captures[1] == nil and "" or "/" .. uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `/new-prefix$(uri_captures[1] == nil and "" or "/" .. uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -478,14 +463,12 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 			firstMatchPath: "/prefix/",
-			expected: kong.Plugin{
-				Name: lo.ToPtr("request-transformer"),
-				Config: kong.Configuration{
-					"replace": map[string]string{
-						"uri": `/new-prefix$(uri_captures[1])`,
-					},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					URI: `/new-prefix$(uri_captures[1])`,
 				},
-			},
+			}},
 			expectedKongRouteModification: kongstate.Route{
 				Route: kong.Route{
 					Paths: []*string{
@@ -495,31 +478,82 @@ func TestGenerateRequestTransformerForURLRewrite(t *testing.T) {
 				},
 			},
 		},
-		// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/3685
 		{
-			name: "valid URLRewriteFilter with unsupported",
+			name: "URLRewriteFilter with hostname",
+			modifier: &gatewayapi.HTTPURLRewriteFilter{
+				Hostname: lo.ToPtr(gatewayapi.PreciseHostname("replaced.host")),
+			},
+			expected: []transformerPlugin{{
+				Type: transformerPluginTypeRequest,
+				Replace: TransformerPluginReplaceConfig{
+					Headers: []string{
+						"host:replaced.host",
+					},
+				},
+				Add: TransformerPluginConfig{
+					Headers: []string{
+						"host:replaced.host",
+					},
+				},
+			}},
+		},
+		{
+			name: "URLRewriteFilter with hostname and path",
 			modifier: &gatewayapi.HTTPURLRewriteFilter{
-				Hostname: lo.ToPtr(gatewayapi.PreciseHostname("hostname")),
+				Path: &gatewayapi.HTTPPathModifier{
+					Type:               gatewayapi.PrefixMatchHTTPPathModifier,
+					ReplacePrefixMatch: lo.ToPtr("/new-prefix"),
+				},
+				Hostname: lo.ToPtr(gatewayapi.PreciseHostname("replaced.host")),
+			},
+			firstMatchPath: "/prefix",
+			expected: []transformerPlugin{
+				{
+					Type: transformerPluginTypeRequest,
+
+					Replace: TransformerPluginReplaceConfig{
+						URI: `/new-prefix$(uri_captures[1])`,
+					},
+				},
+				{
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						Headers: []string{
+							"host:replaced.host",
+						},
+					},
+					Add: TransformerPluginConfig{
+						Headers: []string{
+							"host:replaced.host",
+						},
+					},
+				},
+			},
+			expectedKongRouteModification: kongstate.Route{
+				Route: kong.Route{
+					Paths: []*string{
+						lo.ToPtr("~/prefix$"),
+						lo.ToPtr("~/prefix(/.*)"),
+					},
+				},
 			},
-			expected:    kong.Plugin{},
-			expectedErr: fmt.Errorf("unsupported hostname replace for %s", gatewayapi.HTTPRouteFilterURLRewrite),
 		},
 		{
 			name:        "nil URLRewriteFilter",
 			modifier:    nil,
-			expected:    kong.Plugin{},
+			expected:    nil,
 			expectedErr: errors.New("URLRewrite is not provided"),
 		},
 	}
 
 	for _, tc := range testCases {
 		t.Run(tc.name, func(t *testing.T) {
-			plugin, routeModifier, err := generateRequestTransformerForURLRewrite(tc.modifier, tc.firstMatchPath, false)
+			plugins, routeModifiers, err := generateRequestTransformerForURLRewrite(tc.modifier, tc.firstMatchPath, false)
 			require.Equal(t, tc.expectedErr, err)
-			require.Equal(t, tc.expected, plugin)
+			require.Equal(t, tc.expected, plugins)
 
 			route := kongstate.Route{}
-			if routeModifier != nil {
+			for _, routeModifier := range routeModifiers {
 				routeModifier(&route)
 			}
 			require.Equal(t, tc.expectedKongRouteModification, route)
@@ -566,124 +600,151 @@ func TestGenerateKongRouteModifierForURLRewritePrefixMatch_ExpressionsRouter(t *
 func TestMergePluginsOfTheSameType(t *testing.T) {
 	testCases := []struct {
 		name     string
-		plugins  []kong.Plugin
-		expected []kong.Plugin
+		plugins  []transformerPlugin
+		expected []transformerPlugin
 	}{
 		{
 			name:     "no plugins",
-			plugins:  []kong.Plugin{},
-			expected: []kong.Plugin{},
+			plugins:  []transformerPlugin{},
+			expected: []transformerPlugin{},
 		},
 		{
 			name: "single plugin",
-			plugins: []kong.Plugin{
+			plugins: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 			},
-			expected: []kong.Plugin{
+			expected: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 			},
 		},
 		{
 			name: "multiple plugins of different types",
-			plugins: []kong.Plugin{
+			plugins: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 				{
-					Name: lo.ToPtr("plugin2"),
-				},
-				{
-					Name: lo.ToPtr("plugin3"),
+					Type: transformerPluginTypeResponse,
 				},
 			},
-			expected: []kong.Plugin{
+			expected: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 				{
-					Name: lo.ToPtr("plugin2"),
-				},
-				{
-					Name: lo.ToPtr("plugin3"),
+					Type: transformerPluginTypeResponse,
 				},
 			},
 		},
 		{
 			name: "multiple plugins of the same types",
-			plugins: []kong.Plugin{
+			plugins: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeResponse,
 				},
 				{
-					Name: lo.ToPtr("plugin2"),
+					Type: transformerPluginTypeRequest,
 				},
 				{
-					Name: lo.ToPtr("plugin2"),
+					Type: transformerPluginTypeResponse,
 				},
 			},
-			expected: []kong.Plugin{
+			expected: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
+					Type: transformerPluginTypeRequest,
 				},
 				{
-					Name: lo.ToPtr("plugin2"),
+					Type: transformerPluginTypeResponse,
 				},
 			},
 		},
 		{
 			name: "multiple plugins of the same types with different configurations - configuration is merged",
-			plugins: []kong.Plugin{
+			plugins: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key1": "value1",
+					Type: transformerPluginTypeRequest,
+					Add: TransformerPluginConfig{
+						Headers: []string{"header1:value1"},
+					},
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path1",
 					},
 				},
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key2": "value2",
+					Type: transformerPluginTypeRequest,
+					Add: TransformerPluginConfig{
+						Headers: []string{"header2:value2"},
+					},
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path2",
 					},
 				},
 			},
-			expected: []kong.Plugin{
+			expected: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key1": "value1",
-						"key2": "value2",
+					Type: transformerPluginTypeRequest,
+					Add: TransformerPluginConfig{
+						Headers: []string{"header1:value1", "header2:value2"},
+					},
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path1",
 					},
 				},
 			},
 		},
 		{
 			name: "multiple plugins of the same types with same configuration keys - configuration is merged and the first wins",
-			plugins: []kong.Plugin{
+			plugins: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key1": "value1",
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path1",
 					},
 				},
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key1": "value2",
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path2",
 					},
 				},
 			},
-			expected: []kong.Plugin{
+			expected: []transformerPlugin{
 				{
-					Name: lo.ToPtr("plugin1"),
-					Config: kong.Configuration{
-						"key1": "value1",
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path1",
+					},
+				},
+			},
+		},
+		{
+			name: "multiple plugins of the same types with same configuration keys, first URI empty",
+			plugins: []transformerPlugin{
+				{
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "",
+					},
+				},
+				{
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path2",
+					},
+				},
+			},
+			expected: []transformerPlugin{
+				{
+					Type: transformerPluginTypeRequest,
+					Replace: TransformerPluginReplaceConfig{
+						URI: "path2",
 					},
 				},
 			},
diff --git a/internal/dataplane/translator/translate_httproute_test.go b/internal/dataplane/translator/translate_httproute_test.go
index b4d359fa5..02ead543c 100644
--- a/internal/dataplane/translator/translate_httproute_test.go
+++ b/internal/dataplane/translator/translate_httproute_test.go
@@ -1055,8 +1055,8 @@ func TestIngressRulesFromHTTPRoutes(t *testing.T) {
 										{
 											Name: kong.String("request-transformer"),
 											Config: kong.Configuration{
-												"append": map[string][]string{
-													"headers": {"X-Test-Header-1:test-value-1"},
+												"append": subtranslator.TransformerPluginConfig{
+													Headers: []string{"X-Test-Header-1:test-value-1"},
 												},
 											},
 										},
@@ -1088,8 +1088,8 @@ func TestIngressRulesFromHTTPRoutes(t *testing.T) {
 										{
 											Name: kong.String("request-transformer"),
 											Config: kong.Configuration{
-												"append": map[string][]string{
-													"headers": {"X-Test-Header-2:test-value-2"},
+												"append": subtranslator.TransformerPluginConfig{
+													Headers: []string{"X-Test-Header-2:test-value-2"},
 												},
 											},
 										},
@@ -1465,8 +1465,8 @@ func TestIngressRulesFromHTTPRoutes(t *testing.T) {
 										{
 											Name: kong.String("request-transformer"),
 											Config: kong.Configuration{
-												"append": map[string][]string{
-													"headers": {"X-Test-Header-1:test-value-1"},
+												"append": subtranslator.TransformerPluginConfig{
+													Headers: []string{"X-Test-Header-1:test-value-1"},
 												},
 											},
 										},
@@ -2313,8 +2313,8 @@ func TestIngressRulesFromSplitHTTPRouteMatchWithPriority(t *testing.T) {
 					{
 						Name: kong.String("response-transformer"),
 						Config: kong.Configuration{
-							"add": map[string][]string{
-								"headers": {"Location: http://bar.com:80/v1/foo"},
+							"add": subtranslator.TransformerPluginConfig{
+								Headers: []string{"Location: http://bar.com:80/v1/foo"},
 							},
 						},
 					},
diff --git a/internal/dataplane/translator/translator_test.go b/internal/dataplane/translator/translator_test.go
index 7676913eb..03f6171d3 100644
--- a/internal/dataplane/translator/translator_test.go
+++ b/internal/dataplane/translator/translator_test.go
@@ -5020,14 +5020,12 @@ func TestTranslator_ConfiguredKubernetesObjects(t *testing.T) {
 }
 
 func mustNewTranslator(t *testing.T, storer store.Storer) *Translator {
-	p, err := NewTranslator(zapr.NewLogger(zap.NewNop()), storer, "",
-		FeatureFlags{
-			// We'll assume these are true for all tests.
-			FillIDs:                           true,
-			ReportConfiguredKubernetesObjects: true,
-			KongServiceFacade:                 true,
-		},
-	)
+	p, err := NewTranslator(zapr.NewLogger(zap.NewNop()), storer, "", FeatureFlags{
+		// We'll assume these are true for all tests.
+		FillIDs:                           true,
+		ReportConfiguredKubernetesObjects: true,
+		KongServiceFacade:                 true,
+	})
 	require.NoError(t, err)
 	return p
 }
@@ -5164,3 +5162,34 @@ func TestUpdateTargetMap(t *testing.T) {
 		})
 	}
 }
+
+func TestTranslator_UpdateStore(t *testing.T) {
+	originalStore, err := store.NewFakeStore(store.FakeObjects{})
+	require.NoError(t, err)
+	translator := mustNewTranslator(t, originalStore)
+
+	originalBuildConfigResult := translator.BuildKongConfig()
+
+	newStore, err := store.NewCacheStoresFromObjs(
+		&kongv1.KongConsumer{
+			TypeMeta: metav1.TypeMeta{
+				Kind:       "KongConsumer",
+				APIVersion: kongv1.GroupVersion.String(),
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "consumer1",
+				Namespace: "bar",
+				Annotations: map[string]string{
+					annotations.IngressClassKey: annotations.DefaultIngressClass,
+				},
+			},
+			Username: "consumer1",
+		},
+	)
+	require.NoError(t, err)
+	translator.UpdateCache(newStore)
+
+	newBuildConfigResult := translator.BuildKongConfig()
+	require.NotEqual(t, originalBuildConfigResult.KongState, newBuildConfigResult.KongState, "KongState should be different after updating the store")
+	require.Len(t, newBuildConfigResult.KongState.Consumers, 1, "expected 1 consumer in the KongState")
+}
diff --git a/internal/manager/setup_test.go b/internal/manager/setup_test.go
index 93a42c973..f5a320b6b 100644
--- a/internal/manager/setup_test.go
+++ b/internal/manager/setup_test.go
@@ -22,7 +22,7 @@ import (
 func TestAdminAPIClientFromServiceDiscovery(t *testing.T) {
 	log := logr.Discard()
 	adminAPISvcNN := k8stypes.NamespacedName{Name: "admin-api", Namespace: "kong"}
-	kubeClient := fake.NewClientBuilder().Build()
+	kubeClient := fake.NewFakeClient()
 	genericErr := errors.New("some generic error")
 	someDiscoveredAPI := func(address string) adminapi.DiscoveredAdminAPI {
 		return adminapi.DiscoveredAdminAPI{
diff --git a/internal/store/cache_stores_snapshot_test.go b/internal/store/cache_stores_snapshot_test.go
new file mode 100644
index 000000000..df8a533cd
--- /dev/null
+++ b/internal/store/cache_stores_snapshot_test.go
@@ -0,0 +1,88 @@
+package store_test
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	netv1 "k8s.io/api/networking/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+
+	"github.com/kong/kubernetes-ingress-controller/v3/internal/store"
+)
+
+func TestCacheStores_TakeSnapshot(t *testing.T) {
+	originalStores, err := store.NewCacheStoresFromObjs([]runtime.Object{
+		&netv1.Ingress{
+			TypeMeta: metav1.TypeMeta{
+				APIVersion: "networking.k8s.io/v1",
+				Kind:       "Ingress",
+			},
+			ObjectMeta: metav1.ObjectMeta{
+				Name:      "foo",
+				Namespace: "default",
+				Annotations: map[string]string{
+					"foo": "bar",
+				},
+			},
+		},
+	}...)
+	require.NoError(t, err)
+
+	t.Log("Taking a snapshot of the originalStores")
+	snapshot, err := originalStores.TakeSnapshot()
+	require.NoError(t, err)
+
+	t.Log("Adding a new object to the original store")
+	err = originalStores.Add(&netv1.Ingress{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "networking.k8s.io/v1",
+			Kind:       "Ingress",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "bar",
+			Namespace: "default",
+			Annotations: map[string]string{
+				"foo": "baz",
+			},
+		},
+	})
+	require.NoError(t, err)
+
+	// We'll use ingressMeta in .Get() calls.
+	ingressMeta := &netv1.Ingress{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "networking.k8s.io/v1",
+			Kind:       "Ingress",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "foo",
+			Namespace: "default",
+		},
+	}
+
+	t.Log("Modifying the original object")
+	obj, ok, err := originalStores.IngressV1.Get(ingressMeta)
+	require.NoError(t, err)
+	require.True(t, ok)
+	ingressFromOriginalStore, ok := obj.(*netv1.Ingress)
+	require.True(t, ok)
+	require.Equal(t, "bar", ingressFromOriginalStore.Annotations["foo"])
+	ingressFromOriginalStore.Annotations["foo"] = "qux"
+
+	t.Log("Checking that the original store returns the modified object")
+	obj, ok, err = originalStores.IngressV1.Get(ingressMeta)
+	require.NoError(t, err)
+	require.True(t, ok)
+	ingressFromOriginalStore2, ok := obj.(*netv1.Ingress)
+	require.True(t, ok)
+	require.Same(t, ingressFromOriginalStore, ingressFromOriginalStore2, "Store should return pointers the same object")
+
+	t.Log("Checking that the snapshot returns the unmodified object")
+	obj, ok, err = snapshot.IngressV1.Get(obj)
+	require.NoError(t, err)
+	require.True(t, ok)
+	ingressFromSnapshot, ok := obj.(*netv1.Ingress)
+	require.True(t, ok)
+	require.Equal(t, "bar", ingressFromSnapshot.Annotations["foo"], "Snapshot should not be affected by the change in the original store")
+}
diff --git a/internal/store/store_test.go b/internal/store/store_test.go
index 791bcdc9c..57abfce3f 100644
--- a/internal/store/store_test.go
+++ b/internal/store/store_test.go
@@ -147,7 +147,7 @@ func TestGetIngressClassHandling(t *testing.T) {
 		t.Run(tt.name, func(t *testing.T) {
 			s, err := NewFakeStore(tt.objs)
 			require.NoError(t, err)
-			if got := s.(Store).getIngressClassHandling(); got != tt.want {
+			if got := s.(*Store).getIngressClassHandling(); got != tt.want {
 				t.Errorf("s.getIngressClassHandling() = %v, want %v", got, tt.want)
 			}
 		})
diff --git a/test/conformance/gateway_conformance_test.go b/test/conformance/gateway_conformance_test.go
index c5f47acc3..24e5ed3b0 100644
--- a/test/conformance/gateway_conformance_test.go
+++ b/test/conformance/gateway_conformance_test.go
@@ -32,6 +32,7 @@ var traditionalRoutesSupportedFeatures = []suite.SupportedFeature{
 	// extended features
 	suite.SupportHTTPRouteResponseHeaderModification,
 	suite.SupportHTTPRoutePathRewrite,
+	suite.SupportHTTPRouteHostRewrite,
 	// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/5868
 	// Temporarily disabled and tracking through the following issue.
 	// suite.SupportHTTPRouteBackendTimeout,
@@ -46,6 +47,7 @@ var expressionRoutesSupportedFeatures = []suite.SupportedFeature{
 	suite.SupportHTTPRouteMethodMatching,
 	suite.SupportHTTPRouteResponseHeaderModification,
 	suite.SupportHTTPRoutePathRewrite,
+	suite.SupportHTTPRouteHostRewrite,
 	// TODO: https://github.com/Kong/kubernetes-ingress-controller/issues/5868
 	// Temporarily disabled and tracking through the following issue.
 	// suite.SupportHTTPRouteBackendTimeout,
diff --git a/test/envtest/telemetry_test.go b/test/envtest/telemetry_test.go
index 7a4a5e334..175249c4d 100644
--- a/test/envtest/telemetry_test.go
+++ b/test/envtest/telemetry_test.go
@@ -356,6 +356,7 @@ func verifyTelemetryReport(t *testing.T, k8sVersion *version.Info, report string
 		"<14>"+
 			"signal=kic-ping;"+
 			"db=off;"+
+			"feature-fallbackconfiguration=false;"+
 			"feature-fillids=true;"+
 			"feature-gateway-service-discovery=false;"+
 			"feature-gatewayalpha=false;"+
diff --git a/test/mocks/uuid.go b/test/mocks/uuid.go
new file mode 100644
index 000000000..f0cd6d375
--- /dev/null
+++ b/test/mocks/uuid.go
@@ -0,0 +1,13 @@
+package mocks
+
+import "github.com/kong/kubernetes-ingress-controller/v3/internal/util"
+
+var _ = util.UUIDGenerator(&StaticUUIDGenerator{})
+
+type StaticUUIDGenerator struct {
+	UUID string
+}
+
+func (s StaticUUIDGenerator) NewString() string {
+	return s.UUID
+}