Repository: envoyproxy/ai-gateway
Pull request #493 (2025-03-14 00:14:00)

feat: initial implementation of Inference Extension 
**Commit Message**

This commit scaffolds the foundation for the Inference Extension API [1]. The design documentation was merged in #492. The controller needs to be started with `--enableInferenceExtension=true` to not break the existing controller deployment where the Inference Extension CRDs are not installed.

This commit doesn't implement the actual "metrics-aware" load balancing and instead it just does the random routing out of given (resolved) endpoints. The follow up implementations will add more advanced algorithm while expanding the metrics interface that currently only provides the setter APIs.

The summary of the implementation is:
* Added `kind` field to AIGatewayRouteRuleBackendRef so that it can reference InferencePool.
* InferencePool.Spec.Selector is allowed to specify multiple AIServiceBackend.
* When building up all the extproc config via filterapi.Config, the controller reads the referenced InferencePool and its binding InferenceModels, and group them together into a single filterapi.DynamicLoadBalancing configuration. 
* When the extproc loads the configuration containing DynamicLoadBalancing, it will resolve all the IP addresses for hostnames belonging to the DyanmicLoadbalancing. The presence of DynamicLoadBalancing in th config forces the config watcher reload and refresh the config regardless of the updates. That way, the list of ip addresses will always be updated (eventual consistency anyways) in a non-hot path.
* On the request path, the ChatCompletionProcessor will check the existence of the DynamicLoadBalancing config for the backend selected by the router. If so, it further tries to resolve the ip:port level endpoint selection.
* The selected ip:port will be set to the special header that will be routed to ORIGINAL_DST.
* ORIGINAL_DST cluster will be added by the EG extension sever implementation. Also, the extension server modifies some routes to properly route to that cluster. 

1: https://github.com/kubernetes-sigs/gateway-api-inference-extension

**Related Issues/PRs (if applicable)**

Built on #492
Contributes to #423


mathetake: cc @yuzisun 
mathetake: obviously not working but i think this shows how this can be implemented here with minimal changes 
mathetake: one big missing piece is to generate a special Envoy cluster that has a new LB policy attached; that can be a global config
mathetake: ok get back to this ......
mathetake: ok the control plane is almost done 
mathetake: monday i will continue backfilling the unit tests for control plane and then extproc part 
mathetake: almos there 
mathetake: Was thinking it should be better to use the EG extension server (currently not in use) but encountered a few issues (not sure who's to blame), switching to use PatchPolicy instead
mathetake: controller 100% done 🎉 
kfswain: I can make some time today to look over this from an IGW perspective, as it sounds like certain sections are ready for review. Unless I should hold off?
mathetake: @kfswain yeah, the left is the implementation details + some tests so you can start reviewing  
mathetake: OK i editted the description and added the summary of the implementation here. 95% done - the left is to add the end-to-end test case (extproc, controller tests were already added respectively).
mathetake: i will mark this ready for ready as soon as the e2e test case is added but ready for review anyways 
mathetake: finally ready for review; all tests passing with comprehensive test coverage 
mathetake: sorry that the PR gets larger than I thought, so please refer to the PR description for the overview before diving into it 
mathetake: One thing to consider; i think it also makes sense to do the dns resolution during the reconciliation and returns Requeue: true when the dns refresh is necessary rather than letting extproc do that. We can do the refactoring in that way later as in anyways that's an implementation detail 
yuzisun: Thanks @mathetake ! I am going to take a look tonight.
yuzisun: Thanks @mathetake ! I am going to take a look tonight.
mathetake: No rush ! 
kfswain: I also should be able to do a review in the next 24h. Thanks for this!
mathetake: cool, i think this should be almost ready!

Patch:
diff --git a/.golangci.yml b/.golangci.yml
index fa47d11..7495b79 100644
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -38,7 +38,6 @@ linters-settings:
     # Do not allow non-required aliases.
     no-extra-aliases: false
     alias:
-      # gateway-api
       - pkg: sigs.k8s.io/gateway-api/apis/v1
         alias: gwapiv1
       - pkg: sigs.k8s.io/gateway-api/apis/v1alpha2
@@ -51,7 +50,8 @@ linters-settings:
         alias: egv1a1
       - pkg: github.com/envoyproxy/ai-gateway/api/v1alpha1
         alias: aigv1a1
-      # kubernetes api
+      - pkg: sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2
+        alias: gwaiev1a2
       - pkg: k8s.io/apimachinery/pkg/apis/meta/v1
         alias: metav1
       - pkg: k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1
@@ -64,6 +64,12 @@ linters-settings:
         alias: apierrors
       - pkg: github.com/envoyproxy/ai-gateway/internal/testing
         alias: internaltesting
+      - pkg: github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3
+        alias: clusterv3
+      - pkg: github.com/envoyproxy/go-control-plane/envoy/config/route/v3
+        alias: routev3
+      - pkg: github.com/envoyproxy/gateway/proto/extension
+        alias: egextension
   gci:
     sections:
       # Captures all standard packages if they do not match another section.
diff --git a/api/v1alpha1/api.go b/api/v1alpha1/api.go
index 348c8e3..d79b268 100644
--- a/api/v1alpha1/api.go
+++ b/api/v1alpha1/api.go
@@ -218,8 +218,30 @@ type AIGatewayRouteRule struct {
 	Matches []AIGatewayRouteRuleMatch `json:"matches,omitempty"`
 }
 
-// AIGatewayRouteRuleBackendRef is a reference to a AIServiceBackend with a weight.
+// AIGatewayRouteRuleBackendRefKind specifies the kind of the backend reference.
+type AIGatewayRouteRuleBackendRefKind string
+
+const (
+	// AIGatewayRouteRuleBackendRefAIServiceBackend is the kind of the AIServiceBackend.
+	AIGatewayRouteRuleBackendRefAIServiceBackend AIGatewayRouteRuleBackendRefKind = "AIServiceBackend"
+	// AIGatewayRouteRuleBackendRefInferencePool is the kind of the InferencePool in the Gateway API Inference Extension.
+	// https://github.com/kubernetes-sigs/gateway-api-inference-extension
+	AIGatewayRouteRuleBackendRefInferencePool AIGatewayRouteRuleBackendRefKind = "InferencePool"
+)
+
+// AIGatewayRouteRuleBackendRef is a reference to a backend with a weight.
 type AIGatewayRouteRuleBackendRef struct {
+	// Kind is the kind of the backend, which is either "AIServiceBackend" or "InferencePool" in Gateway API Inference Extension.
+	//
+	// When this references InferencePool, the selector of the InferencePool is used to select (multiple) AIServiceBackend(s)
+	// that can serve the same model sets that the InferencePool binds.
+	//
+	// Default is AIServiceBackend.
+	//
+	// +kubebuilder:validation:Enum=AIServiceBackend;InferencePool
+	// +kubebuilder:default=AIServiceBackend
+	Kind *AIGatewayRouteRuleBackendRefKind `json:"kind,omitempty"`
+
 	// Name is the name of the AIServiceBackend.
 	//
 	// +kubebuilder:validation:Required
diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go
index f294887..02c6626 100644
--- a/api/v1alpha1/zz_generated.deepcopy.go
+++ b/api/v1alpha1/zz_generated.deepcopy.go
@@ -127,7 +127,9 @@ func (in *AIGatewayRouteRule) DeepCopyInto(out *AIGatewayRouteRule) {
 	if in.BackendRefs != nil {
 		in, out := &in.BackendRefs, &out.BackendRefs
 		*out = make([]AIGatewayRouteRuleBackendRef, len(*in))
-		copy(*out, *in)
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
 	}
 	if in.Matches != nil {
 		in, out := &in.Matches, &out.Matches
@@ -151,6 +153,11 @@ func (in *AIGatewayRouteRule) DeepCopy() *AIGatewayRouteRule {
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *AIGatewayRouteRuleBackendRef) DeepCopyInto(out *AIGatewayRouteRuleBackendRef) {
 	*out = *in
+	if in.Kind != nil {
+		in, out := &in.Kind, &out.Kind
+		*out = new(AIGatewayRouteRuleBackendRefKind)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AIGatewayRouteRuleBackendRef.
diff --git a/cmd/aigw/translate.go b/cmd/aigw/translate.go
index 84baeed..2177bda 100644
--- a/cmd/aigw/translate.go
+++ b/cmd/aigw/translate.go
@@ -183,7 +183,7 @@ func translateCustomResourceObjects(
 		WithStatusSubresource(&aigv1a1.AIGatewayRoute{}).
 		WithStatusSubresource(&aigv1a1.AIServiceBackend{}).
 		WithStatusSubresource(&aigv1a1.BackendSecurityPolicy{})
-	_ = controller.ApplyIndexing(ctx, func(_ context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error {
+	_ = controller.ApplyIndexing(ctx, true, func(_ context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error {
 		builder = builder.WithIndex(obj, field, extractValue)
 		return nil
 	}) // Error should never happen.
diff --git a/cmd/controller/main.go b/cmd/controller/main.go
index 34f8bba..12adb21 100644
--- a/cmd/controller/main.go
+++ b/cmd/controller/main.go
@@ -12,7 +12,7 @@ import (
 	"net"
 	"os"
 
-	"github.com/envoyproxy/gateway/proto/extension"
+	egextension "github.com/envoyproxy/gateway/proto/extension"
 	"go.uber.org/zap/zapcore"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/health/grpc_health_v1"
@@ -31,6 +31,7 @@ func parseAndValidateFlags(args []string) (
 	enableLeaderElection bool,
 	logLevel zapcore.Level,
 	extensionServerPort string,
+	enableInfExt bool,
 	err error,
 ) {
 	fs := flag.NewFlagSet("AI Gateway Controller", flag.ContinueOnError)
@@ -60,6 +61,11 @@ func parseAndValidateFlags(args []string) (
 		":1063",
 		"gRPC port for the extension server",
 	)
+	enableInfExtPtr := fs.Bool(
+		"enableInferenceExtension",
+		false,
+		"Enable the Gateway Inference Extetension. When enabling this, the CRDs for the InferenceModel and InferencePool must be installed prior to starting the controller.",
+	)
 
 	if err = fs.Parse(args); err != nil {
 		err = fmt.Errorf("failed to parse flags: %w", err)
@@ -77,7 +83,7 @@ func parseAndValidateFlags(args []string) (
 		err = fmt.Errorf("invalid log level: %q", *logLevelPtr)
 		return
 	}
-	return *extProcLogLevelPtr, *extProcImagePtr, *enableLeaderElectionPtr, zapLogLevel, *extensionServerPortPtr, nil
+	return *extProcLogLevelPtr, *extProcImagePtr, *enableLeaderElectionPtr, zapLogLevel, *extensionServerPortPtr, *enableInfExtPtr, nil
 }
 
 func main() {
@@ -88,6 +94,7 @@ func main() {
 		flagEnableLeaderElection,
 		zapLogLevel,
 		flagExtensionServerPort,
+		enableInfExt,
 		err := parseAndValidateFlags(os.Args[1:])
 	if err != nil {
 		setupLog.Error(err, "failed to parse and validate flags")
@@ -110,8 +117,8 @@ func main() {
 
 	// Start the extension server running alongside the controller.
 	s := grpc.NewServer()
-	extSrv := extensionserver.New(setupLog)
-	extension.RegisterEnvoyGatewayExtensionServer(s, extSrv)
+	extSrv := extensionserver.New(ctrl.Log)
+	egextension.RegisterEnvoyGatewayExtensionServer(s, extSrv)
 	grpc_health_v1.RegisterHealthServer(s, extSrv)
 	go func() {
 		<-ctx.Done()
@@ -128,6 +135,7 @@ func main() {
 		ExtProcImage:         flagExtProcImage,
 		ExtProcLogLevel:      flagExtProcLogLevel,
 		EnableLeaderElection: flagEnableLeaderElection,
+		EnableInfExt:         enableInfExt,
 	}); err != nil {
 		setupLog.Error(err, "failed to start controller")
 	}
diff --git a/cmd/extproc/mainlib/main.go b/cmd/extproc/mainlib/main.go
index b4d450a..681a9bd 100644
--- a/cmd/extproc/mainlib/main.go
+++ b/cmd/extproc/mainlib/main.go
@@ -188,7 +188,7 @@ func startMetricsServer(addr string, logger *slog.Logger) (*http.Server, metric.
 	}
 
 	go func() {
-		logger.Info("Starting metrics server", "address", addr)
+		logger.Info("starting metrics server", "address", addr)
 		if err := server.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
 			logger.Error("Metrics server failed", "error", err)
 		}
diff --git a/examples/inference_extension/README.md b/examples/inference_extension/README.md
new file mode 100644
index 0000000..8edf02d
--- /dev/null
+++ b/examples/inference_extension/README.md
@@ -0,0 +1,2 @@
+This example demonstrates how to use the [Inference Extension API](https://gateway-api-inference-extension.sigs.k8s.io/)in Envoy AI Gateway project.
+The feature can be used only when `--enableInferenceExtension` is set to `true` passed to the Envoy AI Gateway controller. See the helm values.yaml file for more details.
diff --git a/examples/inference_extension/inference_extension.yaml b/examples/inference_extension/inference_extension.yaml
new file mode 100644
index 0000000..c378e95
--- /dev/null
+++ b/examples/inference_extension/inference_extension.yaml
@@ -0,0 +1,133 @@
+# Copyright Envoy AI Gateway Authors
+# SPDX-License-Identifier: Apache-2.0
+# The full text of the Apache license is available in the LICENSE file at
+# the root of the repo.
+
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+  name: inference-extension-example
+spec:
+  controllerName: gateway.envoyproxy.io/gatewayclass-controller
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+  name: inference-extension-example
+  namespace: default
+spec:
+  gatewayClassName: inference-extension-example
+  listeners:
+    - name: http
+      protocol: HTTP
+      port: 80
+---
+apiVersion: aigateway.envoyproxy.io/v1alpha1
+kind: AIGatewayRoute
+metadata:
+  name: inference-extension-example
+  namespace: default
+spec:
+  schema:
+    name: OpenAI
+  targetRefs:
+    - name: inference-extension-example
+      kind: Gateway
+      group: gateway.networking.k8s.io
+  rules:
+    - matches:
+        - headers:
+            - type: Exact
+              name: x-target-inference-extension
+              value: "yes"
+      backendRefs:
+        - name: inference-extension-example-pool  # The name of the InferencePool that this route will route to.
+          # Explicitly specify the kind of the backend to be InferencePool.
+          kind: InferencePool
+---
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: inference-extension-example-pool
+spec:
+  targetPortNumber: 8080
+  selector:
+    # Select multiple AIServiceBackend objects to bind to the InferencePool.
+    app: my-backend
+  extensionRef:
+    # Specify the static name "envoy-ai-gateway" to bind the InferencePool to the Envoy AI Gateway.
+    # This indicates that the InferencePool will be managed by the Envoy AI Gateway.
+    name: envoy-ai-gateway
+---
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: inference-extension-example
+spec:
+  modelName: mistral:latest
+  criticality: Critical
+  poolRef:
+    # Bind the InferenceModel to the InferencePool.
+    name: inference-extension-example-pool
+---
+apiVersion: aigateway.envoyproxy.io/v1alpha1
+kind: AIServiceBackend
+metadata:
+  name: inference-extension-example-testupstream
+  namespace: default
+  labels:
+    # Indicate the backend is selected by the InferencePool.
+    app: my-backend
+spec:
+  schema:
+    name: OpenAI
+  backendRef:
+    name: inference-extension-example-testupstream
+    kind: Service
+    port: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: inference-extension-example-testupstream
+  namespace: default
+spec:
+  selector:
+    app: inference-extension-example-testupstream
+  ports:
+    - protocol: TCP
+      port: 8080
+      targetPort: 8080
+  # The headless service allows the IP addresses of the pods to be resolved via the Service DNS.
+  clusterIP: None
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: inference-extension-example-testupstream
+  namespace: default
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: inference-extension-example-testupstream
+  template:
+    metadata:
+      labels:
+        app: inference-extension-example-testupstream
+    spec:
+      containers:
+        - name: testupstream
+          image: docker.io/envoyproxy/ai-gateway-testupstream:latest
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 8080
+          env:
+            - name: TESTUPSTREAM_ID
+              value: test
+          readinessProbe:
+            httpGet:
+              path: /health
+              port: 8080
+            initialDelaySeconds: 1
+            periodSeconds: 1
diff --git a/filterapi/filterconfig.go b/filterapi/filterconfig.go
index f76ebe2..65958bc 100644
--- a/filterapi/filterconfig.go
+++ b/filterapi/filterconfig.go
@@ -156,15 +156,55 @@ type RouteRule struct {
 // besides that this abstracts the concept of a backend at Envoy Gateway level to a simple name.
 type Backend struct {
 	// Name of the backend, which is the value in the final routing decision
-	// matching the header key specified in the [Config.BackendRoutingHeaderKey].
+	// matching the header key specified in the [filterapi.Config.BackendRoutingHeaderKey].
 	Name string `json:"name"`
 	// Schema specifies the API schema of the output format of requests from.
 	Schema VersionedAPISchema `json:"schema"`
 	// Weight is the weight of the backend in the routing decision.
+	//
+	// When DynamicLoadBalancing is specified, the weight is ignored.
 	Weight int `json:"weight"`
 	// Auth is the authn/z configuration for the backend. Optional.
-	// TODO: refactor after https://github.com/envoyproxy/ai-gateway/pull/43.
 	Auth *BackendAuth `json:"auth,omitempty"`
+	// DynamicLoadBalancing is the dynamic backend configuration which forces the AI filter to
+	// route the request to a specific endpoint (ip:port) instead of the normal cluster routing.
+	//
+	// When this is specified, the AI Gateway filter assume that the ORIGINAL_DST cluster is configured
+	// to route the request to an endpoint from `x-ai-eg-original-dst` header.
+	DynamicLoadBalancing *DynamicLoadBalancing `json:"dynamicLoadBalancing,omitempty"`
+}
+
+// DynamicLoadBalancing corresponds to InferencePool and InferenceModels belonging to the same pool.
+type DynamicLoadBalancing struct {
+	// Models that can be served by this backend. If not matched, the 404 is returned to the client.
+	//
+	// If multiple models are provided, the request is routed to the backend based on the weights, criticality, etc.
+	Models []DynamicLoadBalancingModel `json:"models,omitempty"`
+	// Backends can be either ip:port or hostname:port.
+	Backends []DynamicLoadBalancingBackend `json:"backends,omitempty"`
+}
+
+// DynamicLoadBalancingModel corresponds to InferenceModel in the Inference Extension.
+type DynamicLoadBalancingModel struct {
+	// Name is the name of the model.
+	Name string `json:"name"`
+	// Weight is the weight of the model in the routing decision when multiple models are provided.
+	Weight *int `json:"weight,omitempty"`
+	// TODO: Criticality?
+}
+
+// DynamicLoadBalancingBackend corresponds to a single AIServiceBackend that is selected by the
+// InferencePool. It is basically a wrapper of Backend with additional information to do
+// the IP address level dynamic load balancing.
+type DynamicLoadBalancingBackend struct {
+	Backend
+	// Hostnames is the hostname of this backend. The filter will resolve the hostname to the IP address
+	// asynchronously and use the resolved IP address to route the request.
+	Hostnames []string `json:"hostNames,omitempty"`
+	// IP is the IP address of the endpoint.
+	IPs []string `json:"ips,omitempty"`
+	// Port is the port of the endpoint.
+	Port int32 `json:"port"`
 }
 
 // BackendAuth corresponds partially to BackendSecurityPolicy in api/v1alpha1/api.go.
diff --git a/go.mod b/go.mod
index 0306f59..dbbc029 100644
--- a/go.mod
+++ b/go.mod
@@ -16,6 +16,7 @@ require (
 	github.com/go-logr/logr v1.4.2
 	github.com/google/cel-go v0.22.1
 	github.com/google/go-cmp v0.7.0
+	github.com/miekg/dns v1.1.63
 	github.com/openai/openai-go v0.1.0-alpha.65
 	github.com/prometheus/client_golang v1.21.1
 	github.com/stretchr/testify v1.10.0
@@ -36,6 +37,7 @@ require (
 	k8s.io/utils v0.0.0-20241210054802-24370beab758
 	sigs.k8s.io/controller-runtime v0.20.4
 	sigs.k8s.io/gateway-api v1.2.1
+	sigs.k8s.io/gateway-api-inference-extension v0.0.0-20250320205432-189f0dcf9aa0
 	sigs.k8s.io/yaml v1.4.0
 )
 
@@ -276,7 +278,6 @@ require (
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mattn/go-runewidth v0.0.16 // indirect
 	github.com/mgechev/revive v1.6.1 // indirect
-	github.com/miekg/dns v1.1.63 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/go-homedir v1.1.0 // indirect
 	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
diff --git a/go.sum b/go.sum
index fa08349..87aeaa1 100644
--- a/go.sum
+++ b/go.sum
@@ -900,8 +900,8 @@ github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108
 github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
 github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
 github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
-github.com/onsi/ginkgo/v2 v2.22.2 h1:/3X8Panh8/WwhU/3Ssa6rCKqPLuAkVY2I0RoyDLySlU=
-github.com/onsi/ginkgo/v2 v2.22.2/go.mod h1:oeMosUL+8LtarXBHu/c0bx2D/K9zyQ6uX3cTyztHwsk=
+github.com/onsi/ginkgo/v2 v2.23.0 h1:FA1xjp8ieYDzlgS5ABTpdUDB7wtngggONc8a7ku2NqQ=
+github.com/onsi/ginkgo/v2 v2.23.0/go.mod h1:zXTP6xIp3U8aVuXN8ENK9IXRaTjFnpVB9mGmaSRvxnM=
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
 github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
@@ -1665,6 +1665,8 @@ sigs.k8s.io/controller-tools v0.17.1 h1:bQ+dKCS7jY9AgpefenBDtm6geJZCHVKbegpLynxg
 sigs.k8s.io/controller-tools v0.17.1/go.mod h1:3QXAdrmdxYuQ4MifvbCAFD9wLXn7jylnfBPYS4yVDdc=
 sigs.k8s.io/gateway-api v1.2.1 h1:fZZ/+RyRb+Y5tGkwxFKuYuSRQHu9dZtbjenblleOLHM=
 sigs.k8s.io/gateway-api v1.2.1/go.mod h1:EpNfEXNjiYfUJypf0eZ0P5iXA9ekSGWaS1WgPaM42X0=
+sigs.k8s.io/gateway-api-inference-extension v0.0.0-20250320205432-189f0dcf9aa0 h1:YAKLiKBNc81iOted1cKtGyedIyLhl+uL3i9/ratthEk=
+sigs.k8s.io/gateway-api-inference-extension v0.0.0-20250320205432-189f0dcf9aa0/go.mod h1:J+5yQfHgtPouF2rqYexLkYUtwCTyHY4Rel1ielpxhz4=
 sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
 sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
 sigs.k8s.io/kind v0.8.1/go.mod h1:oNKTxUVPYkV9lWzY6CVMNluVq8cBsyq+UgPJdvA3uu4=
diff --git a/internal/controller/ai_gateway_route.go b/internal/controller/ai_gateway_route.go
index ef76b0a..d1bb2a7 100644
--- a/internal/controller/ai_gateway_route.go
+++ b/internal/controller/ai_gateway_route.go
@@ -9,6 +9,7 @@ import (
 	"context"
 	"fmt"
 	"path"
+	"sort"
 
 	egv1a1 "github.com/envoyproxy/gateway/api/v1alpha1"
 	"github.com/go-logr/logr"
@@ -23,12 +24,14 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	ctrlutil "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	gwapiv1 "sigs.k8s.io/gateway-api/apis/v1"
 	"sigs.k8s.io/yaml"
 
 	aigv1a1 "github.com/envoyproxy/ai-gateway/api/v1alpha1"
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/internal/controller/rotators"
+	"github.com/envoyproxy/ai-gateway/internal/extensionserver"
 	"github.com/envoyproxy/ai-gateway/internal/llmcostcel"
 )
 
@@ -284,57 +287,38 @@ func (c *AIGatewayRouteController) reconcileExtProcConfigMap(ctx context.Context
 		rule := &spec.Rules[i]
 		ec.Rules[i].Backends = make([]filterapi.Backend, len(rule.BackendRefs))
 		for j := range rule.BackendRefs {
-			backend := &rule.BackendRefs[j]
-			key := fmt.Sprintf("%s.%s", backend.Name, aiGatewayRoute.Namespace)
-			ec.Rules[i].Backends[j].Name = key
-			ec.Rules[i].Backends[j].Weight = backend.Weight
-			var backendObj *aigv1a1.AIServiceBackend
-			backendObj, err = c.backend(ctx, aiGatewayRoute.Namespace, backend.Name)
-			if err != nil {
-				return fmt.Errorf("failed to get AIServiceBackend %s: %w", key, err)
-			}
-			ec.Rules[i].Backends[j].Schema.Name = filterapi.APISchemaName(backendObj.Spec.APISchema.Name)
-			ec.Rules[i].Backends[j].Schema.Version = backendObj.Spec.APISchema.Version
-
-			if bspRef := backendObj.Spec.BackendSecurityPolicyRef; bspRef != nil {
-				volumeName := backendSecurityPolicyVolumeName(
-					i, j, string(backendObj.Spec.BackendSecurityPolicyRef.Name),
-				)
-				var backendSecurityPolicy *aigv1a1.BackendSecurityPolicy
-				backendSecurityPolicy, err = c.backendSecurityPolicy(ctx, aiGatewayRoute.Namespace, string(bspRef.Name))
+			backendRef := &rule.BackendRefs[j]
+			ecBackendConfig := &ec.Rules[i].Backends[j]
+			key := fmt.Sprintf("%s.%s", backendRef.Name, aiGatewayRoute.Namespace)
+			ecBackendConfig.Name = key
+			ecBackendConfig.Weight = backendRef.Weight
+			if isInferencePoolRef(backendRef) {
+				var pool *gwaiev1a2.InferencePool
+				var referencedAIServiceBackends []aigv1a1.AIServiceBackend
+				pool, referencedAIServiceBackends, err = c.getPoolAndReferencedAIServiceBackends(ctx, aiGatewayRoute.Namespace, backendRef.Name)
 				if err != nil {
-					return fmt.Errorf("failed to get BackendSecurityPolicy %s: %w", bspRef.Name, err)
+					return fmt.Errorf("failed to get pool and referenced AIServiceBackends: %w", err)
 				}
-
-				switch backendSecurityPolicy.Spec.Type {
-				case aigv1a1.BackendSecurityPolicyTypeAPIKey:
-					ec.Rules[i].Backends[j].Auth = &filterapi.BackendAuth{
-						APIKey: &filterapi.APIKeyAuth{Filename: path.Join(backendSecurityMountPath(volumeName), apiKey)},
-					}
-				case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
-					if backendSecurityPolicy.Spec.AWSCredentials == nil {
-						return fmt.Errorf("AWSCredentials type selected but not defined %s", backendSecurityPolicy.Name)
-					}
-					if awsCred := backendSecurityPolicy.Spec.AWSCredentials; awsCred.CredentialsFile != nil || awsCred.OIDCExchangeToken != nil {
-						ec.Rules[i].Backends[j].Auth = &filterapi.BackendAuth{
-							AWSAuth: &filterapi.AWSAuth{
-								CredentialFileName: path.Join(backendSecurityMountPath(volumeName), awsCredentialsKey),
-								Region:             backendSecurityPolicy.Spec.AWSCredentials.Region,
-							},
-						}
-					}
-				case aigv1a1.BackendSecurityPolicyTypeAzureCredentials:
-					if backendSecurityPolicy.Spec.AzureCredentials == nil {
-						return fmt.Errorf("AzureCredentials type selected but not defined %s", backendSecurityPolicy.Name)
-					}
-					ec.Rules[i].Backends[j].Auth = &filterapi.BackendAuth{
-						AzureAuth: &filterapi.AzureAuth{
-							Filename: path.Join(backendSecurityMountPath(volumeName), azureAccessTokenKey),
-						},
+				ecBackendConfig.DynamicLoadBalancing, err = c.createDynamicLoadBalancing(ctx, i, j, pool, referencedAIServiceBackends)
+				if err != nil {
+					return fmt.Errorf("failed to create dynamic load balancing: %w", err)
+				}
+			} else {
+				var backendObj *aigv1a1.AIServiceBackend
+				backendObj, err = c.backend(ctx, aiGatewayRoute.Namespace, backendRef.Name)
+				if err != nil {
+					return fmt.Errorf("failed to get AIServiceBackend %s: %w", key, err)
+				}
+				ecBackendConfig.Schema.Name = filterapi.APISchemaName(backendObj.Spec.APISchema.Name)
+				ecBackendConfig.Schema.Version = backendObj.Spec.APISchema.Version
+				if bspRef := backendObj.Spec.BackendSecurityPolicyRef; bspRef != nil {
+					volumeName := backendSecurityPolicyVolumeName(
+						i, j, string(backendObj.Spec.BackendSecurityPolicyRef.Name),
+					)
+					ecBackendConfig.Auth, err = c.bspToFilterAPIAuth(ctx, aiGatewayRoute.Namespace, string(bspRef.Name), volumeName)
+					if err != nil {
+						return fmt.Errorf("failed to create backend auth: %w", err)
 					}
-				default:
-					return fmt.Errorf("invalid backend security type %s for policy %s", backendSecurityPolicy.Spec.Type,
-						backendSecurityPolicy.Name)
 				}
 			}
 		}
@@ -401,49 +385,99 @@ func (c *AIGatewayRouteController) reconcileExtProcConfigMap(ctx context.Context
 	return nil
 }
 
+func (c *AIGatewayRouteController) bspToFilterAPIAuth(ctx context.Context, namespace, bspName, volumeName string) (*filterapi.BackendAuth, error) {
+	backendSecurityPolicy, err := c.backendSecurityPolicy(ctx, namespace, bspName)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get BackendSecurityPolicy %s: %w", bspName, err)
+	}
+	switch backendSecurityPolicy.Spec.Type {
+	case aigv1a1.BackendSecurityPolicyTypeAPIKey:
+		return &filterapi.BackendAuth{
+			APIKey: &filterapi.APIKeyAuth{Filename: path.Join(backendSecurityMountPath(volumeName), apiKey)},
+		}, nil
+	case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
+		if backendSecurityPolicy.Spec.AWSCredentials == nil {
+			return nil, fmt.Errorf("AWSCredentials type selected but not defined %s", backendSecurityPolicy.Name)
+		}
+		if awsCred := backendSecurityPolicy.Spec.AWSCredentials; awsCred.CredentialsFile != nil || awsCred.OIDCExchangeToken != nil {
+			return &filterapi.BackendAuth{
+				AWSAuth: &filterapi.AWSAuth{
+					CredentialFileName: path.Join(backendSecurityMountPath(volumeName), awsCredentialsKey),
+					Region:             backendSecurityPolicy.Spec.AWSCredentials.Region,
+				},
+			}, nil
+		}
+		return nil, nil
+	case aigv1a1.BackendSecurityPolicyTypeAzureCredentials:
+		if backendSecurityPolicy.Spec.AzureCredentials == nil {
+			return nil, fmt.Errorf("AzureCredentials type selected but not defined %s", backendSecurityPolicy.Name)
+		}
+		return &filterapi.BackendAuth{
+			AzureAuth: &filterapi.AzureAuth{
+				Filename: path.Join(backendSecurityMountPath(volumeName), azureAccessTokenKey),
+			},
+		}, nil
+	default:
+		return nil, fmt.Errorf("invalid backend security type %s for policy %s", backendSecurityPolicy.Spec.Type,
+			backendSecurityPolicy.Name)
+	}
+}
+
+var inferencePoolDefaultTimeout = &gwapiv1.HTTPRouteTimeouts{Request: ptr.To[gwapiv1.Duration]("30s")}
+
 // newHTTPRoute updates the HTTPRoute with the new AIGatewayRoute.
 func (c *AIGatewayRouteController) newHTTPRoute(ctx context.Context, dst *gwapiv1.HTTPRoute, aiGatewayRoute *aigv1a1.AIGatewayRoute) error {
-	var backends []*aigv1a1.AIServiceBackend
 	dedup := make(map[string]struct{})
+	rewriteFilters := []gwapiv1.HTTPRouteFilter{{
+		Type: gwapiv1.HTTPRouteFilterExtensionRef,
+		ExtensionRef: &gwapiv1.LocalObjectReference{
+			Group: "gateway.envoyproxy.io",
+			Kind:  "HTTPRouteFilter",
+			Name:  hostRewriteHTTPFilterName,
+		},
+	}}
+	var rules []gwapiv1.HTTPRouteRule
 	for _, rule := range aiGatewayRoute.Spec.Rules {
-		for _, br := range rule.BackendRefs {
-			key := fmt.Sprintf("%s.%s", br.Name, aiGatewayRoute.Namespace)
-			if _, ok := dedup[key]; ok {
+		for i := range rule.BackendRefs {
+			br := &rule.BackendRefs[i]
+			dstName := fmt.Sprintf("%s.%s", br.Name, aiGatewayRoute.Namespace)
+			if _, ok := dedup[dstName]; ok {
 				continue
 			}
-			dedup[key] = struct{}{}
-			backend, err := c.backend(ctx, aiGatewayRoute.Namespace, br.Name)
-			if err != nil {
-				return fmt.Errorf("AIServiceBackend %s not found", key)
+			dedup[dstName] = struct{}{}
+
+			var backendRefs []gwapiv1.HTTPBackendRef
+			var timeouts *gwapiv1.HTTPRouteTimeouts
+			if isInferencePoolRef(br) {
+				// When the target is InferencePool, we don't need the HTTPRoute level setting, but
+				// will route to ORIGINAL_DST, so we don't need to set the backendRefs.
+				//
+				// However, to properly route to the ORIGINAL_DST, we use the constant OriginalDstClusterName
+				// as the selected backend name.
+				dstName = extensionserver.OriginalDstClusterName
+
+				// TODO: make the timeout configurable. One way is to move the timeout setting from
+				// 	AIServiceBackend to AIGatewayRouteBackendRef.
+				timeouts = inferencePoolDefaultTimeout
+			} else {
+				backend, err := c.backend(ctx, aiGatewayRoute.Namespace, br.Name)
+				if err != nil {
+					return fmt.Errorf("AIServiceBackend %s not found", dstName)
+				}
+				backendRefs = []gwapiv1.HTTPBackendRef{
+					{BackendRef: gwapiv1.BackendRef{BackendObjectReference: backend.Spec.BackendRef}},
+				}
+				timeouts = backend.Spec.Timeouts
 			}
-			backends = append(backends, backend)
-		}
-	}
-
-	rewriteFilters := []gwapiv1.HTTPRouteFilter{
-		{
-			Type: gwapiv1.HTTPRouteFilterExtensionRef,
-			ExtensionRef: &gwapiv1.LocalObjectReference{
-				Group: "gateway.envoyproxy.io",
-				Kind:  "HTTPRouteFilter",
-				Name:  hostRewriteHTTPFilterName,
-			},
-		},
-	}
-	rules := make([]gwapiv1.HTTPRouteRule, len(backends))
-	for i, b := range backends {
-		key := fmt.Sprintf("%s.%s", b.Name, b.Namespace)
-		rule := gwapiv1.HTTPRouteRule{
-			BackendRefs: []gwapiv1.HTTPBackendRef{
-				{BackendRef: gwapiv1.BackendRef{BackendObjectReference: b.Spec.BackendRef}},
-			},
-			Matches: []gwapiv1.HTTPRouteMatch{
-				{Headers: []gwapiv1.HTTPHeaderMatch{{Name: selectedBackendHeaderKey, Value: key}}},
-			},
-			Filters:  rewriteFilters,
-			Timeouts: b.Spec.Timeouts,
+			rules = append(rules, gwapiv1.HTTPRouteRule{
+				BackendRefs: backendRefs,
+				Matches: []gwapiv1.HTTPRouteMatch{
+					{Headers: []gwapiv1.HTTPHeaderMatch{{Name: selectedBackendHeaderKey, Value: dstName}}},
+				},
+				Filters:  rewriteFilters,
+				Timeouts: timeouts,
+			})
 		}
-		rules[i] = rule
 	}
 
 	// Adds the default route rule with "/" path. This is necessary because Envoy's router selects the backend
@@ -456,11 +490,8 @@ func (c *AIGatewayRouteController) newHTTPRoute(ctx context.Context, dst *gwapiv
 	// the actual routing at all.
 	if len(rules) > 0 {
 		rules = append(rules, gwapiv1.HTTPRouteRule{
+			Name:    ptr.To[gwapiv1.SectionName]("unreachable"),
 			Matches: []gwapiv1.HTTPRouteMatch{{Path: &gwapiv1.HTTPPathMatch{Value: ptr.To("/")}}},
-			BackendRefs: []gwapiv1.HTTPBackendRef{
-				// It can be any valid backend reference because it will not be used.
-				{BackendRef: gwapiv1.BackendRef{BackendObjectReference: backends[0].Spec.BackendRef}},
-			},
 		})
 	}
 
@@ -639,52 +670,86 @@ func (c *AIGatewayRouteController) mountBackendSecurityPolicySecrets(ctx context
 		rule := &aiGatewayRoute.Spec.Rules[i]
 		for j := range rule.BackendRefs {
 			backendRef := &rule.BackendRefs[j]
-			backend, err := c.backend(ctx, aiGatewayRoute.Namespace, backendRef.Name)
-			if err != nil {
-				return nil, fmt.Errorf("failed to get backend %s: %w", backendRef.Name, err)
-			}
-
-			if backendSecurityPolicyRef := backend.Spec.BackendSecurityPolicyRef; backendSecurityPolicyRef != nil {
-				backendSecurityPolicy, err := c.backendSecurityPolicy(ctx, aiGatewayRoute.Namespace, string(backendSecurityPolicyRef.Name))
+			if isInferencePoolRef(backendRef) {
+				_, referencedAIServiceBackends, err := c.getPoolAndReferencedAIServiceBackends(ctx, aiGatewayRoute.Namespace, backendRef.Name)
 				if err != nil {
-					return nil, fmt.Errorf("failed to get backend security policy %s: %w", backendSecurityPolicyRef.Name, err)
+					return nil, fmt.Errorf("failed to get pool and referenced AIServiceBackends: %w", err)
 				}
-
-				var secretName string
-				switch backendSecurityPolicy.Spec.Type {
-				case aigv1a1.BackendSecurityPolicyTypeAPIKey:
-					secretName = string(backendSecurityPolicy.Spec.APIKey.SecretRef.Name)
-				case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
-					if awsCred := backendSecurityPolicy.Spec.AWSCredentials; awsCred.CredentialsFile != nil {
-						secretName = string(backendSecurityPolicy.Spec.AWSCredentials.CredentialsFile.SecretRef.Name)
-					} else {
-						secretName = rotators.GetBSPSecretName(backendSecurityPolicy.Name)
+				for k := range referencedAIServiceBackends {
+					backend := &referencedAIServiceBackends[k]
+					if backendSecurityPolicyRef := backend.Spec.BackendSecurityPolicyRef; backendSecurityPolicyRef != nil {
+						volumeName := backendSecurityPolicyForInferencePoolVolumeName(i, j, k, string(backend.Spec.BackendSecurityPolicyRef.Name))
+						volume, volumeMount, err := c.backendSecurityPolicyVolumes(ctx, aiGatewayRoute.Namespace,
+							string(backend.Spec.BackendSecurityPolicyRef.Name), volumeName)
+						if err != nil {
+							return nil, fmt.Errorf("failed to populate backend security policy volume: %w", err)
+						}
+						spec.Volumes = append(spec.Volumes, volume)
+						container.VolumeMounts = append(container.VolumeMounts, volumeMount)
 					}
-				case aigv1a1.BackendSecurityPolicyTypeAzureCredentials:
-					secretName = rotators.GetBSPSecretName(backendSecurityPolicy.Name)
-				default:
-					return nil, fmt.Errorf("backend security policy %s is not supported", backendSecurityPolicy.Spec.Type)
+				}
+			} else {
+				backend, err := c.backend(ctx, aiGatewayRoute.Namespace, backendRef.Name)
+				if err != nil {
+					return nil, fmt.Errorf("failed to get backend %s: %w", backendRef.Name, err)
 				}
 
-				volumeName := backendSecurityPolicyVolumeName(i, j, string(backend.Spec.BackendSecurityPolicyRef.Name))
-				spec.Volumes = append(spec.Volumes, corev1.Volume{
-					Name: volumeName,
-					VolumeSource: corev1.VolumeSource{
-						Secret: &corev1.SecretVolumeSource{SecretName: secretName},
-					},
-				})
-
-				container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{
-					Name:      volumeName,
-					MountPath: backendSecurityMountPath(volumeName),
-					ReadOnly:  true,
-				})
+				if backendSecurityPolicyRef := backend.Spec.BackendSecurityPolicyRef; backendSecurityPolicyRef != nil {
+					volumeName := backendSecurityPolicyVolumeName(i, j, string(backend.Spec.BackendSecurityPolicyRef.Name))
+					volume, volumeMount, err := c.backendSecurityPolicyVolumes(ctx, aiGatewayRoute.Namespace,
+						string(backendSecurityPolicyRef.Name), volumeName)
+					if err != nil {
+						return nil, fmt.Errorf("failed to populate backend security policy volume: %w", err)
+					}
+					spec.Volumes = append(spec.Volumes, volume)
+					container.VolumeMounts = append(container.VolumeMounts, volumeMount)
+				}
 			}
 		}
 	}
 	return spec, nil
 }
 
+func (c *AIGatewayRouteController) backendSecurityPolicyVolumes(ctx context.Context, bspNamespace, bspName, volumeName string) (
+	volume corev1.Volume, volumeMount corev1.VolumeMount, err error,
+) {
+	backendSecurityPolicy, err := c.backendSecurityPolicy(ctx, bspNamespace, bspName)
+	if err != nil {
+		err = fmt.Errorf("failed to get backend security policy %s: %w", bspName, err)
+		return
+	}
+
+	var secretName string
+	switch backendSecurityPolicy.Spec.Type {
+	case aigv1a1.BackendSecurityPolicyTypeAPIKey:
+		secretName = string(backendSecurityPolicy.Spec.APIKey.SecretRef.Name)
+	case aigv1a1.BackendSecurityPolicyTypeAWSCredentials:
+		if awsCred := backendSecurityPolicy.Spec.AWSCredentials; awsCred.CredentialsFile != nil {
+			secretName = string(backendSecurityPolicy.Spec.AWSCredentials.CredentialsFile.SecretRef.Name)
+		} else {
+			secretName = rotators.GetBSPSecretName(backendSecurityPolicy.Name)
+		}
+	case aigv1a1.BackendSecurityPolicyTypeAzureCredentials:
+		secretName = rotators.GetBSPSecretName(backendSecurityPolicy.Name)
+	default:
+		err = fmt.Errorf("backend security policy %s is not supported", backendSecurityPolicy.Spec.Type)
+		return
+	}
+
+	volume = corev1.Volume{
+		Name: volumeName,
+		VolumeSource: corev1.VolumeSource{
+			Secret: &corev1.SecretVolumeSource{SecretName: secretName},
+		},
+	}
+	volumeMount = corev1.VolumeMount{
+		Name:      volumeName,
+		MountPath: backendSecurityMountPath(volumeName),
+		ReadOnly:  true,
+	}
+	return
+}
+
 func (c *AIGatewayRouteController) backend(ctx context.Context, namespace, name string) (*aigv1a1.AIServiceBackend, error) {
 	backend := &aigv1a1.AIServiceBackend{}
 	if err := c.client.Get(ctx, client.ObjectKey{Name: name, Namespace: namespace}, backend); err != nil {
@@ -701,6 +766,11 @@ func (c *AIGatewayRouteController) backendSecurityPolicy(ctx context.Context, na
 	return backendSecurityPolicy, nil
 }
 
+func backendSecurityPolicyForInferencePoolVolumeName(ruleIndex, backendRefIndex, inPoolIndex int, name string) string {
+	// Note: do not use "." as it's not allowed in the volume name.
+	return fmt.Sprintf("rule%d-backref%d-inpool%d-%s", ruleIndex, backendRefIndex, inPoolIndex, name)
+}
+
 func backendSecurityPolicyVolumeName(ruleIndex, backendRefIndex int, name string) string {
 	// Note: do not use "." as it's not allowed in the volume name.
 	return fmt.Sprintf("rule%d-backref%d-%s", ruleIndex, backendRefIndex, name)
@@ -717,3 +787,144 @@ func (c *AIGatewayRouteController) updateAIGatewayRouteStatus(ctx context.Contex
 		c.logger.Error(err, "failed to update AIGatewayRoute status")
 	}
 }
+
+// getPoolAndReferencedAIServiceBackends returns the InferencePool and the referenced AIServiceBackends via its selector.
+//
+// The returned AIServiceBackends are sorted by name for the deterministic behavior.
+func (c *AIGatewayRouteController) getPoolAndReferencedAIServiceBackends(ctx context.Context, inferencePoolNamespace, inferencePoolName string) (
+	*gwaiev1a2.InferencePool, []aigv1a1.AIServiceBackend, error,
+) {
+	// Find the referenced InferencePool.
+	var pool gwaiev1a2.InferencePool
+	if err := c.client.Get(ctx, client.ObjectKey{Name: inferencePoolName, Namespace: inferencePoolNamespace}, &pool); err != nil {
+		return nil, nil, fmt.Errorf("failed to get InferencePool %s: %w", inferencePoolName, err)
+	}
+
+	// Gather the AIServiceBackend from the pool.Spec.Selector.
+	labels := make(client.MatchingLabels, len(pool.Spec.Selector))
+	for k, v := range pool.Spec.Selector {
+		labels[string(k)] = string(v)
+	}
+	var aiServiceBackends aigv1a1.AIServiceBackendList
+	if err := c.client.List(ctx, &aiServiceBackends, labels); err != nil {
+		return nil, nil, fmt.Errorf("failed to list AIServiceBackends: %w", err)
+	}
+	sort.Slice(aiServiceBackends.Items, func(i, j int) bool {
+		return aiServiceBackends.Items[i].Name < aiServiceBackends.Items[j].Name
+	})
+	return &pool, aiServiceBackends.Items, nil
+}
+
+func (c *AIGatewayRouteController) createDynamicLoadBalancing(ctx context.Context,
+	ruleIndex, backendRefIndex int,
+	pool *gwaiev1a2.InferencePool,
+	referencedAIServiceBackends []aigv1a1.AIServiceBackend,
+) (*filterapi.DynamicLoadBalancing, error) {
+	ret := &filterapi.DynamicLoadBalancing{Backends: make([]filterapi.DynamicLoadBalancingBackend, len(referencedAIServiceBackends))}
+	var err error
+	for i, b := range referencedAIServiceBackends {
+		sp := b.Spec
+		dynB := filterapi.DynamicLoadBalancingBackend{
+			Port: pool.Spec.TargetPortNumber,
+			Backend: filterapi.Backend{
+				Name: b.Name,
+				Schema: filterapi.VersionedAPISchema{
+					Name:    filterapi.APISchemaName(b.Spec.APISchema.Name),
+					Version: b.Spec.APISchema.Version,
+				},
+			},
+		}
+		if bspRef := b.Spec.BackendSecurityPolicyRef; bspRef != nil {
+			volumeName := backendSecurityPolicyForInferencePoolVolumeName(
+				ruleIndex, backendRefIndex, i, string(bspRef.Name))
+			dynB.Backend.Auth, err = c.bspToFilterAPIAuth(ctx, pool.Namespace, string(bspRef.Name), volumeName)
+			if err != nil {
+				return nil, fmt.Errorf("failed to create backend auth: %w", err)
+			}
+		}
+		clientObjectKey := client.ObjectKey{Name: string(sp.BackendRef.Name), Namespace: pool.Namespace}
+		if sp.BackendRef.Namespace != nil {
+			clientObjectKey.Namespace = string(*sp.BackendRef.Namespace)
+		}
+		// TODO: better check sp.BackendRef.Group as well?
+		switch ptr.Deref(sp.BackendRef.Kind, "Service") {
+		case "Service": // TODO: we should reconcile Service objects to update the section created below when the Service changes.
+			var svc *corev1.Service
+			svc, err = c.kube.CoreV1().Services(clientObjectKey.Namespace).Get(ctx, clientObjectKey.Name, metav1.GetOptions{})
+			if err != nil {
+				return nil, fmt.Errorf("failed to get Service '%s': %w", clientObjectKey.String(), err)
+			}
+			hasTargetPort := false
+			for _, p := range svc.Spec.Ports {
+				if p.Port == pool.Spec.TargetPortNumber {
+					hasTargetPort = true
+					break
+				}
+			}
+			if !hasTargetPort {
+				return nil, fmt.Errorf("port %d not found in Service %s", pool.Spec.TargetPortNumber, sp.BackendRef.Name)
+			}
+			// TODO: at the moment, we assume that all target services are local, i.e. no externalName, etc.
+			//
+			// Note: do not resolve the (pod) IPs at this level which will be resolved by the external processor since
+			// this is the reconciliation subroutine happening when AIServiceBackend (or its transitive referencing resources)
+			// changes.
+			//
+			// This means that if the service is headless, the external processor will resolve the Pod IPs, otherwise
+			// the static cluster IP.
+			//
+			// TODO: we assume that the cluster domain is "cluster.local" which is the default in Kubernetes.
+			// 	Read /etc/resolv.conf and use the search domain to add the suffix instead of hardcoding it.
+			dynB.Hostnames = append(dynB.Hostnames, fmt.Sprintf("%s.%s.svc.cluster.local", svc.Name, svc.Namespace))
+		case "Backend": // TODO: we should reconcile Backend objects to update the section created below when the Backend changes.
+			var backend egv1a1.Backend
+			if err := c.client.Get(ctx, clientObjectKey, &backend); err != nil {
+				return nil, fmt.Errorf("failed to get Backend '%s': %w", clientObjectKey.String(), err)
+			}
+			for _, ep := range backend.Spec.Endpoints {
+				switch {
+				case ep.IP != nil:
+					if ep.IP.Port != pool.Spec.TargetPortNumber {
+						return nil, fmt.Errorf("port mismatch: InferecePool %s has port %d, but Backend %s has port %d",
+							pool.Name, pool.Spec.TargetPortNumber, backend.Name, ep.IP.Port)
+					}
+					dynB.IPs = append(dynB.IPs, ep.IP.Address)
+				case ep.FQDN != nil:
+					if ep.FQDN.Port != pool.Spec.TargetPortNumber {
+						return nil, fmt.Errorf("port mismatch: InferecePool %s has port %d, but Backend %s has port %d",
+							pool.Name, pool.Spec.TargetPortNumber, backend.Name, ep.FQDN.Port)
+					}
+					dynB.Hostnames = append(dynB.Hostnames, ep.FQDN.Hostname)
+				default:
+					return nil, fmt.Errorf("invalid backend endpoint: %v", ep)
+				}
+			}
+		}
+		ret.Backends[i] = dynB
+	}
+
+	// Gather the models that belong to the pool.
+	var models gwaiev1a2.InferenceModelList
+	if err := c.client.List(ctx, &models, client.MatchingFields{
+		k8sClientIndexInferencePoolToReferencingInferenceModel: fmt.Sprintf("%s.%s", pool.Name, pool.Namespace),
+	}); err != nil {
+		return nil, fmt.Errorf("failed to list InferenceModels: %w", err)
+	}
+	for _, model := range models.Items {
+		ret.Models = append(ret.Models, filterapi.DynamicLoadBalancingModel{Name: model.Spec.ModelName})
+		for _, targetModel := range model.Spec.TargetModels {
+			var weight *int
+			if targetModel.Weight != nil {
+				_weight := int(*targetModel.Weight)
+				weight = &_weight
+			}
+			ret.Models = append(ret.Models, filterapi.DynamicLoadBalancingModel{Name: targetModel.Name, Weight: weight})
+		}
+	}
+	return ret, nil
+}
+
+// isInferencePoolRef returns true if AIGatewayRouteRuleBackendRef references an InferencePool reference.
+func isInferencePoolRef(ref *aigv1a1.AIGatewayRouteRuleBackendRef) bool {
+	return ref.Kind != nil && *ref.Kind == aigv1a1.AIGatewayRouteRuleBackendRefInferencePool
+}
diff --git a/internal/controller/controller.go b/internal/controller/controller.go
index f9c255a..ef3afee 100644
--- a/internal/controller/controller.go
+++ b/internal/controller/controller.go
@@ -24,6 +24,7 @@ import (
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	gwapiv1 "sigs.k8s.io/gateway-api/apis/v1"
 	gwapiv1b1 "sigs.k8s.io/gateway-api/apis/v1beta1"
 
@@ -37,6 +38,7 @@ func init() {
 	utilruntime.Must(egv1a1.AddToScheme(Scheme))
 	utilruntime.Must(gwapiv1.Install(Scheme))
 	utilruntime.Must(gwapiv1b1.Install(Scheme))
+	utilruntime.Must(gwaiev1a2.AddToScheme(Scheme))
 }
 
 // Scheme contains the necessary schemes for the AI Gateway.
@@ -53,6 +55,8 @@ type Options struct {
 	// EnableLeaderElection enables leader election for the controller manager.
 	// Enabling this ensures there is only one active controller manager.
 	EnableLeaderElection bool
+	// EnableInfExt enables the Gateway API Inference Extension.
+	EnableInfExt bool
 }
 
 type (
@@ -65,6 +69,9 @@ type (
 	// syncBackendSecurityPolicyFn is a function that syncs a BackendSecurityPolicy. This is used to cross the controller boundary
 	// from Secret to BackendSecurityPolicy when a Secret is referenced by a BackendSecurityPolicy.
 	syncBackendSecurityPolicyFn func(context.Context, *aigv1a1.BackendSecurityPolicy) error
+	// syncInferencePoolFn is a function that syncs an InferencePool. This is used to cross the controller boundary
+	// from InferenceModel to InferencePool when an InferenceModel is referenced by an InferencePool.
+	syncInferencePoolFn func(context.Context, *gwaiev1a2.InferencePool) error
 )
 
 // StartControllers starts the controllers for the AI Gateway.
@@ -85,7 +92,7 @@ func StartControllers(ctx context.Context, config *rest.Config, logger logr.Logg
 
 	c := mgr.GetClient()
 	indexer := mgr.GetFieldIndexer()
-	if err = ApplyIndexing(ctx, indexer.IndexField); err != nil {
+	if err = ApplyIndexing(ctx, options.EnableInfExt, indexer.IndexField); err != nil {
 		return fmt.Errorf("failed to apply indexing: %w", err)
 	}
 
@@ -115,6 +122,24 @@ func StartControllers(ctx context.Context, config *rest.Config, logger logr.Logg
 		return fmt.Errorf("failed to create controller for BackendSecurityPolicy: %w", err)
 	}
 
+	if options.EnableInfExt {
+		inferencePoolC := newInferencePoolController(c,
+			kubernetes.NewForConfigOrDie(config), logger.WithName("inference-pool"),
+			routeC.syncAIGatewayRoute)
+		if err = TypedControllerBuilderForCRD(mgr, &gwaiev1a2.InferencePool{}).
+			Complete(inferencePoolC); err != nil {
+			return fmt.Errorf("failed to create controller for InferencePool: %w", err)
+		}
+
+		inferenceModelC := newInferenceModelController(c,
+			kubernetes.NewForConfigOrDie(config), logger.WithName("inference-model"),
+			inferencePoolC.syncInferencePool)
+		if err = TypedControllerBuilderForCRD(mgr, &gwaiev1a2.InferenceModel{}).
+			Complete(inferenceModelC); err != nil {
+			return fmt.Errorf("failed to create controller for InferenceModel: %w", err)
+		}
+	}
+
 	secretC := NewSecretController(c, kubernetes.NewForConfigOrDie(config), logger.
 		WithName("secret"), backendSecurityPolicyC.syncBackendSecurityPolicy)
 	// Do not use TypedControllerBuilderForCRD for secret, as changing a secret content doesn't change the generation.
@@ -152,25 +177,45 @@ const (
 	// k8sClientIndexBackendSecurityPolicyToReferencingAIServiceBackend is the index name that maps from a BackendSecurityPolicy
 	// to the AIServiceBackend that references it.
 	k8sClientIndexBackendSecurityPolicyToReferencingAIServiceBackend = "BackendSecurityPolicyToReferencingAIServiceBackend"
+	// k8sClientIndexInferencePoolToReferencingAIServiceBackend is the index name that maps from an InferencePool to the AIServiceBackend
+	// that references it.
+	k8sClientIndexInferencePoolToReferencingAIGatewayRoute = "InferencePoolToReferencingAIGatewayRoute"
+	// k8sClientIndexInferencePoolToReferencingInferenceModel is the index name that maps from an InferencePool to the InferenceModel
+	// that references it.
+	k8sClientIndexInferencePoolToReferencingInferenceModel = "InferencePoolToReferencingInferenceModel"
 )
 
 // ApplyIndexing applies indexing to the given indexer. This is exported for testing purposes.
-func ApplyIndexing(ctx context.Context, indexer func(ctx context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error) error {
+func ApplyIndexing(ctx context.Context, enableInfExt bool, indexer func(ctx context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error) error {
 	err := indexer(ctx, &aigv1a1.AIGatewayRoute{},
 		k8sClientIndexBackendToReferencingAIGatewayRoute, aiGatewayRouteIndexFunc)
 	if err != nil {
 		return fmt.Errorf("failed to index field for AIGatewayRoute: %w", err)
 	}
+	if enableInfExt {
+		err = indexer(ctx, &aigv1a1.AIGatewayRoute{},
+			k8sClientIndexInferencePoolToReferencingAIGatewayRoute, aiGatewayRouteIndexFuncForInferencePool)
+		if err != nil {
+			return fmt.Errorf("failed to index field for InferencePool to AIGatewayRoute: %w", err)
+		}
+	}
 	err = indexer(ctx, &aigv1a1.AIServiceBackend{},
 		k8sClientIndexBackendSecurityPolicyToReferencingAIServiceBackend, aiServiceBackendIndexFunc)
 	if err != nil {
-		return fmt.Errorf("failed to index field for AIServiceBackend: %w", err)
+		return fmt.Errorf("failed to index field for BackendSecurityPolicy to AIServiceBackend: %w", err)
 	}
 	err = indexer(ctx, &aigv1a1.BackendSecurityPolicy{},
 		k8sClientIndexSecretToReferencingBackendSecurityPolicy, backendSecurityPolicyIndexFunc)
 	if err != nil {
 		return fmt.Errorf("failed to index field for BackendSecurityPolicy: %w", err)
 	}
+	if enableInfExt {
+		err = indexer(ctx, &gwaiev1a2.InferenceModel{},
+			k8sClientIndexInferencePoolToReferencingInferenceModel, inferenceModelIndexFunc)
+		if err != nil {
+			return fmt.Errorf("failed to index field for InferenceModel: %w", err)
+		}
+	}
 	return nil
 }
 
@@ -186,6 +231,20 @@ func aiGatewayRouteIndexFunc(o client.Object) []string {
 	return ret
 }
 
+func aiGatewayRouteIndexFuncForInferencePool(o client.Object) []string {
+	aiServiceBackend := o.(*aigv1a1.AIGatewayRoute)
+	var ret []string
+	for _, r := range aiServiceBackend.Spec.Rules {
+		for _, backend := range r.BackendRefs {
+			if backend.Kind != nil && *backend.Kind == aigv1a1.AIGatewayRouteRuleBackendRefInferencePool {
+				ns := o.GetNamespace()
+				ret = append(ret, fmt.Sprintf("%s.%s", backend.Name, ns))
+			}
+		}
+	}
+	return ret
+}
+
 func aiServiceBackendIndexFunc(o client.Object) []string {
 	aiServiceBackend := o.(*aigv1a1.AIServiceBackend)
 	var ret []string
@@ -195,6 +254,13 @@ func aiServiceBackendIndexFunc(o client.Object) []string {
 	return ret
 }
 
+func inferenceModelIndexFunc(o client.Object) []string {
+	inferenceModel := o.(*gwaiev1a2.InferenceModel)
+	poolRef := inferenceModel.Spec.PoolRef
+	ns := o.GetNamespace()
+	return []string{fmt.Sprintf("%s.%s", poolRef.Name, ns)}
+}
+
 func backendSecurityPolicyIndexFunc(o client.Object) []string {
 	backendSecurityPolicy := o.(*aigv1a1.BackendSecurityPolicy)
 	var key string
diff --git a/internal/controller/infext_model.go b/internal/controller/infext_model.go
new file mode 100644
index 0000000..fa99ad6
--- /dev/null
+++ b/internal/controller/infext_model.go
@@ -0,0 +1,81 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/go-logr/logr"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/kubernetes"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+)
+
+func newInferenceModelController(client client.Client, kube kubernetes.Interface,
+	logger logr.Logger, syncAIServiceBackend syncInferencePoolFn,
+) *inferenceModelController {
+	return &inferenceModelController{
+		client:              client,
+		kubeClient:          kube,
+		logger:              logger,
+		syncInferencePoolFn: syncAIServiceBackend,
+	}
+}
+
+// inferenceModelController implements reconcile.TypedReconciler for gwaiev1a2.InferenceModel.
+type inferenceModelController struct {
+	client              client.Client
+	kubeClient          kubernetes.Interface
+	logger              logr.Logger
+	syncInferencePoolFn syncInferencePoolFn
+}
+
+// Reconcile implements the reconcile.Reconciler for gwaiev1a2.InferenceModel.
+func (c *inferenceModelController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+	var inferenceModel gwaiev1a2.InferenceModel
+	if err := c.client.Get(ctx, req.NamespacedName, &inferenceModel); err != nil {
+		if apierrors.IsNotFound(err) {
+			c.logger.Info("Deleting Inference Model",
+				"namespace", req.Namespace, "name", req.Name)
+			return ctrl.Result{}, nil
+		}
+		return ctrl.Result{}, err
+	}
+	if err := c.sync(ctx, &inferenceModel); err != nil {
+		// TODO: status update.
+		return ctrl.Result{}, fmt.Errorf("failed to sync InferenceModel: %w", err)
+	}
+	// TODO: status update.
+	return ctrl.Result{}, nil
+}
+
+func (c *inferenceModelController) sync(ctx context.Context, im *gwaiev1a2.InferenceModel) error {
+	poolRef := im.Spec.PoolRef
+	if poolRef.Kind != "InferencePool" {
+		return fmt.Errorf("unexpected poolRef.kind %s", poolRef.Kind)
+	}
+	inferencePoolName := im.Spec.PoolRef.Name
+	var inferencePool gwaiev1a2.InferencePool
+	if err := c.client.Get(ctx, types.NamespacedName{
+		Namespace: im.Namespace, Name: string(inferencePoolName),
+	}, &inferencePool,
+	); err != nil {
+		if apierrors.IsNotFound(err) {
+			c.logger.Info("InferencePool not found.",
+				"namespace", im.Namespace, "name", im.Name)
+			return nil
+		}
+		return err
+	}
+	if err := c.syncInferencePoolFn(ctx, &inferencePool); err != nil {
+		return fmt.Errorf("failed to sync InferencePool: %w", err)
+	}
+	return nil
+}
diff --git a/internal/controller/infext_pool.go b/internal/controller/infext_pool.go
new file mode 100644
index 0000000..c34b69e
--- /dev/null
+++ b/internal/controller/infext_pool.go
@@ -0,0 +1,81 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package controller
+
+import (
+	"context"
+	"fmt"
+
+	"github.com/go-logr/logr"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/client-go/kubernetes"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+
+	aigv1a1 "github.com/envoyproxy/ai-gateway/api/v1alpha1"
+)
+
+func newInferencePoolController(client client.Client, kube kubernetes.Interface,
+	logger logr.Logger, syncAIGatewayRouteFn syncAIGatewayRouteFn,
+) *inferencePoolController {
+	return &inferencePoolController{
+		client:               client,
+		kubeClient:           kube,
+		logger:               logger,
+		syncAIGatewayRouteFn: syncAIGatewayRouteFn,
+	}
+}
+
+// inferencePoolController implements reconcile.TypedReconciler for gwaiev1a2.InferencePool.
+type inferencePoolController struct {
+	client               client.Client
+	kubeClient           kubernetes.Interface
+	logger               logr.Logger
+	syncAIGatewayRouteFn syncAIGatewayRouteFn
+}
+
+// Reconcile implements the reconcile.Reconciler for gwaiev1a2.InferencePool.
+func (c *inferencePoolController) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
+	var inferencePool gwaiev1a2.InferencePool
+	if err := c.client.Get(ctx, req.NamespacedName, &inferencePool); err != nil {
+		if apierrors.IsNotFound(err) {
+			c.logger.Info("Deleting Inference Pool",
+				"namespace", req.Namespace, "name", req.Name)
+			return ctrl.Result{}, nil
+		}
+		return ctrl.Result{}, err
+	}
+	if ref := inferencePool.Spec.ExtensionRef; ref == nil || ref.Name != "envoy-ai-gateway" {
+		c.logger.Info("Skipping InferencePool with extensionRef.Name not equal to 'envoy-ai-gateway'",
+			"namespace", req.Namespace, "name", req.Name)
+		return ctrl.Result{}, nil
+	}
+	if err := c.syncInferencePool(ctx, &inferencePool); err != nil {
+		// TODO: status update.
+		return ctrl.Result{}, fmt.Errorf("failed to sync InferencePool: %w", err)
+	}
+	// TODO: status update.
+	return ctrl.Result{}, nil
+}
+
+func (c *inferencePoolController) syncInferencePool(ctx context.Context, inferencePool *gwaiev1a2.InferencePool) error {
+	var aisbs aigv1a1.AIGatewayRouteList
+	if err := c.client.List(ctx, &aisbs,
+		client.MatchingFields{
+			k8sClientIndexInferencePoolToReferencingAIGatewayRoute: fmt.Sprintf(
+				"%s.%s", inferencePool.Name, inferencePool.Namespace),
+		}); err != nil {
+		return fmt.Errorf("failed to list AIServiceBackends: %w", err)
+	}
+	for i := range aisbs.Items {
+		aisb := &aisbs.Items[i]
+		if err := c.syncAIGatewayRouteFn(ctx, aisb); err != nil {
+			return fmt.Errorf("failed to sync AIServiceBackend: %w", err)
+		}
+	}
+	return nil
+}
diff --git a/internal/extensionserver/extensionserver.go b/internal/extensionserver/extensionserver.go
index bf48f2b..15734a1 100644
--- a/internal/extensionserver/extensionserver.go
+++ b/internal/extensionserver/extensionserver.go
@@ -8,16 +8,19 @@ package extensionserver
 import (
 	"context"
 
-	pb "github.com/envoyproxy/gateway/proto/extension"
+	egextension "github.com/envoyproxy/gateway/proto/extension"
+	clusterv3 "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3"
+	routev3 "github.com/envoyproxy/go-control-plane/envoy/config/route/v3"
 	"github.com/go-logr/logr"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/health/grpc_health_v1"
 	"google.golang.org/grpc/status"
+	"google.golang.org/protobuf/types/known/durationpb"
 )
 
 // Server is the implementation of the EnvoyGatewayExtensionServer interface.
 type Server struct {
-	pb.UnimplementedEnvoyGatewayExtensionServer
+	egextension.UnimplementedEnvoyGatewayExtensionServer
 	log logr.Logger
 }
 
@@ -36,3 +39,70 @@ func (s *Server) Check(context.Context, *grpc_health_v1.HealthCheckRequest) (*gr
 func (s *Server) Watch(*grpc_health_v1.HealthCheckRequest, grpc_health_v1.Health_WatchServer) error {
 	return status.Error(codes.Unimplemented, "Watch is not implemented")
 }
+
+const (
+	// originalDstHeaderName is the header name that will be used to pass the original destination endpoint in the form of "ip:port".
+	originalDstHeaderName = "x-ai-eg-original-dst"
+	// OriginalDstClusterName is the global name of the original destination cluster.
+	OriginalDstClusterName = "original_destination_cluster"
+)
+
+// PostTranslateModify allows an extension to modify the clusters and secrets in the xDS config.
+//
+// Currently, this adds an ORIGINAL_DST cluster to the list of clusters unconditionally.
+func (s *Server) PostTranslateModify(_ context.Context, req *egextension.PostTranslateModifyRequest) (*egextension.PostTranslateModifyResponse, error) {
+	for _, cluster := range req.Clusters {
+		if cluster.Name == OriginalDstClusterName {
+			// The cluster already exists, no need to add it again.
+			s.log.Info("original_dst cluster already exists in the list of clusters")
+			return nil, nil
+		}
+	}
+	// Append the following cluster to the list of clusters:
+	//   name: original_destination_cluster
+	//   connectTimeout: 60s
+	//   lbPolicy: CLUSTER_PROVIDED
+	//   originalDstLbConfig:
+	//     httpHeaderName: x-ai-eg-original-dst
+	//     useHttpHeader: true
+	//   type: ORIGINAL_DST
+	req.Clusters = append(req.Clusters, &clusterv3.Cluster{
+		Name:                 OriginalDstClusterName,
+		ClusterDiscoveryType: &clusterv3.Cluster_Type{Type: clusterv3.Cluster_ORIGINAL_DST},
+		LbPolicy:             clusterv3.Cluster_CLUSTER_PROVIDED,
+		LbConfig: &clusterv3.Cluster_OriginalDstLbConfig_{
+			OriginalDstLbConfig: &clusterv3.Cluster_OriginalDstLbConfig{
+				UseHttpHeader: true, HttpHeaderName: originalDstHeaderName,
+			},
+		},
+		ConnectTimeout: &durationpb.Duration{Seconds: 60},
+	})
+	response := &egextension.PostTranslateModifyResponse{Clusters: req.Clusters, Secrets: req.Secrets}
+	s.log.Info("Added original_dst cluster to the list of clusters")
+	return response, nil
+}
+
+// PostVirtualHostModify allows an extension to modify the virtual hosts in the xDS config.
+//
+// Currently, this replaces the route that has "x-ai-eg-selected-backend" pointing to "original_destination_cluster" to route to the original destination cluster.
+func (s *Server) PostVirtualHostModify(_ context.Context, req *egextension.PostVirtualHostModifyRequest) (*egextension.PostVirtualHostModifyResponse, error) {
+	if req.VirtualHost == nil || len(req.VirtualHost.Routes) == 0 {
+		return nil, nil
+	}
+	for _, route := range req.VirtualHost.Routes {
+		for _, h := range route.Match.Headers {
+			if h.Name != "x-ai-eg-selected-backend" {
+				continue
+			}
+			matcher, ok := h.HeaderMatchSpecifier.(*routev3.HeaderMatcher_StringMatch)
+			if !ok || matcher.StringMatch.GetExact() != OriginalDstClusterName {
+				s.log.Info("unexpected header value", "header", h)
+				continue
+			}
+			route.Action = &routev3.Route_Route{
+				Route: &routev3.RouteAction{ClusterSpecifier: &routev3.RouteAction_Cluster{Cluster: OriginalDstClusterName}},
+			}
+		}
+	}
+	return &egextension.PostVirtualHostModifyResponse{VirtualHost: req.VirtualHost}, nil
+}
diff --git a/internal/extproc/chatcompletion_processor.go b/internal/extproc/chatcompletion_processor.go
index deb018f..cc30b21 100644
--- a/internal/extproc/chatcompletion_processor.go
+++ b/internal/extproc/chatcompletion_processor.go
@@ -57,6 +57,10 @@ type chatCompletionProcessor struct {
 	metrics x.ChatCompletionMetrics
 	// stream is set to true if the request is a streaming request.
 	stream bool
+	// dynamicLB is not nil if the originally selected backend has dynamic load balancing.
+	// TODO: this is not currently used but can be used to do a failover to the whole another backend as per the
+	// the comment in https://github.com/envoyproxy/ai-gateway/issues/34#issuecomment-2743810926.
+	dynamicLB *filterapi.DynamicLoadBalancing
 }
 
 // selectTranslator selects the translator based on the output schema.
@@ -119,7 +123,28 @@ func (c *chatCompletionProcessor) ProcessRequestBody(ctx context.Context, rawBod
 		}
 		return nil, fmt.Errorf("failed to calculate route: %w", err)
 	}
-	c.logger.Info("Selected backend", "backend", b.Name, "schema", b.Schema)
+
+	var headers []*corev3.HeaderValueOption
+	c.dynamicLB = b.DynamicLoadBalancing
+	selectedBackendHeaderValue := b.Name
+	if c.dynamicLB != nil {
+		lb, ok := c.config.dynamicLoadBalancers[c.dynamicLB]
+		if !ok {
+			// If it's not found, that should be a BUG.
+			panic("BUG: failed to find dynamic load balancer")
+		}
+		b, headers, err = lb.SelectChatCompletionsEndpoint(model, c.metrics)
+		if err != nil {
+			return nil, fmt.Errorf("failed to select endpoint: %w", err)
+		}
+		// The selected backend is the dynamic load balancer name.
+		// TODO: we should make this constant as a part of the filterapi package.
+		//  However, that will likely to change after https://github.com/envoyproxy/envoy/pull/38757
+		// 	so for now, we keep it as an inline string.
+		selectedBackendHeaderValue = "original_destination_cluster"
+	}
+
+	c.logger.Info("selected backend", "backend", b.Name, "schema", b.Schema)
 	c.metrics.SetBackend(b)
 
 	if err = c.selectTranslator(b.Schema); err != nil {
@@ -134,13 +159,15 @@ func (c *chatCompletionProcessor) ProcessRequestBody(ctx context.Context, rawBod
 	if headerMutation == nil {
 		headerMutation = &extprocv3.HeaderMutation{}
 	}
-	// Set the model name to the request header with the key `x-ai-gateway-llm-model-name`.
 	headerMutation.SetHeaders = append(headerMutation.SetHeaders, &corev3.HeaderValueOption{
+		// Set the model name to the request header with the key `x-ai-eg-model`.
 		Header: &corev3.HeaderValue{Key: c.config.modelNameHeaderKey, RawValue: []byte(model)},
 	}, &corev3.HeaderValueOption{
-		Header: &corev3.HeaderValue{Key: c.config.selectedBackendHeaderKey, RawValue: []byte(b.Name)},
+		// Also set the selected backend to the request header with the key `x-ai-eg-selected-backend`.
+		Header: &corev3.HeaderValue{Key: c.config.selectedBackendHeaderKey, RawValue: []byte(selectedBackendHeaderValue)},
 	})
-
+	headerMutation.SetHeaders = append(headerMutation.SetHeaders, headers...)
+	// The cluster-based routing is only used when the selected backend is not using dynamic load balancing.
 	if authHandler, ok := c.config.backendAuthHandlers[b.Name]; ok {
 		if err = authHandler.Do(ctx, c.requestHeaders, headerMutation, bodyMutation); err != nil {
 			return nil, fmt.Errorf("failed to do auth request: %w", err)
@@ -169,6 +196,10 @@ func (c *chatCompletionProcessor) ProcessResponseHeaders(ctx context.Context, he
 			c.metrics.RecordRequestCompletion(ctx, false)
 		}
 	}()
+	// TODO: check the status code and use the dynamic load balancing to retry the request per the comment in
+	// 	https://github.com/envoyproxy/ai-gateway/issues/34#issuecomment-2743810926
+	_ = c.dynamicLB
+
 	c.responseHeaders = headersToMap(headers)
 	if enc := c.responseHeaders["content-encoding"]; enc != "" {
 		c.responseEncoding = enc
diff --git a/internal/extproc/dynlb/dynlb.go b/internal/extproc/dynlb/dynlb.go
new file mode 100644
index 0000000..074a101
--- /dev/null
+++ b/internal/extproc/dynlb/dynlb.go
@@ -0,0 +1,171 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+// Package dynlb provides the dynamic load balancer implementation that selects a specific ip:port pair based on
+// the model name and the metrics.
+package dynlb
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"log/slog"
+	"math/rand"
+	"os"
+	"strings"
+
+	corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
+	"github.com/miekg/dns"
+
+	"github.com/envoyproxy/ai-gateway/filterapi"
+	"github.com/envoyproxy/ai-gateway/filterapi/x"
+)
+
+// originalDstHeaderName is the header name that will be used to pass the original destination endpoint in the form of "ip:port".
+const originalDstHeaderName = "x-ai-eg-original-dst"
+
+var dnsServerEndpoint = func() string {
+	if v := os.Getenv("DNS_SERVER_ADDR"); v != "" {
+		return v
+	}
+	config, err := dns.ClientConfigFromFile("/etc/resolv.conf")
+	if err != nil {
+		log.Fatalf("failed to read /etc/resolv.conf: %v", err)
+	}
+	if len(config.Servers) == 0 {
+		log.Fatal("no DNS servers found at /etc/resolv.conf")
+	}
+	return config.Servers[0] + ":" + config.Port
+}()
+
+// DynamicLoadBalancer is the interface for the dynamic load balancer.
+// This corresponds to the dynamicLoadBalancing field in the filter config.
+//
+// This must be concurrency-safe as it will be shared across multiple requests/goroutines.
+type DynamicLoadBalancer interface {
+	// SelectChatCompletionsEndpoint selects an endpoint from the given load balancer to serve the chat completion request.
+	//
+	// The selection result is reflected in the headers to be added to the request, returned as a slice of HeaderValueOption.
+	//
+	// This also returns the selected backend filterapi.Backend to perform per-Backend level operations such rate limiting.
+	SelectChatCompletionsEndpoint(model string, _ x.ChatCompletionMetrics) (
+		selected *filterapi.Backend, headers []*corev3.HeaderValueOption, err error,
+	)
+}
+
+// NewDynamicLoadBalancer returns a new implementation of the DynamicLoadBalancer interface.
+//
+// This is called asynchronously by the config watcher, not on the hot path. The returned DynamicLoadBalancer
+// will be reused for multiple requests/goroutines.
+func NewDynamicLoadBalancer(ctx context.Context, logger *slog.Logger, dyn *filterapi.DynamicLoadBalancing) (DynamicLoadBalancer, error) {
+	return newDynamicLoadBalancer(ctx, logger, dyn, dnsServerEndpoint)
+}
+
+// dynamicLoadBalancer implements NewDynamicLoadBalancer but decoupled for testing.
+func newDynamicLoadBalancer(ctx context.Context, logger *slog.Logger, dyn *filterapi.DynamicLoadBalancing, dnsServerAddr string) (DynamicLoadBalancer, error) {
+	ret := &dynamicLoadBalancer{
+		logger: logger,
+		models: make(map[string]filterapi.DynamicLoadBalancingModel, len(dyn.Models)),
+	}
+
+	// TODO: maybe reuse the client for multiple queries.
+	client := dns.Client{}
+	conn, err := client.Dial(dnsServerAddr)
+	if err != nil {
+		return nil, fmt.Errorf("failed to dial DNS server: %w", err)
+	}
+	defer conn.Close()
+	for _, b := range dyn.Backends {
+		for _, ip := range b.IPs {
+			ret.endpoints = append(ret.endpoints, endpoint{
+				ipPort:  []byte(fmt.Sprintf("%s:%d", ip, b.Port)),
+				backend: &b.Backend,
+			})
+		}
+		logger.Info("resolving hostnames to IP addresses", slog.String("hostnames", strings.Join(b.Hostnames, ",")))
+		// Resolves all hostnames to IP addresses.
+		for _, hostname := range b.Hostnames {
+			// Append a dot if the hostname is not fully qualified.
+			fqdn := hostname
+			if !strings.HasSuffix(fqdn, ".") {
+				fqdn += "."
+			}
+			msg := new(dns.Msg)
+			// TODO: add support for TypeAAAA for IPv6.
+			msg.SetQuestion(fqdn, dns.TypeA)
+			response, _, err := client.ExchangeWithConnContext(ctx, msg, conn)
+			if err != nil {
+				return nil, fmt.Errorf("failed to query DNS server: %w", err)
+			}
+			if response.Rcode != dns.RcodeSuccess {
+				return nil, fmt.Errorf("DNS query failed: %s", dns.RcodeToString[response.Rcode])
+			}
+
+			for _, answer := range response.Answer {
+				if aRecord, ok := answer.(*dns.A); ok {
+					logger.Info("resolved IP address", slog.String("hostname", hostname), slog.String("ip", aRecord.A.String()))
+					ret.endpoints = append(ret.endpoints, endpoint{
+						ipPort:   []byte(fmt.Sprintf("%s:%d", aRecord.A.String(), b.Port)),
+						backend:  &b.Backend,
+						hostname: hostname,
+					})
+				}
+			}
+		}
+	}
+	for _, m := range dyn.Models {
+		ret.models[m.Name] = m
+	}
+	return ret, nil
+}
+
+// dynamicLoadBalancer implements DynamicLoadBalancer.
+type dynamicLoadBalancer struct {
+	logger    *slog.Logger
+	models    map[string]filterapi.DynamicLoadBalancingModel
+	endpoints []endpoint
+}
+
+// endpoint represents an endpoint, a pair of IP and port, which belongs to a backend.
+type endpoint struct {
+	ipPort []byte
+	// hostname is the hostname used to resolve the IP address. Can be empty if the IP is not resolved from a hostname.
+	hostname string
+	// backend is the backend that this ip:port pair belongs to.
+	backend *filterapi.Backend
+}
+
+// SelectChatCompletionsEndpoint implements [DynamicLoadBalancer.SelectChatCompletionsEndpoint].
+//
+// TODO: expand x.ChatCompletionMetrics to add getter methods to be able to make a decision based on the metrics.
+// TODO: this might need to return dynamic metadata instead of headers.
+func (dlb *dynamicLoadBalancer) SelectChatCompletionsEndpoint(model string, _ x.ChatCompletionMetrics) (
+	selected *filterapi.Backend, headers []*corev3.HeaderValueOption, err error,
+) {
+	m, ok := dlb.models[model]
+	if !ok {
+		err = fmt.Errorf("model %s is not found in the dynamic load balancer", model)
+		return
+	}
+
+	// TODO: use the filterapi.DynamicLoadBalancingModel to make a decision.
+	_ = m
+	// Pick random backend for now. TODO: use the metrics to make a decision as commented above.
+	// TODO: Use non blocking rand (if it's really random).
+	ep := dlb.endpoints[rand.Intn(len(dlb.endpoints))] // nolint:gosec
+	dlb.logger.Info("selected endpoint", slog.String("endpoint", string(ep.ipPort)))
+
+	selected = ep.backend
+	headers = []*corev3.HeaderValueOption{
+		{Header: &corev3.HeaderValue{Key: originalDstHeaderName, RawValue: ep.ipPort}},
+	}
+	if hn := ep.hostname; hn != "" {
+		// TODO: Set host header if the IP is resolved from a hostname. Without this, it is likely that we cannot
+		// 	route the requests to external services that reject requests with the mismatching host header.
+		// 	Currently, EG API doesn't support allow us to set mutation_rules.
+		_ = hn
+	}
+	return
+}
diff --git a/internal/extproc/processor.go b/internal/extproc/processor.go
index 97e6ffd..b73ac01 100644
--- a/internal/extproc/processor.go
+++ b/internal/extproc/processor.go
@@ -16,6 +16,7 @@ import (
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/filterapi/x"
 	"github.com/envoyproxy/ai-gateway/internal/extproc/backendauth"
+	"github.com/envoyproxy/ai-gateway/internal/extproc/dynlb"
 )
 
 // processorConfig is the configuration for the processor.
@@ -29,6 +30,7 @@ type processorConfig struct {
 	metadataNamespace                            string
 	requestCosts                                 []processorConfigRequestCost
 	declaredModels                               []string
+	dynamicLoadBalancers                         map[*filterapi.DynamicLoadBalancing]dynlb.DynamicLoadBalancer
 }
 
 // processorConfigRequestCost is the configuration for the request cost.
diff --git a/internal/extproc/server.go b/internal/extproc/server.go
index 9754690..359a5bc 100644
--- a/internal/extproc/server.go
+++ b/internal/extproc/server.go
@@ -26,6 +26,7 @@ import (
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/filterapi/x"
 	"github.com/envoyproxy/ai-gateway/internal/extproc/backendauth"
+	"github.com/envoyproxy/ai-gateway/internal/extproc/dynlb"
 	"github.com/envoyproxy/ai-gateway/internal/extproc/router"
 	"github.com/envoyproxy/ai-gateway/internal/llmcostcel"
 )
@@ -61,6 +62,7 @@ func (s *Server) LoadConfig(ctx context.Context, config *filterapi.Config) error
 	var (
 		backendAuthHandlers = make(map[string]backendauth.Handler)
 		declaredModels      []string
+		dynamicLBs          = make(map[*filterapi.DynamicLoadBalancing]dynlb.DynamicLoadBalancer)
 	)
 	for _, r := range config.Rules {
 		for _, b := range r.Backends {
@@ -70,6 +72,12 @@ func (s *Server) LoadConfig(ctx context.Context, config *filterapi.Config) error
 					return fmt.Errorf("cannot create backend auth handler: %w", err)
 				}
 			}
+			if b.DynamicLoadBalancing != nil {
+				dynamicLBs[b.DynamicLoadBalancing], err = dynlb.NewDynamicLoadBalancer(ctx, s.logger, b.DynamicLoadBalancing)
+				if err != nil {
+					return fmt.Errorf("cannot create dynamic load balancer: %w", err)
+				}
+			}
 		}
 		// Collect declared models from configured header routes. These will be used to
 		// serve requests to the /v1/models endpoint.
@@ -110,6 +118,7 @@ func (s *Server) LoadConfig(ctx context.Context, config *filterapi.Config) error
 		metadataNamespace:        config.MetadataNamespace,
 		requestCosts:             costs,
 		declaredModels:           declaredModels,
+		dynamicLoadBalancers:     dynamicLBs,
 	}
 	s.config = newConfig // This is racey, but we don't care.
 	return nil
diff --git a/internal/extproc/watcher.go b/internal/extproc/watcher.go
index bb387d6..164d7ff 100644
--- a/internal/extproc/watcher.go
+++ b/internal/extproc/watcher.go
@@ -30,6 +30,9 @@ type configWatcher struct {
 	l               *slog.Logger
 	current         string
 	usingDefaultCfg bool
+	// hasDynamicLB is true if the current configuration has at least one backend with dynamic load balancing.
+	// We need to force a reload of the configuration regardless of the file modification time.
+	hasDynamicLB bool
 }
 
 // StartConfigWatcher starts a watcher for the given path and Receiver.
@@ -56,9 +59,11 @@ func (cw *configWatcher) watch(ctx context.Context, tick time.Duration) {
 			cw.l.Info("stop watching the config file", slog.String("path", cw.path))
 			return
 		case <-ticker.C:
-			if err := cw.loadConfig(ctx); err != nil {
+			perTickCtx, cancel := context.WithTimeout(ctx, tick)
+			if err := cw.loadConfig(perTickCtx); err != nil {
 				cw.l.Error("failed to update config", slog.String("error", err.Error()))
 			}
+			cancel()
 		}
 	}
 }
@@ -90,7 +95,7 @@ func (cw *configWatcher) loadConfig(ctx context.Context) error {
 		cw.usingDefaultCfg = true
 	} else {
 		cw.usingDefaultCfg = false
-		if stat.ModTime().Sub(cw.lastMod) <= 0 {
+		if stat.ModTime().Sub(cw.lastMod) <= 0 && !cw.hasDynamicLB {
 			return nil
 		}
 		cw.l.Info("loading a new config", slog.String("path", cw.path))
@@ -109,7 +114,20 @@ func (cw *configWatcher) loadConfig(ctx context.Context) error {
 		cw.diff(previous, cw.current)
 	}
 
-	return cw.rcv.LoadConfig(ctx, cfg)
+	if err = cw.rcv.LoadConfig(ctx, cfg); err != nil {
+		return fmt.Errorf("failed to load config: %w", err)
+	}
+
+	cw.hasDynamicLB = false
+	for _, rule := range cfg.Rules {
+		for _, backend := range rule.Backends {
+			if backend.DynamicLoadBalancing != nil {
+				cw.hasDynamicLB = true
+				break
+			}
+		}
+	}
+	return nil
 }
 
 func (cw *configWatcher) diff(oldConfig, newConfig string) {
diff --git a/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_aigatewayroutes.yaml b/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_aigatewayroutes.yaml
index 576d6d4..096f6f8 100644
--- a/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_aigatewayroutes.yaml
+++ b/manifests/charts/ai-gateway-helm/crds/aigateway.envoyproxy.io_aigatewayroutes.yaml
@@ -282,8 +282,21 @@ spec:
                         The namespace of each backend is "local", i.e. the same namespace as the AIGatewayRoute.
                       items:
                         description: AIGatewayRouteRuleBackendRef is a reference to
-                          a AIServiceBackend with a weight.
+                          a backend with a weight.
                         properties:
+                          kind:
+                            default: AIServiceBackend
+                            description: |-
+                              Kind is the kind of the backend, which is either "AIServiceBackend" or "InferencePool" in Gateway API Inference Extension.
+
+                              When this references InferencePool, the selector of the InferencePool is used to select (multiple) AIServiceBackend(s)
+                              that can serve the same model sets that the InferencePool binds.
+
+                              Default is AIServiceBackend.
+                            enum:
+                            - AIServiceBackend
+                            - InferencePool
+                            type: string
                           name:
                             description: Name is the name of the AIServiceBackend.
                             minLength: 1
diff --git a/manifests/charts/ai-gateway-helm/templates/deployment.yaml b/manifests/charts/ai-gateway-helm/templates/deployment.yaml
index b6e8674..70814b0 100644
--- a/manifests/charts/ai-gateway-helm/templates/deployment.yaml
+++ b/manifests/charts/ai-gateway-helm/templates/deployment.yaml
@@ -43,6 +43,7 @@ spec:
             - -logLevel={{ .Values.controller.logLevel }}
             - --extProcImage={{ .Values.extProc.repository }}:{{ .Values.extProc.tag | default .Chart.AppVersion }}
             - --extProcLogLevel={{ .Values.extProc.logLevel }}
+            - --enableInferenceExtension={{ .Values.controller.enableInferenceExtension }}
             {{- if .Values.controller.leaderElection.enabled }}
             - --enableLeaderElection=true
             {{- end }}
diff --git a/manifests/charts/ai-gateway-helm/templates/serviceaccount.yaml b/manifests/charts/ai-gateway-helm/templates/serviceaccount.yaml
index adf69b4..974ec96 100644
--- a/manifests/charts/ai-gateway-helm/templates/serviceaccount.yaml
+++ b/manifests/charts/ai-gateway-helm/templates/serviceaccount.yaml
@@ -28,6 +28,12 @@ rules:
     resources: ["*"]
     verbs: ["*"]
   ######################
+  - apiGroups:
+      - inference.networking.x-k8s.io
+    resources:
+      - '*'
+    verbs:
+      - '*'
   - apiGroups:
       - gateway.networking.k8s.io
     resources:
diff --git a/manifests/charts/ai-gateway-helm/values.yaml b/manifests/charts/ai-gateway-helm/values.yaml
index fceee46..742c748 100644
--- a/manifests/charts/ai-gateway-helm/values.yaml
+++ b/manifests/charts/ai-gateway-helm/values.yaml
@@ -17,6 +17,13 @@ controller:
   nameOverride: ""
   fullnameOverride: "ai-gateway-controller"
 
+  # Set to true to enable the support for Gateway API Inference Extension custom resources.
+  # https://github.com/kubernetes-sigs/gateway-api-inference-extension
+  #
+  # When enabled, InferencePool objects can be referenced in AIGatewayRoute's rule destination.
+  # See the documentation of AIGatewayRouteRuleBackendRef API for more details.
+  enableInferenceExtension: false
+
   # -- Service Account --
   serviceAccount:
     # Specifies whether a service account should be created
diff --git a/manifests/envoy-gateway-config/config.yaml b/manifests/envoy-gateway-config/config.yaml
index 14e110d..53d5763 100644
--- a/manifests/envoy-gateway-config/config.yaml
+++ b/manifests/envoy-gateway-config/config.yaml
@@ -40,6 +40,16 @@ data:
     extensionApis:
       enableEnvoyPatchPolicy: true
       enableBackend: true
+    extensionManager:
+      hooks:
+        xdsTranslator:
+          post:
+            - VirtualHost
+            - Translation
+      service:
+        fqdn:
+          hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local
+          port: 1063
     rateLimit:
       backend:
         type: Redis
diff --git a/site/docs/api/api.mdx b/site/docs/api/api.mdx
index d3d8d44..4dfd8d2 100644
--- a/site/docs/api/api.mdx
+++ b/site/docs/api/api.mdx
@@ -300,6 +300,7 @@ BackendSecurityPolicyList contains a list of BackendSecurityPolicy
 - [AIGatewayFilterConfigType](#aigatewayfilterconfigtype)
 - [AIGatewayRouteRule](#aigatewayrouterule)
 - [AIGatewayRouteRuleBackendRef](#aigatewayrouterulebackendref)
+- [AIGatewayRouteRuleBackendRefKind](#aigatewayrouterulebackendrefkind)
 - [AIGatewayRouteRuleMatch](#aigatewayrouterulematch)
 - [AIGatewayRouteSpec](#aigatewayroutespec)
 - [AIGatewayRouteStatus](#aigatewayroutestatus)
@@ -429,13 +430,19 @@ AIGatewayRouteRule is a rule that defines the routing behavior of the AIGatewayR
 **Appears in:**
 - [AIGatewayRouteRule](#aigatewayrouterule)
 
-AIGatewayRouteRuleBackendRef is a reference to a AIServiceBackend with a weight.
+AIGatewayRouteRuleBackendRef is a reference to a backend with a weight.
 
 ##### Fields
 
 
 
 <ApiField
+  name="kind"
+  type="[AIGatewayRouteRuleBackendRefKind](#aigatewayrouterulebackendrefkind)"
+  required="true"
+  defaultValue="AIServiceBackend"
+  description="Kind is the kind of the backend, which is either `AIServiceBackend` or `InferencePool` in Gateway API Inference Extension.<br />When this references InferencePool, the selector of the InferencePool is used to select (multiple) AIServiceBackend(s)<br />that can serve the same model sets that the InferencePool binds.<br />Default is AIServiceBackend."
+/><ApiField
   name="name"
   type="string"
   required="true"
@@ -449,6 +456,30 @@ AIGatewayRouteRuleBackendRef is a reference to a AIServiceBackend with a weight.
 />
 
 
+#### AIGatewayRouteRuleBackendRefKind
+
+**Underlying type:** string
+
+**Appears in:**
+- [AIGatewayRouteRuleBackendRef](#aigatewayrouterulebackendref)
+
+AIGatewayRouteRuleBackendRefKind specifies the kind of the backend reference.
+
+
+
+##### Possible Values
+
+<ApiField
+  name="AIServiceBackend"
+  type="enum"
+  required="false"
+  description="AIGatewayRouteRuleBackendRefAIServiceBackend is the kind of the AIServiceBackend.<br />"
+/><ApiField
+  name="InferencePool"
+  type="enum"
+  required="false"
+  description="AIGatewayRouteRuleBackendRefInferencePool is the kind of the InferencePool in the Gateway API Inference Extension.<br />https://github.com/kubernetes-sigs/gateway-api-inference-extension<br />"
+/>
 #### AIGatewayRouteRuleMatch
 
 


Test patch:
diff --git a/cmd/aigw/testdata/translate_basic.out.yaml b/cmd/aigw/testdata/translate_basic.out.yaml
index 9726a75..673b104 100644
--- a/cmd/aigw/testdata/translate_basic.out.yaml
+++ b/cmd/aigw/testdata/translate_basic.out.yaml
@@ -62,13 +62,10 @@ spec:
         - headers:
             - name: x-ai-eg-selected-backend
               value: envoy-ai-gateway-basic-testupstream.default
-    - backendRefs:
-        - group: gateway.envoyproxy.io
-          kind: Backend
-          name: envoy-ai-gateway-basic-openai
-      matches:
+    - matches:
         - path:
             value: /
+      name: unreachable
 ---
 apiVersion: gateway.envoyproxy.io/v1alpha1
 kind: EnvoyExtensionPolicy
diff --git a/cmd/controller/main_test.go b/cmd/controller/main_test.go
index bb5c312..3ffc709 100644
--- a/cmd/controller/main_test.go
+++ b/cmd/controller/main_test.go
@@ -13,12 +13,13 @@ import (
 
 func Test_parseAndValidateFlags(t *testing.T) {
 	t.Run("no flags", func(t *testing.T) {
-		extProcLogLevel, extProcImage, enableLeaderElection, logLevel, extensionServerPort, err := parseAndValidateFlags([]string{})
+		extProcLogLevel, extProcImage, enableLeaderElection, logLevel, extensionServerPort, enableInfExt, err := parseAndValidateFlags([]string{})
 		require.Equal(t, "info", extProcLogLevel)
 		require.Equal(t, "docker.io/envoyproxy/ai-gateway-extproc:latest", extProcImage)
 		require.True(t, enableLeaderElection)
 		require.Equal(t, "info", logLevel.String())
 		require.Equal(t, ":1063", extensionServerPort)
+		require.False(t, enableInfExt)
 		require.NoError(t, err)
 	})
 	t.Run("all flags", func(t *testing.T) {
@@ -36,13 +37,15 @@ func Test_parseAndValidateFlags(t *testing.T) {
 					tc.dash + "enableLeaderElection=false",
 					tc.dash + "logLevel=debug",
 					tc.dash + "port=:8080",
+					tc.dash + "enableInferenceExtension=true",
 				}
-				extProcLogLevel, extProcImage, enableLeaderElection, logLevel, extensionServerPort, err := parseAndValidateFlags(args)
+				extProcLogLevel, extProcImage, enableLeaderElection, logLevel, extensionServerPort, enableInfExt, err := parseAndValidateFlags(args)
 				require.Equal(t, "debug", extProcLogLevel)
 				require.Equal(t, "example.com/extproc:latest", extProcImage)
 				require.False(t, enableLeaderElection)
 				require.Equal(t, "debug", logLevel.String())
 				require.Equal(t, ":8080", extensionServerPort)
+				require.True(t, enableInfExt)
 				require.NoError(t, err)
 			})
 		}
@@ -66,7 +69,7 @@ func Test_parseAndValidateFlags(t *testing.T) {
 			},
 		} {
 			t.Run(tc.name, func(t *testing.T) {
-				_, _, _, _, _, err := parseAndValidateFlags(tc.flags)
+				_, _, _, _, _, _, err := parseAndValidateFlags(tc.flags)
 				require.ErrorContains(t, err, tc.expErr)
 			})
 		}
diff --git a/internal/controller/ai_gateway_route_test.go b/internal/controller/ai_gateway_route_test.go
index 7038b82..3245147 100644
--- a/internal/controller/ai_gateway_route_test.go
+++ b/internal/controller/ai_gateway_route_test.go
@@ -27,12 +27,14 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	gwapiv1 "sigs.k8s.io/gateway-api/apis/v1"
 	gwapiv1a2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
 
 	aigv1a1 "github.com/envoyproxy/ai-gateway/api/v1alpha1"
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/internal/controller/rotators"
+	"github.com/envoyproxy/ai-gateway/internal/extensionserver"
 )
 
 func TestAIGatewayRouteController_Reconcile(t *testing.T) {
@@ -226,7 +228,7 @@ func requireNewFakeClientWithIndexes(t *testing.T) client.Client {
 		WithStatusSubresource(&aigv1a1.AIGatewayRoute{}).
 		WithStatusSubresource(&aigv1a1.AIServiceBackend{}).
 		WithStatusSubresource(&aigv1a1.BackendSecurityPolicy{})
-	err := ApplyIndexing(t.Context(), func(_ context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error {
+	err := ApplyIndexing(t.Context(), true, func(_ context.Context, obj client.Object, field string, extractValue client.IndexerFunc) error {
 		builder = builder.WithIndex(obj, field, extractValue)
 		return nil
 	})
@@ -285,8 +287,8 @@ func TestAIGatewayRouterController_syncAIGatewayRoute(t *testing.T) {
 		require.Equal(t, "apple.ns1", updatedHTTPRoute.Spec.Rules[0].Matches[0].Headers[0].Value)
 		require.Equal(t, "some-backend2", string(updatedHTTPRoute.Spec.Rules[1].BackendRefs[0].BackendRef.Name))
 		require.Equal(t, "orange.ns1", updatedHTTPRoute.Spec.Rules[1].Matches[0].Headers[0].Value)
-		// Defaulting to the first backend.
-		require.Equal(t, "some-backend1", string(updatedHTTPRoute.Spec.Rules[2].BackendRefs[0].BackendRef.Name))
+		// Defaulting to the empty path, which shouldn't reach in practice.
+		require.Empty(t, updatedHTTPRoute.Spec.Rules[2].BackendRefs)
 		require.Equal(t, "/", *updatedHTTPRoute.Spec.Rules[2].Matches[0].Path.Value)
 	})
 
@@ -335,6 +337,12 @@ func Test_newHTTPRoute(t *testing.T) {
 						{
 							BackendRefs: []aigv1a1.AIGatewayRouteRuleBackendRef{{Name: "foo", Weight: 1}},
 						},
+						{
+							BackendRefs: []aigv1a1.AIGatewayRouteRuleBackendRef{{
+								Name: "pool",
+								Kind: ptr.To(aigv1a1.AIGatewayRouteRuleBackendRefInferencePool),
+							}},
+						},
 					},
 				},
 			}
@@ -408,12 +416,18 @@ func Test_newHTTPRoute(t *testing.T) {
 					BackendRefs: []gwapiv1.HTTPBackendRef{{BackendRef: gwapiv1.BackendRef{BackendObjectReference: gwapiv1.BackendObjectReference{Name: "some-backend4", Namespace: refNs}}}},
 					Timeouts:    &gwapiv1.HTTPRouteTimeouts{Request: &timeout1, BackendRequest: &timeout2},
 				},
+				{
+					Matches: []gwapiv1.HTTPRouteMatch{
+						{Headers: []gwapiv1.HTTPHeaderMatch{{Name: selectedBackendHeaderKey, Value: extensionserver.OriginalDstClusterName}}},
+					},
+					Timeouts: inferencePoolDefaultTimeout,
+				},
 			}
-			require.Len(t, httpRoute.Spec.Rules, 5) // 4 backends + 1 for the default rule.
+			require.Len(t, httpRoute.Spec.Rules, 6) // 5 backends + 1 for the default rule.
 			for i, r := range httpRoute.Spec.Rules {
 				t.Run(fmt.Sprintf("rule-%d", i), func(t *testing.T) {
-					if i == 4 {
-						require.Equal(t, expRules[0].BackendRefs, r.BackendRefs)
+					if i == 5 {
+						require.Empty(t, r.BackendRefs)
 						require.NotNil(t, r.Matches[0].Path)
 						require.Equal(t, "/", *r.Matches[0].Path.Value)
 					} else {
@@ -959,7 +973,10 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 
 	for _, backend := range []*aigv1a1.AIServiceBackend{
 		{
-			ObjectMeta: metav1.ObjectMeta{Name: "apple", Namespace: "ns"},
+			ObjectMeta: metav1.ObjectMeta{
+				Name: "apple", Namespace: "ns",
+				Labels: map[string]string{"inference-pool-target:": "yeah"},
+			},
 			Spec: aigv1a1.AIServiceBackendSpec{
 				APISchema: aigv1a1.VersionedAPISchema{
 					Name: aigv1a1.APISchemaAWSBedrock,
@@ -969,7 +986,10 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 			},
 		},
 		{
-			ObjectMeta: metav1.ObjectMeta{Name: "pineapple", Namespace: "ns"},
+			ObjectMeta: metav1.ObjectMeta{
+				Name: "pineapple", Namespace: "ns",
+				Labels: map[string]string{"inference-pool-target:": "yeah"},
+			},
 			Spec: aigv1a1.AIServiceBackendSpec{
 				APISchema: aigv1a1.VersionedAPISchema{
 					Name: aigv1a1.APISchemaAWSBedrock,
@@ -999,6 +1019,18 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 				BackendSecurityPolicyRef: &gwapiv1.LocalObjectReference{Name: "some-other-backend-security-policy-4"},
 			},
 		},
+		{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: "no-security-policy", Namespace: "ns",
+				Labels: map[string]string{"inference-pool-target:": "yeah"},
+			},
+			Spec: aigv1a1.AIServiceBackendSpec{
+				APISchema: aigv1a1.VersionedAPISchema{
+					Name: aigv1a1.APISchemaAWSBedrock,
+				},
+				BackendRef: gwapiv1.BackendObjectReference{Name: "some-backend1", Namespace: ptr.To[gwapiv1.Namespace]("ns")},
+			},
+		},
 	} {
 		require.NoError(t, fakeClient.Create(t.Context(), backend, &client.CreateOptions{}))
 		require.NotNil(t, c)
@@ -1040,10 +1072,26 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 						{Headers: []gwapiv1.HTTPHeaderMatch{{Name: aigv1a1.AIModelHeaderKey, Value: "some-ai-4"}}},
 					},
 				},
+				{
+					BackendRefs: []aigv1a1.AIGatewayRouteRuleBackendRef{
+						{Name: "inference-pool", Weight: 1, Kind: ptr.To(aigv1a1.AIGatewayRouteRuleBackendRefInferencePool)},
+					},
+					Matches: []aigv1a1.AIGatewayRouteRuleMatch{
+						{Headers: []gwapiv1.HTTPHeaderMatch{{Name: "whatever", Value: "yes"}}},
+					},
+				},
 			},
 		},
 	}
 
+	// Create an inference pool
+	require.NoError(t, fakeClient.Create(t.Context(), &gwaiev1a2.InferencePool{
+		ObjectMeta: metav1.ObjectMeta{Name: "inference-pool", Namespace: "ns"},
+		Spec: gwaiev1a2.InferencePoolSpec{
+			Selector: map[gwaiev1a2.LabelKey]gwaiev1a2.LabelValue{"inference-pool-target:": "yeah"},
+		},
+	}, &client.CreateOptions{}))
+
 	spec := corev1.PodSpec{
 		Volumes: []corev1.Volume{
 			{
@@ -1067,34 +1115,70 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 	updatedSpec, err := c.mountBackendSecurityPolicySecrets(t.Context(), &spec, &aiGateway)
 	require.NoError(t, err)
 
-	require.Len(t, updatedSpec.Volumes, 5)
-	require.Len(t, updatedSpec.Containers[0].VolumeMounts, 5)
-	// API Key.
-	require.Equal(t, "some-secret-policy-1", updatedSpec.Volumes[1].VolumeSource.Secret.SecretName)
-	require.Equal(t, "rule0-backref0-some-other-backend-security-policy-1", updatedSpec.Volumes[1].Name)
-	require.Equal(t, "rule0-backref0-some-other-backend-security-policy-1", updatedSpec.Containers[0].VolumeMounts[1].Name)
-	require.Equal(t, "/etc/backend_security_policy/rule0-backref0-some-other-backend-security-policy-1", updatedSpec.Containers[0].VolumeMounts[1].MountPath)
-	// AWS CredentialFile.
-	require.Equal(t, "some-secret-policy-3", updatedSpec.Volumes[2].VolumeSource.Secret.SecretName)
-	require.Equal(t, "rule1-backref0-some-other-backend-security-policy-aws", updatedSpec.Volumes[2].Name)
-	require.Equal(t, "rule1-backref0-some-other-backend-security-policy-aws", updatedSpec.Containers[0].VolumeMounts[2].Name)
-	require.Equal(t, "/etc/backend_security_policy/rule1-backref0-some-other-backend-security-policy-aws", updatedSpec.Containers[0].VolumeMounts[2].MountPath)
-	// AWS OIDC.
-	require.Equal(t, rotators.GetBSPSecretName("aws-oidc-name"), updatedSpec.Volumes[3].VolumeSource.Secret.SecretName)
-	require.Equal(t, "rule2-backref0-aws-oidc-name", updatedSpec.Volumes[3].Name)
-	require.Equal(t, "rule2-backref0-aws-oidc-name", updatedSpec.Containers[0].VolumeMounts[3].Name)
-	require.Equal(t, "/etc/backend_security_policy/rule2-backref0-aws-oidc-name", updatedSpec.Containers[0].VolumeMounts[3].MountPath)
-	// Azure Credentials.
-	require.Equal(t, rotators.GetBSPSecretName("some-other-backend-security-policy-4"), updatedSpec.Volumes[4].VolumeSource.Secret.SecretName)
-	require.Equal(t, "rule3-backref0-some-other-backend-security-policy-4", updatedSpec.Volumes[4].Name)
-	require.Equal(t, "rule3-backref0-some-other-backend-security-policy-4", updatedSpec.Containers[0].VolumeMounts[4].Name)
-	require.Equal(t, "/etc/backend_security_policy/rule3-backref0-some-other-backend-security-policy-4", updatedSpec.Containers[0].VolumeMounts[4].MountPath)
+	// Volumes and volume mounts start with one configmap, and then 6 more for the security policies.
+	require.Len(t, updatedSpec.Volumes, 7)
+	require.Len(t, updatedSpec.Containers[0].VolumeMounts, 7)
+	// Ensure that all security policies are mounted correctly.
+	for i, tc := range []struct {
+		name       string
+		secretName string
+		volumeName string
+		mountPath  string
+	}{
+		{
+			name:       "API Key",
+			secretName: "some-secret-policy-1",
+			volumeName: "rule0-backref0-some-other-backend-security-policy-1",
+			mountPath:  "/etc/backend_security_policy/rule0-backref0-some-other-backend-security-policy-1",
+		},
+		{
+			name:       "AWS CredentialFile",
+			secretName: "some-secret-policy-3",
+			volumeName: "rule1-backref0-some-other-backend-security-policy-aws",
+			mountPath:  "/etc/backend_security_policy/rule1-backref0-some-other-backend-security-policy-aws",
+		},
+		{
+			name:       "AWS OIDC",
+			secretName: rotators.GetBSPSecretName("aws-oidc-name"),
+			volumeName: "rule2-backref0-aws-oidc-name",
+			mountPath:  "/etc/backend_security_policy/rule2-backref0-aws-oidc-name",
+		},
+		{
+			name:       "Azure Credentials",
+			secretName: rotators.GetBSPSecretName("some-other-backend-security-policy-4"),
+			volumeName: "rule3-backref0-some-other-backend-security-policy-4",
+			mountPath:  "/etc/backend_security_policy/rule3-backref0-some-other-backend-security-policy-4",
+		},
+		{
+			name:       "InfernecePool.Ref[0]",
+			secretName: "some-secret-policy-1",
+			volumeName: "rule4-backref0-inpool0-some-other-backend-security-policy-1",
+			mountPath:  "/etc/backend_security_policy/rule4-backref0-inpool0-some-other-backend-security-policy-1",
+		},
+		{
+			name:       "InfernecePool.Ref[2]",
+			secretName: "some-secret-policy-3",
+			volumeName: "rule4-backref0-inpool2-some-other-backend-security-policy-aws",
+			mountPath:  "/etc/backend_security_policy/rule4-backref0-inpool2-some-other-backend-security-policy-aws",
+		},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			index := i + 1 // +1 to skip the configmap volume.
+			require.Equal(t, tc.secretName, updatedSpec.Volumes[index].VolumeSource.Secret.SecretName)
+			require.Equal(t, tc.volumeName, updatedSpec.Volumes[index].Name)
+			require.Equal(t, tc.volumeName, updatedSpec.Containers[0].VolumeMounts[index].Name)
+			require.Equal(t, tc.mountPath, updatedSpec.Containers[0].VolumeMounts[index].MountPath)
+		})
+	}
 
 	require.NoError(t, fakeClient.Delete(t.Context(), &aigv1a1.AIServiceBackend{ObjectMeta: metav1.ObjectMeta{Name: "apple", Namespace: "ns"}}, &client.DeleteOptions{}))
 
 	// Update to new security policy.
 	backend := aigv1a1.AIServiceBackend{
-		ObjectMeta: metav1.ObjectMeta{Name: "apple", Namespace: "ns"},
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "apple", Namespace: "ns",
+			Labels: map[string]string{"inference-pool-target:": "yeah"},
+		},
 		Spec: aigv1a1.AIServiceBackendSpec{
 			APISchema: aigv1a1.VersionedAPISchema{
 				Name: aigv1a1.APISchemaAWSBedrock,
@@ -1110,8 +1194,8 @@ func TestAIGatewayRouteController_MountBackendSecurityPolicySecrets(t *testing.T
 	updatedSpec, err = c.mountBackendSecurityPolicySecrets(t.Context(), &spec, &aiGateway)
 	require.NoError(t, err)
 
-	require.Len(t, updatedSpec.Volumes, 5)
-	require.Len(t, updatedSpec.Containers[0].VolumeMounts, 5)
+	require.Len(t, updatedSpec.Volumes, 7)
+	require.Len(t, updatedSpec.Containers[0].VolumeMounts, 7)
 	require.Equal(t, "some-secret-policy-2", updatedSpec.Volumes[1].VolumeSource.Secret.SecretName)
 	require.Equal(t, "rule0-backref0-some-other-backend-security-policy-2", updatedSpec.Volumes[1].Name)
 	require.Equal(t, "rule0-backref0-some-other-backend-security-policy-2", updatedSpec.Containers[0].VolumeMounts[1].Name)
@@ -1192,3 +1276,179 @@ func TestAIGatewayRouteController_updateAIGatewayRouteStatus(t *testing.T) {
 	require.Equal(t, "ok", updatedRoute.Status.Conditions[0].Message)
 	require.Equal(t, aigv1a1.ConditionTypeAccepted, updatedRoute.Status.Conditions[0].Type)
 }
+
+func TestAIGatewayRouteController_createDynamicLoadBalancing(t *testing.T) {
+	fakeClient := requireNewFakeClientWithIndexes(t)
+	fakeKube := fake2.NewClientset()
+	s := NewAIGatewayRouteController(fakeClient, fakeKube, logr.Discard(), uuid2.NewUUID, "foo", "debug")
+
+	inferencePool := &gwaiev1a2.InferencePool{
+		ObjectMeta: metav1.ObjectMeta{Name: "mypool", Namespace: "default"},
+		Spec:       gwaiev1a2.InferencePoolSpec{TargetPortNumber: 1234},
+	}
+	t.Run("k8s svc", func(t *testing.T) {
+		t.Run("not found", func(t *testing.T) {
+			_, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{
+					Name:      "bar",
+					Namespace: ptr.To[gwapiv1.Namespace]("some-random-ns"),
+				}},
+			}})
+			require.ErrorContains(t, err, "failed to get Service 'some-random-ns/bar': services \"bar\" not found")
+		})
+		t.Run("port not match", func(t *testing.T) {
+			_, err := fakeKube.CoreV1().Services("default").Create(t.Context(), &corev1.Service{
+				ObjectMeta: metav1.ObjectMeta{Name: "different-port", Namespace: "default"},
+				Spec: corev1.ServiceSpec{
+					Ports: []corev1.ServicePort{{Port: 12345}},
+				},
+			}, metav1.CreateOptions{})
+			require.NoError(t, err)
+			_, err = s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec:       aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{Name: "different-port"}},
+			}})
+			require.ErrorContains(t, err, "port 1234 not found in Service different-port")
+		})
+		t.Run("ok", func(t *testing.T) {
+			_, err := fakeKube.CoreV1().Services("default").Create(t.Context(), &corev1.Service{
+				ObjectMeta: metav1.ObjectMeta{Name: "okbackend", Namespace: "default"},
+				Spec: corev1.ServiceSpec{
+					Ports: []corev1.ServicePort{{Port: 1234}},
+				},
+			}, metav1.CreateOptions{})
+			require.NoError(t, err)
+			require.NoError(t, fakeClient.Create(t.Context(), &aigv1a1.BackendSecurityPolicy{
+				ObjectMeta: metav1.ObjectMeta{Name: "some-backend-security-policy", Namespace: "default"},
+				Spec: aigv1a1.BackendSecurityPolicySpec{
+					Type: aigv1a1.BackendSecurityPolicyTypeAPIKey,
+					APIKey: &aigv1a1.BackendSecurityPolicyAPIKey{
+						SecretRef: &gwapiv1.SecretObjectReference{Name: "some-secret-policy", Namespace: ptr.To[gwapiv1.Namespace]("default")},
+					},
+				},
+			}))
+			dyn, err := s.createDynamicLoadBalancing(t.Context(), 555, 999, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "okbackend", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{
+					BackendRef:               gwapiv1.BackendObjectReference{Name: "okbackend"},
+					BackendSecurityPolicyRef: &gwapiv1.LocalObjectReference{Name: "some-backend-security-policy"},
+				},
+			}})
+			require.NoError(t, err)
+			require.Len(t, dyn.Backends, 1)
+			require.Equal(t, "okbackend", dyn.Backends[0].Name)
+			require.Equal(t, []string{"okbackend.default.svc.cluster.local"}, dyn.Backends[0].Hostnames)
+			require.Equal(t, &filterapi.BackendAuth{APIKey: &filterapi.APIKeyAuth{
+				Filename: "/etc/backend_security_policy/rule555-backref999-inpool0-some-backend-security-policy/apiKey",
+			}}, dyn.Backends[0].Auth)
+		})
+	})
+
+	t.Run("eg backend", func(t *testing.T) {
+		t.Run("not found", func(t *testing.T) {
+			_, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{
+					Name:      "bar",
+					Namespace: ptr.To[gwapiv1.Namespace]("some-random-ns"),
+					Kind:      ptr.To[gwapiv1.Kind]("Backend"),
+				}},
+			}})
+			require.ErrorContains(t, err, "failed to get Backend 'some-random-ns/bar': backends.gateway.envoyproxy.io \"bar\" not found")
+		})
+		t.Run("IP.Port not match", func(t *testing.T) {
+			require.NoError(t, fakeClient.Create(t.Context(), &egv1a1.Backend{
+				ObjectMeta: metav1.ObjectMeta{Name: "bar", Namespace: "default"},
+				Spec: egv1a1.BackendSpec{
+					Endpoints: []egv1a1.BackendEndpoint{
+						{IP: &egv1a1.IPEndpoint{Port: 11111}},
+					},
+				},
+			}, &client.CreateOptions{}))
+			_, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{
+					Name: "bar",
+					Kind: ptr.To[gwapiv1.Kind]("Backend"),
+				}},
+			}})
+			require.ErrorContains(t, err, "port mismatch: InferecePool mypool has port 1234, but Backend bar has port 11111")
+		})
+		t.Run("FQDN.Port not match", func(t *testing.T) {
+			require.NoError(t, fakeClient.Create(t.Context(), &egv1a1.Backend{
+				ObjectMeta: metav1.ObjectMeta{Name: "fqdnport", Namespace: "default"},
+				Spec: egv1a1.BackendSpec{
+					Endpoints: []egv1a1.BackendEndpoint{{FQDN: &egv1a1.FQDNEndpoint{Port: 11111}}},
+				},
+			}, &client.CreateOptions{}))
+			_, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{
+					Name: "fqdnport",
+					Kind: ptr.To[gwapiv1.Kind]("Backend"),
+				}},
+			}})
+			require.ErrorContains(t, err, "port mismatch: InferecePool mypool has port 1234, but Backend fqdnport has port 11111")
+		})
+		t.Run("ok", func(t *testing.T) {
+			require.NoError(t, fakeClient.Create(t.Context(), &egv1a1.Backend{
+				ObjectMeta: metav1.ObjectMeta{Name: "okbackend", Namespace: "default"},
+				Spec: egv1a1.BackendSpec{
+					Endpoints: []egv1a1.BackendEndpoint{
+						{FQDN: &egv1a1.FQDNEndpoint{Port: 1234, Hostname: "cat.com"}},
+						{IP: &egv1a1.IPEndpoint{Port: 1234, Address: "1.1.1.1"}},
+					},
+				},
+			}, &client.CreateOptions{}))
+			dyn, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, []aigv1a1.AIServiceBackend{{
+				ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+				Spec: aigv1a1.AIServiceBackendSpec{BackendRef: gwapiv1.BackendObjectReference{
+					Name: "okbackend",
+					Kind: ptr.To[gwapiv1.Kind]("Backend"),
+				}},
+			}})
+			require.NoError(t, err)
+			require.Len(t, dyn.Backends, 1)
+			require.Equal(t, "foo", dyn.Backends[0].Name)
+			require.Equal(t, []string{"cat.com"}, dyn.Backends[0].Hostnames)
+			require.Equal(t, []string{"1.1.1.1"}, dyn.Backends[0].IPs)
+		})
+	})
+
+	t.Run("models", func(t *testing.T) {
+		inferencePool := &gwaiev1a2.InferencePool{
+			ObjectMeta: metav1.ObjectMeta{Name: "mypool", Namespace: "default"},
+			Spec:       gwaiev1a2.InferencePoolSpec{TargetPortNumber: 1234},
+		}
+
+		require.NoError(t, fakeClient.Create(t.Context(), &gwaiev1a2.InferenceModel{
+			ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"},
+			Spec: gwaiev1a2.InferenceModelSpec{
+				ModelName: "model1",
+				PoolRef:   gwaiev1a2.PoolObjectReference{Name: "mypool"},
+			},
+		}, &client.CreateOptions{}))
+
+		require.NoError(t, fakeClient.Create(t.Context(), &gwaiev1a2.InferenceModel{
+			ObjectMeta: metav1.ObjectMeta{Name: "bar", Namespace: "default"},
+			Spec: gwaiev1a2.InferenceModelSpec{
+				ModelName: "model2",
+				PoolRef:   gwaiev1a2.PoolObjectReference{Name: "mypool"},
+				TargetModels: []gwaiev1a2.TargetModel{
+					{Name: "model3"},
+					{Name: "model4", Weight: ptr.To(int32(1))},
+				},
+			},
+		}, &client.CreateOptions{}))
+
+		dyn, err := s.createDynamicLoadBalancing(t.Context(), 0, 0, inferencePool, nil)
+		require.NoError(t, err)
+		require.ElementsMatch(t, []filterapi.DynamicLoadBalancingModel{
+			{Name: "model1"},
+			{Name: "model2"},
+			{Name: "model3"},
+			{Name: "model4", Weight: ptr.To(1)},
+		}, dyn.Models)
+	})
+}
diff --git a/internal/controller/ai_service_backend_test.go b/internal/controller/ai_service_backend_test.go
index fa6054c..7405dae 100644
--- a/internal/controller/ai_service_backend_test.go
+++ b/internal/controller/ai_service_backend_test.go
@@ -15,7 +15,6 @@ import (
 	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
-	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 	gwapiv1 "sigs.k8s.io/gateway-api/apis/v1"
 	gwapiv1a2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
@@ -91,10 +90,7 @@ func TestAIServiceBackendController_Reconcile(t *testing.T) {
 }
 
 func Test_AiServiceBackendIndexFunc(t *testing.T) {
-	c := fake.NewClientBuilder().
-		WithScheme(Scheme).
-		WithIndex(&aigv1a1.AIServiceBackend{}, k8sClientIndexBackendSecurityPolicyToReferencingAIServiceBackend, aiServiceBackendIndexFunc).
-		Build()
+	c := requireNewFakeClientWithIndexes(t)
 
 	// Create Backend Security Policies.
 	for _, bsp := range []*aigv1a1.BackendSecurityPolicy{
diff --git a/internal/controller/controller_test.go b/internal/controller/controller_test.go
index 5327ec6..d39bb69 100644
--- a/internal/controller/controller_test.go
+++ b/internal/controller/controller_test.go
@@ -14,6 +14,7 @@ import (
 	"k8s.io/utils/ptr"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	gwapiv1 "sigs.k8s.io/gateway-api/apis/v1"
 	gwapiv1a2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
 
@@ -25,10 +26,7 @@ func TestMain(m *testing.M) {
 }
 
 func Test_aiGatewayRouteIndexFunc(t *testing.T) {
-	c := fake.NewClientBuilder().
-		WithScheme(Scheme).
-		WithIndex(&aigv1a1.AIGatewayRoute{}, k8sClientIndexBackendToReferencingAIGatewayRoute, aiGatewayRouteIndexFunc).
-		Build()
+	c := requireNewFakeClientWithIndexes(t)
 
 	// Create a AIGatewayRoute.
 	aiGatewayRoute := &aigv1a1.AIGatewayRoute{
@@ -47,6 +45,8 @@ func Test_aiGatewayRouteIndexFunc(t *testing.T) {
 					BackendRefs: []aigv1a1.AIGatewayRouteRuleBackendRef{
 						{Name: "backend1", Weight: 1},
 						{Name: "backend2", Weight: 1},
+						{Name: "inference-pool1", Weight: 1, Kind: ptr.To(aigv1a1.AIGatewayRouteRuleBackendRefInferencePool)},
+						{Name: "inference-pool2", Weight: 1, Kind: ptr.To(aigv1a1.AIGatewayRouteRuleBackendRefInferencePool)},
 					},
 				},
 			},
@@ -66,6 +66,18 @@ func Test_aiGatewayRouteIndexFunc(t *testing.T) {
 	require.NoError(t, err)
 	require.Len(t, aiGatewayRoutes.Items, 1)
 	require.Equal(t, aiGatewayRoute.Name, aiGatewayRoutes.Items[0].Name)
+
+	err = c.List(t.Context(), &aiGatewayRoutes,
+		client.MatchingFields{k8sClientIndexInferencePoolToReferencingAIGatewayRoute: "inference-pool1.default"})
+	require.NoError(t, err)
+	require.Len(t, aiGatewayRoutes.Items, 1)
+	require.Equal(t, aiGatewayRoute.Name, aiGatewayRoutes.Items[0].Name)
+
+	err = c.List(t.Context(), &aiGatewayRoutes,
+		client.MatchingFields{k8sClientIndexInferencePoolToReferencingAIGatewayRoute: "inference-pool2.default"})
+	require.NoError(t, err)
+	require.Len(t, aiGatewayRoutes.Items, 1)
+	require.Equal(t, aiGatewayRoute.Name, aiGatewayRoutes.Items[0].Name)
 }
 
 func Test_backendSecurityPolicyIndexFunc(t *testing.T) {
@@ -165,3 +177,46 @@ func Test_getSecretNameAndNamespace(t *testing.T) {
 	secretRef.Namespace = nil
 	require.Equal(t, "mysecret.foo", getSecretNameAndNamespace(secretRef, "foo"))
 }
+
+func Test_inferenceModelIndexFunc(t *testing.T) {
+	c := requireNewFakeClientWithIndexes(t)
+
+	for _, inf := range []*gwaiev1a2.InferenceModel{
+		{
+			ObjectMeta: metav1.ObjectMeta{Name: "mymodel", Namespace: "default"},
+			Spec: gwaiev1a2.InferenceModelSpec{
+				ModelName: "mymodel",
+				PoolRef:   gwaiev1a2.PoolObjectReference{Name: "mypool"},
+			},
+		},
+		{
+			ObjectMeta: metav1.ObjectMeta{Name: "mymodel2", Namespace: "default"},
+			Spec: gwaiev1a2.InferenceModelSpec{
+				ModelName: "mymodel2",
+				PoolRef:   gwaiev1a2.PoolObjectReference{Name: "mypool"},
+			},
+		},
+		{
+			ObjectMeta: metav1.ObjectMeta{Name: "mymodel3", Namespace: "default"},
+			Spec: gwaiev1a2.InferenceModelSpec{
+				ModelName: "mymodel2",
+				PoolRef:   gwaiev1a2.PoolObjectReference{Name: "anotherpool"},
+			},
+		},
+	} {
+		require.NoError(t, c.Create(t.Context(), inf))
+	}
+
+	var inferenceModels gwaiev1a2.InferenceModelList
+	err := c.List(t.Context(), &inferenceModels,
+		client.MatchingFields{k8sClientIndexInferencePoolToReferencingInferenceModel: "mypool.default"})
+	require.NoError(t, err)
+	require.Len(t, inferenceModels.Items, 2)
+	require.ElementsMatch(t, []string{"mymodel", "mymodel2"}, []string{inferenceModels.Items[0].Name, inferenceModels.Items[1].Name})
+
+	err = c.List(t.Context(), &inferenceModels,
+		client.MatchingFields{k8sClientIndexInferencePoolToReferencingInferenceModel: "anotherpool.default"})
+	require.NoError(t, err)
+	require.Len(t, inferenceModels.Items, 1)
+	require.Equal(t, "mymodel3", inferenceModels.Items[0].Name)
+}
diff --git a/internal/controller/infext_model_test.go b/internal/controller/infext_model_test.go
new file mode 100644
index 0000000..1404710
--- /dev/null
+++ b/internal/controller/infext_model_test.go
@@ -0,0 +1,77 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package controller
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	fake2 "k8s.io/client-go/kubernetes/fake"
+	ctrl "sigs.k8s.io/controller-runtime"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+
+	internaltesting "github.com/envoyproxy/ai-gateway/internal/testing"
+)
+
+func TestInferenceModelControllerReconcile(t *testing.T) {
+	client := requireNewFakeClientWithIndexes(t)
+
+	require.NoError(t, client.Create(t.Context(), &gwaiev1a2.InferencePool{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "inference-pool",
+			Namespace: "default",
+		},
+		Spec: gwaiev1a2.InferencePoolSpec{},
+	}))
+	require.NoError(t, client.Create(t.Context(), &gwaiev1a2.InferenceModel{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "inference-pool",
+			Namespace: "default",
+		},
+		Spec: gwaiev1a2.InferenceModelSpec{PoolRef: gwaiev1a2.PoolObjectReference{
+			Kind: "InferencePool",
+			Name: "inference-pool",
+		}},
+	}))
+	require.NoError(t, client.Create(t.Context(), &gwaiev1a2.InferenceModel{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "ref-non-existing-pool",
+			Namespace: "default",
+		},
+		Spec: gwaiev1a2.InferenceModelSpec{PoolRef: gwaiev1a2.PoolObjectReference{
+			Kind: "InferencePool",
+			Name: "non-existing-pool",
+		}},
+	}))
+
+	syncFn := internaltesting.NewSyncFnImpl[gwaiev1a2.InferencePool]()
+	infCtrl := newInferenceModelController(client, fake2.NewClientset(), ctrl.Log, syncFn.Sync)
+
+	t.Run("ok", func(t *testing.T) {
+		defer syncFn.Reset()
+		_, err := infCtrl.Reconcile(t.Context(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "inference-pool", Namespace: "default"}})
+		require.NoError(t, err)
+		actual := syncFn.GetItems()
+		require.Len(t, actual, 1)
+		require.Equal(t, "inference-pool", actual[0].Name)
+	})
+	t.Run("referencing pool not exist", func(t *testing.T) {
+		defer syncFn.Reset()
+		_, err := infCtrl.Reconcile(t.Context(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "ref-non-existing-pool", Namespace: "default"}})
+		require.NoError(t, err)
+		actual := syncFn.GetItems()
+		require.Empty(t, actual)
+	})
+	t.Run("inference model not exist", func(t *testing.T) {
+		defer syncFn.Reset()
+		_, err := infCtrl.Reconcile(t.Context(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "inference-pool2", Namespace: "default"}})
+		require.NoError(t, err)
+		actual := syncFn.GetItems()
+		require.Empty(t, actual)
+	})
+}
diff --git a/internal/controller/infext_pool_test.go b/internal/controller/infext_pool_test.go
new file mode 100644
index 0000000..24460a9
--- /dev/null
+++ b/internal/controller/infext_pool_test.go
@@ -0,0 +1,80 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package controller
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	fake2 "k8s.io/client-go/kubernetes/fake"
+	"k8s.io/utils/ptr"
+	ctrl "sigs.k8s.io/controller-runtime"
+	gwaiev1a2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+	gwapiv1a2 "sigs.k8s.io/gateway-api/apis/v1alpha2"
+
+	aigv1a1 "github.com/envoyproxy/ai-gateway/api/v1alpha1"
+	internaltesting "github.com/envoyproxy/ai-gateway/internal/testing"
+)
+
+func TestInferencePoolControllerReconcile(t *testing.T) {
+	client := requireNewFakeClientWithIndexes(t)
+	require.NoError(t, client.Create(t.Context(), &aigv1a1.AIGatewayRoute{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "myroute",
+			Namespace: "default",
+		},
+		Spec: aigv1a1.AIGatewayRouteSpec{
+			TargetRefs: []gwapiv1a2.LocalPolicyTargetReferenceWithSectionName{
+				{LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{Name: "mytarget"}},
+				{LocalPolicyTargetReference: gwapiv1a2.LocalPolicyTargetReference{Name: "mytarget2"}},
+			},
+			Rules: []aigv1a1.AIGatewayRouteRule{
+				{
+					Matches: []aigv1a1.AIGatewayRouteRuleMatch{},
+					BackendRefs: []aigv1a1.AIGatewayRouteRuleBackendRef{
+						{Name: "backend2", Weight: 1},
+						{Name: "inference-pool", Weight: 1, Kind: ptr.To(aigv1a1.AIGatewayRouteRuleBackendRefInferencePool)},
+					},
+				},
+			},
+		},
+	}))
+
+	require.NoError(t, client.Create(t.Context(), &gwaiev1a2.InferencePool{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "inference-pool",
+			Namespace: "default",
+		},
+		Spec: gwaiev1a2.InferencePoolSpec{
+			EndpointPickerConfig: gwaiev1a2.EndpointPickerConfig{
+				ExtensionRef: &gwaiev1a2.Extension{
+					ExtensionReference: gwaiev1a2.ExtensionReference{Name: "envoy-ai-gateway"},
+				},
+			},
+		},
+	}))
+
+	syncFn := internaltesting.NewSyncFnImpl[aigv1a1.AIGatewayRoute]()
+	infCtrl := newInferencePoolController(client, fake2.NewClientset(), ctrl.Log, syncFn.Sync)
+
+	t.Run("exist", func(t *testing.T) {
+		defer syncFn.Reset()
+		_, err := infCtrl.Reconcile(t.Context(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "inference-pool", Namespace: "default"}})
+		require.NoError(t, err)
+		actual := syncFn.GetItems()
+		require.Len(t, actual, 1)
+		require.Equal(t, "myroute", actual[0].Name)
+	})
+	t.Run("not found", func(t *testing.T) {
+		defer syncFn.Reset()
+		_, err := infCtrl.Reconcile(t.Context(), ctrl.Request{NamespacedName: types.NamespacedName{Name: "not-found", Namespace: "default"}})
+		require.NoError(t, err)
+		actual := syncFn.GetItems()
+		require.Empty(t, actual)
+	})
+}
diff --git a/internal/extensionserver/extension_server_test.go b/internal/extensionserver/extension_server_test.go
deleted file mode 100644
index e8e7df8..0000000
--- a/internal/extensionserver/extension_server_test.go
+++ /dev/null
@@ -1,34 +0,0 @@
-// Copyright Envoy AI Gateway Authors
-// SPDX-License-Identifier: Apache-2.0
-// The full text of the Apache license is available in the LICENSE file at
-// the root of the repo.
-
-package extensionserver
-
-import (
-	"testing"
-
-	"github.com/go-logr/logr"
-	"github.com/stretchr/testify/require"
-)
-
-func TestNew(t *testing.T) {
-	logger := logr.Discard()
-	s := New(logger)
-	require.NotNil(t, s)
-}
-
-func TestCheck(t *testing.T) {
-	logger := logr.Discard()
-	s := New(logger)
-	_, err := s.Check(t.Context(), nil)
-	require.NoError(t, err)
-}
-
-func TestWatch(t *testing.T) {
-	logger := logr.Discard()
-	s := New(logger)
-	err := s.Watch(nil, nil)
-	require.Error(t, err)
-	require.Equal(t, "rpc error: code = Unimplemented desc = Watch is not implemented", err.Error())
-}
diff --git a/internal/extensionserver/extensionserver_test.go b/internal/extensionserver/extensionserver_test.go
new file mode 100644
index 0000000..6136eb7
--- /dev/null
+++ b/internal/extensionserver/extensionserver_test.go
@@ -0,0 +1,111 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package extensionserver
+
+import (
+	"testing"
+
+	egextension "github.com/envoyproxy/gateway/proto/extension"
+	clusterv3 "github.com/envoyproxy/go-control-plane/envoy/config/cluster/v3"
+	routev3 "github.com/envoyproxy/go-control-plane/envoy/config/route/v3"
+	matcherv3 "github.com/envoyproxy/go-control-plane/envoy/type/matcher/v3"
+	"github.com/go-logr/logr"
+	"github.com/stretchr/testify/require"
+)
+
+func TestNew(t *testing.T) {
+	logger := logr.Discard()
+	s := New(logger)
+	require.NotNil(t, s)
+}
+
+func TestCheck(t *testing.T) {
+	logger := logr.Discard()
+	s := New(logger)
+	_, err := s.Check(t.Context(), nil)
+	require.NoError(t, err)
+}
+
+func TestWatch(t *testing.T) {
+	logger := logr.Discard()
+	s := New(logger)
+	err := s.Watch(nil, nil)
+	require.Error(t, err)
+	require.Equal(t, "rpc error: code = Unimplemented desc = Watch is not implemented", err.Error())
+}
+
+func TestServerPostTranslateModify(t *testing.T) {
+	t.Run("existing", func(t *testing.T) {
+		s := New(logr.Discard())
+		res, err := s.PostTranslateModify(t.Context(), &egextension.PostTranslateModifyRequest{
+			Clusters: []*clusterv3.Cluster{
+				{Name: OriginalDstClusterName},
+			},
+		})
+		require.Nil(t, res)
+		require.NoError(t, err)
+	})
+	t.Run("not existing", func(t *testing.T) {
+		s := New(logr.Discard())
+		res, err := s.PostTranslateModify(t.Context(), &egextension.PostTranslateModifyRequest{
+			Clusters: []*clusterv3.Cluster{
+				{Name: "foo"},
+			},
+		})
+		require.NotNil(t, res)
+		require.NoError(t, err)
+		require.Len(t, res.Clusters, 2)
+		require.Equal(t, "foo", res.Clusters[0].Name)
+		require.Equal(t, OriginalDstClusterName, res.Clusters[1].Name)
+	})
+}
+
+func TestServerPostVirtualHostModify(t *testing.T) {
+	t.Run("nil virtual host", func(t *testing.T) {
+		s := New(logr.Discard())
+		res, err := s.PostVirtualHostModify(t.Context(), &egextension.PostVirtualHostModifyRequest{})
+		require.Nil(t, res)
+		require.NoError(t, err)
+	})
+	t.Run("zero routes", func(t *testing.T) {
+		s := New(logr.Discard())
+		res, err := s.PostVirtualHostModify(t.Context(), &egextension.PostVirtualHostModifyRequest{
+			VirtualHost: &routev3.VirtualHost{},
+		})
+		require.Nil(t, res)
+		require.NoError(t, err)
+	})
+	t.Run("route", func(t *testing.T) {
+		s := New(logr.Discard())
+		res, err := s.PostVirtualHostModify(t.Context(), &egextension.PostVirtualHostModifyRequest{
+			VirtualHost: &routev3.VirtualHost{
+				Routes: []*routev3.Route{
+					{Name: "foo", Match: &routev3.RouteMatch{
+						Headers: []*routev3.HeaderMatcher{
+							{
+								Name: "x-ai-eg-selected-backend",
+								HeaderMatchSpecifier: &routev3.HeaderMatcher_StringMatch{
+									StringMatch: &matcherv3.StringMatcher{
+										MatchPattern: &matcherv3.StringMatcher_Exact{
+											Exact: OriginalDstClusterName,
+										},
+									},
+								},
+							},
+						},
+					}},
+				},
+			},
+		})
+		require.NotNil(t, res)
+		require.NoError(t, err)
+		// Original route should be first per the code comment.
+		require.Equal(t, "foo", res.VirtualHost.Routes[0].Name)
+		// Ensure that the action has been updated.
+		require.Equal(t, OriginalDstClusterName, res.VirtualHost.Routes[0].Action.(*routev3.Route_Route).
+			Route.ClusterSpecifier.(*routev3.RouteAction_Cluster).Cluster)
+	})
+}
diff --git a/internal/extproc/chatcompletion_processor_test.go b/internal/extproc/chatcompletion_processor_test.go
index b77d4fa..7ca04fb 100644
--- a/internal/extproc/chatcompletion_processor_test.go
+++ b/internal/extproc/chatcompletion_processor_test.go
@@ -22,6 +22,7 @@ import (
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/filterapi/x"
 	"github.com/envoyproxy/ai-gateway/internal/apischema/openai"
+	"github.com/envoyproxy/ai-gateway/internal/extproc/dynlb"
 	"github.com/envoyproxy/ai-gateway/internal/extproc/translator"
 	"github.com/envoyproxy/ai-gateway/internal/llmcostcel"
 )
@@ -317,49 +318,79 @@ func TestChatCompletion_ProcessRequestBody(t *testing.T) {
 				require.False(t, p.stream) // On error, stream should be false regardless of the input.
 			})
 			t.Run("ok", func(t *testing.T) {
-				someBody := bodyFromModel(t, "some-model")
-				headers := map[string]string{":path": "/foo"}
-				rt := mockRouter{
-					t: t, expHeaders: headers, retBackendName: "some-backend",
-					retVersionedAPISchema: filterapi.VersionedAPISchema{Name: "some-schema", Version: "v10.0"},
-				}
-				headerMut := &extprocv3.HeaderMutation{}
-				bodyMut := &extprocv3.BodyMutation{}
+				for _, tc := range []struct {
+					name  string
+					dynlb *mockDynamicLB
+				}{
+					{name: "no-dynlb"},
+					{name: "dynlb", dynlb: &mockDynamicLB{
+						backedName: "some-backend",
+						headers:    []*corev3.HeaderValueOption{{Header: &corev3.HeaderValue{Key: "foo", Value: "bar"}}},
+					}},
+				} {
+					t.Run(tc.name, func(t *testing.T) {
+						someBody := bodyFromModel(t, "some-model")
+						headers := map[string]string{":path": "/foo"}
+						dynLb := &filterapi.DynamicLoadBalancing{}
+						rt := mockRouter{
+							t: t, expHeaders: headers, retBackendName: "some-backend",
+							retVersionedAPISchema: filterapi.VersionedAPISchema{Name: "some-schema", Version: "v10.0"},
+						}
+						if tc.dynlb != nil {
+							rt.retBackendDynamicLB = dynLb
+						}
+						headerMut := &extprocv3.HeaderMutation{}
+						bodyMut := &extprocv3.BodyMutation{}
 
-				var expBody openai.ChatCompletionRequest
-				require.NoError(t, json.Unmarshal(someBody, &expBody))
-				mt := mockTranslator{t: t, expRequestBody: &expBody, retHeaderMutation: headerMut, retBodyMutation: bodyMut}
-				mm := &mockChatCompletionMetrics{}
-				p := &chatCompletionProcessor{
-					config: &processorConfig{
-						router:                   rt,
-						selectedBackendHeaderKey: "x-ai-gateway-backend-key",
-						modelNameHeaderKey:       "x-ai-gateway-model-key",
-					},
-					requestHeaders: headers,
-					logger:         slog.Default(),
-					metrics:        mm,
-					translator:     mt,
-				}
-				resp, err := p.ProcessRequestBody(t.Context(), &extprocv3.HttpBody{Body: someBody})
-				require.NoError(t, err)
-				require.Equal(t, mt, p.translator)
-				require.NotNil(t, resp)
-				commonRes := resp.Response.(*extprocv3.ProcessingResponse_RequestBody).RequestBody.Response
-				require.Equal(t, headerMut, commonRes.HeaderMutation)
-				require.Equal(t, bodyMut, commonRes.BodyMutation)
+						var expBody openai.ChatCompletionRequest
+						require.NoError(t, json.Unmarshal(someBody, &expBody))
+						mt := mockTranslator{t: t, expRequestBody: &expBody, retHeaderMutation: headerMut, retBodyMutation: bodyMut}
+						mm := &mockChatCompletionMetrics{}
+						p := &chatCompletionProcessor{
+							config: &processorConfig{
+								router:                   rt,
+								selectedBackendHeaderKey: "x-ai-gateway-backend-key",
+								modelNameHeaderKey:       "x-ai-gateway-model-key",
+								dynamicLoadBalancers: map[*filterapi.DynamicLoadBalancing]dynlb.DynamicLoadBalancer{
+									dynLb: tc.dynlb,
+								},
+							},
+							requestHeaders: headers,
+							logger:         slog.Default(),
+							metrics:        mm,
+							translator:     mt,
+						}
+						resp, err := p.ProcessRequestBody(t.Context(), &extprocv3.HttpBody{Body: someBody})
+						require.NoError(t, err)
+						require.Equal(t, mt, p.translator)
+						require.NotNil(t, resp)
+						commonRes := resp.Response.(*extprocv3.ProcessingResponse_RequestBody).RequestBody.Response
+						require.Equal(t, headerMut, commonRes.HeaderMutation)
+						require.Equal(t, bodyMut, commonRes.BodyMutation)
 
-				mm.RequireRequestNotCompleted(t)
-				mm.RequireSelected(t, "some-model", "some-backend")
+						mm.RequireRequestNotCompleted(t)
+						mm.RequireSelected(t, "some-model", "some-backend")
 
-				// Check the model and backend headers are set in headerMut.
-				hdrs := headerMut.SetHeaders
-				require.Len(t, hdrs, 2)
-				require.Equal(t, "x-ai-gateway-model-key", hdrs[0].Header.Key)
-				require.Equal(t, "some-model", string(hdrs[0].Header.RawValue))
-				require.Equal(t, "x-ai-gateway-backend-key", hdrs[1].Header.Key)
-				require.Equal(t, "some-backend", string(hdrs[1].Header.RawValue))
-				require.Equal(t, stream, p.stream)
+						// Check the model and backend headers are set in headerMut.
+						hdrs := headerMut.SetHeaders
+						if tc.dynlb != nil {
+							require.Len(t, hdrs, 3)
+							require.Equal(t, "x-ai-gateway-model-key", hdrs[0].Header.Key)
+							require.Equal(t, "some-model", string(hdrs[0].Header.RawValue))
+							require.Equal(t, "x-ai-gateway-backend-key", hdrs[1].Header.Key)
+							require.Equal(t, "original_destination_cluster", string(hdrs[1].Header.RawValue))
+							require.Equal(t, "foo", hdrs[2].Header.Key)
+							require.Equal(t, "bar", hdrs[2].Header.Value)
+						} else {
+							require.Len(t, hdrs, 2)
+							require.Equal(t, "x-ai-gateway-model-key", hdrs[0].Header.Key)
+							require.Equal(t, "some-model", string(hdrs[0].Header.RawValue))
+							require.Equal(t, "x-ai-gateway-backend-key", hdrs[1].Header.Key)
+							require.Equal(t, "some-backend", string(hdrs[1].Header.RawValue))
+						}
+						require.Equal(t, stream, p.stream)
+					})
+				}
 			})
 		})
 	}
diff --git a/internal/extproc/dynlb/dynlb_test.go b/internal/extproc/dynlb/dynlb_test.go
new file mode 100644
index 0000000..dcb7983
--- /dev/null
+++ b/internal/extproc/dynlb/dynlb_test.go
@@ -0,0 +1,104 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package dynlb
+
+import (
+	"log/slog"
+	"testing"
+
+	corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
+	"github.com/stretchr/testify/require"
+
+	"github.com/envoyproxy/ai-gateway/filterapi"
+	internaltesting "github.com/envoyproxy/ai-gateway/internal/testing"
+)
+
+func Test_newDynamicLoadBalancer(t *testing.T) {
+	addr := internaltesting.RequireNewTestDNSServer(t)
+	f := &filterapi.DynamicLoadBalancing{
+		Backends: []filterapi.DynamicLoadBalancingBackend{
+			{
+				IPs:  []string{"1.2.3.4"},
+				Port: 8080,
+			},
+			{
+				Hostnames: []string{"foo.io", "example.com"},
+				Port:      9999,
+			},
+			{
+				Hostnames: []string{"something.io"},
+				Port:      4444,
+			},
+		},
+		Models: []filterapi.DynamicLoadBalancingModel{},
+	}
+
+	_dlb, err := newDynamicLoadBalancer(t.Context(), slog.Default(), f, addr)
+	require.NoError(t, err)
+	dlb, ok := _dlb.(*dynamicLoadBalancer)
+	require.True(t, ok)
+
+	for _, m := range f.Models {
+		require.Equal(t, m, dlb.models[m.Name])
+	}
+	require.ElementsMatch(t, []endpoint{
+		{
+			ipPort:  []byte("1.2.3.4:8080"),
+			backend: &f.Backends[0].Backend,
+		},
+		{
+			ipPort:   []byte("1.1.1.1:9999"),
+			hostname: "foo.io",
+			backend:  &f.Backends[1].Backend,
+		},
+		{
+			ipPort:   []byte("2.2.2.2:9999"),
+			hostname: "example.com",
+			backend:  &f.Backends[1].Backend,
+		},
+		{
+			ipPort:   []byte("3.3.3.3:4444"),
+			hostname: "something.io",
+			backend:  &f.Backends[2].Backend,
+		},
+		{
+			ipPort:   []byte("4.4.4.4:4444"),
+			hostname: "something.io",
+			backend:  &f.Backends[2].Backend,
+		},
+	}, dlb.endpoints)
+}
+
+func TestDynamicLoadBalancingSelectChatCompletionsEndpoint(t *testing.T) {
+	// TODO: currently this is mostly for test coverage, need to add more tests as we add more features.
+	dlb := &dynamicLoadBalancer{
+		logger: slog.Default(),
+		endpoints: []endpoint{
+			{ipPort: []byte("1.1.1.1:8080"), backend: &filterapi.Backend{Name: "foo"}, hostname: "foo.io"},
+		},
+		models: map[string]filterapi.DynamicLoadBalancingModel{"foo": {}},
+	}
+	t.Run("model name not found", func(t *testing.T) {
+		_, _, err := dlb.SelectChatCompletionsEndpoint("aaaaaaaaaaaaa", nil)
+		require.ErrorContains(t, err, "model aaaaaaaaaaaaa is not found in the dynamic load balancer")
+	})
+	t.Run("ok", func(t *testing.T) {
+		backend, headers, err := dlb.SelectChatCompletionsEndpoint("foo", nil)
+		require.NoError(t, err)
+		require.Equal(t, &filterapi.Backend{Name: "foo"}, backend)
+		require.Len(t, headers, 1)
+		for _, h := range []*corev3.HeaderValueOption{
+			{
+				Header: &corev3.HeaderValue{
+					Key:      originalDstHeaderName,
+					RawValue: []byte("1.1.1.1:8080"),
+				},
+			},
+		} {
+			require.Contains(t, headers, h)
+		}
+	})
+}
diff --git a/internal/extproc/mocks_test.go b/internal/extproc/mocks_test.go
index 51bce55..4c13400 100644
--- a/internal/extproc/mocks_test.go
+++ b/internal/extproc/mocks_test.go
@@ -117,13 +117,14 @@ type mockRouter struct {
 	expHeaders            map[string]string
 	retBackendName        string
 	retVersionedAPISchema filterapi.VersionedAPISchema
+	retBackendDynamicLB   *filterapi.DynamicLoadBalancing
 	retErr                error
 }
 
 // Calculate implements [router.Router.Calculate].
 func (m mockRouter) Calculate(headers map[string]string) (*filterapi.Backend, error) {
 	require.Equal(m.t, m.expHeaders, headers)
-	b := &filterapi.Backend{Name: m.retBackendName, Schema: m.retVersionedAPISchema}
+	b := &filterapi.Backend{Name: m.retBackendName, Schema: m.retVersionedAPISchema, DynamicLoadBalancing: m.retBackendDynamicLB}
 	return b, m.retErr
 }
 
@@ -239,3 +240,16 @@ func (m *mockChatCompletionMetrics) RequireTokensRecorded(t *testing.T, count in
 }
 
 var _ x.ChatCompletionMetrics = &mockChatCompletionMetrics{}
+
+// mockDynamicLB implements dynlb.DynamicLoadBalancer for testing.
+type mockDynamicLB struct {
+	backedName string
+	headers    []*corev3.HeaderValueOption
+}
+
+// SelectChatCompletionsEndpoint implements dynlb.DynamicLoadBalancer.
+func (m *mockDynamicLB) SelectChatCompletionsEndpoint(string, x.ChatCompletionMetrics) (
+	selected *filterapi.Backend, headers []*corev3.HeaderValueOption, err error,
+) {
+	return &filterapi.Backend{Name: m.backedName}, m.headers, nil
+}
diff --git a/internal/extproc/watcher_test.go b/internal/extproc/watcher_test.go
index b878611..7515c4c 100644
--- a/internal/extproc/watcher_test.go
+++ b/internal/extproc/watcher_test.go
@@ -162,6 +162,8 @@ rules:
 	}, 1*time.Second, tickInterval)
 	require.NotEqual(t, firstCfg, rcv.getConfig())
 
+	secondConfig := rcv.getConfig()
+
 	// Verify the buffer contains the updated loading.
 	require.Eventually(t, func() bool {
 		return strings.Contains(buf.String(), "loading a new config")
@@ -175,6 +177,36 @@ rules:
 	// Wait for a couple ticks to verify config is not reloaded if file does not change.
 	time.Sleep(2 * tickInterval)
 	require.Equal(t, int32(3), rcv.loadCount.Load())
+
+	// Change it to the dynamic load balancing config.
+	cfg = `
+schema:
+  name: OpenAI
+selectedBackendHeaderKey: x-ai-eg-selected-backend
+modelNameHeaderKey: x-model-name
+rules:
+- backends:
+  - name: openai
+    dynamicLoadBalancing: {} # This should force the reload regardless of the config change.
+    schema:
+      name: OpenAI
+  headers:
+  - name: x-model-name
+    value: gpt4.4444
+`
+	require.NoError(t, os.WriteFile(path, []byte(cfg), 0o600))
+
+	// Verify the config has been updated.
+	require.Eventually(t, func() bool {
+		return rcv.getConfig() != secondConfig
+	}, 1*time.Second, tickInterval)
+	require.NotEqual(t, firstCfg, rcv.getConfig())
+	require.Equal(t, int32(4), rcv.loadCount.Load())
+
+	// Then waits for the 10 counts of the config to be loaded.
+	require.Eventually(t, func() bool {
+		return rcv.loadCount.Load() == 10
+	}, 5*time.Second, tickInterval)
 }
 
 func TestDiff(t *testing.T) {
diff --git a/internal/testing/dnsserver.go b/internal/testing/dnsserver.go
new file mode 100644
index 0000000..d934321
--- /dev/null
+++ b/internal/testing/dnsserver.go
@@ -0,0 +1,89 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+package internaltesting
+
+import (
+	"net"
+	"testing"
+	"time"
+
+	"github.com/miekg/dns"
+	"github.com/stretchr/testify/require"
+)
+
+// RequireNewTestDNSServer starts a new DNS server that responds to all queries with A records
+// with the fixed IP addresses.
+func RequireNewTestDNSServer(t *testing.T) (addr string) {
+	mux := dns.NewServeMux()
+	mux.HandleFunc(".", func(w dns.ResponseWriter, r *dns.Msg) {
+		msg := dns.Msg{}
+		msg.SetReply(r)
+		msg.Authoritative = true
+		for _, q := range r.Question {
+			var ips []string
+			switch q.Qtype {
+			case dns.TypeA:
+				switch q.Name {
+				case "foo.io.":
+					ips = append(ips, "1.1.1.1")
+				case "example.com.":
+					ips = append(ips, "2.2.2.2")
+				case "mylocal.io.":
+					ips = append(ips, "0.0.0.0")
+				default:
+					ips = append(ips, "3.3.3.3")
+					ips = append(ips, "4.4.4.4")
+				}
+			default:
+				t.Fatalf("Unsupported query type: %v", q.Qtype)
+			}
+			for _, ip := range ips {
+				rr, err := dns.NewRR(q.Name + " A " + ip)
+				require.NoError(t, err)
+				msg.Answer = append(msg.Answer, rr)
+			}
+		}
+		require.NoError(t, w.WriteMsg(&msg))
+	})
+	p, err := net.ListenPacket("udp", "0.0.0.0:")
+	require.NoError(t, err)
+	server := &dns.Server{PacketConn: p, Handler: mux}
+	go func() {
+		require.NoError(t, server.ActivateAndServe())
+	}()
+	t.Cleanup(func() {
+		_ = server.ShutdownContext(t.Context())
+	})
+
+	addr = p.LocalAddr().String()
+
+	// Wait for the server to start.
+	require.Eventually(t, func() bool {
+		client := dns.Client{Net: "udp"}
+		msg := new(dns.Msg)
+		msg.SetQuestion("example.com.", dns.TypeA)
+		response, _, err := client.ExchangeContext(t.Context(), msg, addr)
+		if err != nil {
+			t.Logf("Failed to exchange DNS message: %v", err)
+			return false
+		}
+		if response.Rcode != dns.RcodeSuccess {
+			t.Logf("DNS query failed: %s", dns.RcodeToString[response.Rcode])
+			return false
+		}
+		for _, answer := range response.Answer {
+			if aRecord, ok := answer.(*dns.A); ok {
+				if aRecord.A.String() == "2.2.2.2" {
+					return true
+				}
+			}
+			t.Logf("Unexpected answer: %v", answer)
+		}
+		t.Logf("No A record found")
+		return false
+	}, 5*time.Second, 100*time.Millisecond)
+	return
+}
diff --git a/tests/controller/controller_test.go b/tests/controller/controller_test.go
index 697da37..80a6199 100644
--- a/tests/controller/controller_test.go
+++ b/tests/controller/controller_test.go
@@ -513,7 +513,7 @@ func TestBackendSecurityPolicyController(t *testing.T) {
 	opt := ctrl.Options{Scheme: c.Scheme(), LeaderElection: false, Controller: config.Controller{SkipNameValidation: ptr.To(true)}}
 	mgr, err := ctrl.NewManager(cfg, opt)
 	require.NoError(t, err)
-	require.NoError(t, controller.ApplyIndexing(t.Context(), mgr.GetFieldIndexer().IndexField))
+	require.NoError(t, controller.ApplyIndexing(t.Context(), true, mgr.GetFieldIndexer().IndexField))
 
 	pc := controller.NewBackendSecurityPolicyController(mgr.GetClient(), k, defaultLogger(), syncAIServiceBackend.Sync)
 	err = controller.TypedControllerBuilderForCRD(mgr, &aigv1a1.BackendSecurityPolicy{}).Complete(pc)
@@ -645,7 +645,7 @@ func TestAIServiceBackendController(t *testing.T) {
 	opt := ctrl.Options{Scheme: c.Scheme(), LeaderElection: false, Controller: config.Controller{SkipNameValidation: ptr.To(true)}}
 	mgr, err := ctrl.NewManager(cfg, opt)
 	require.NoError(t, err)
-	require.NoError(t, controller.ApplyIndexing(t.Context(), mgr.GetFieldIndexer().IndexField))
+	require.NoError(t, controller.ApplyIndexing(t.Context(), true, mgr.GetFieldIndexer().IndexField))
 
 	bc := controller.NewAIServiceBackendController(mgr.GetClient(), k, defaultLogger(), syncAIGatewayRoute.Sync)
 	err = controller.TypedControllerBuilderForCRD(mgr, &aigv1a1.AIServiceBackend{}).Complete(bc)
@@ -778,7 +778,7 @@ func TestSecretController(t *testing.T) {
 
 	err = ctrl.NewControllerManagedBy(mgr).For(&corev1.Secret{}).Complete(sc)
 	require.NoError(t, err)
-	require.NoError(t, controller.ApplyIndexing(t.Context(), mgr.GetFieldIndexer().IndexField))
+	require.NoError(t, controller.ApplyIndexing(t.Context(), true, mgr.GetFieldIndexer().IndexField))
 
 	go func() { require.NoError(t, mgr.Start(t.Context())) }()
 
diff --git a/tests/crdcel/main_test.go b/tests/crdcel/main_test.go
index 3d2d08b..0455865 100644
--- a/tests/crdcel/main_test.go
+++ b/tests/crdcel/main_test.go
@@ -48,6 +48,10 @@ func TestAIGatewayRoutes(t *testing.T) {
 			name:   "no_target_refs.yaml",
 			expErr: `spec.targetRefs: Invalid value: 0: spec.targetRefs in body should have at least 1 items`,
 		},
+		{
+			name:   "invalid_backendref.yaml",
+			expErr: `AIGatewayRoute.aigateway.envoyproxy.io "invalid-backendref" is invalid: [spec.rules[0].backendRefs[0].kind: Unsupported value: "Foo": supported values: "AIServiceBackend", "InferencePool"`,
+		},
 	} {
 		t.Run(tc.name, func(t *testing.T) {
 			data, err := testdata.ReadFile(path.Join("testdata/aigatewayroutes", tc.name))
diff --git a/tests/crdcel/testdata/aigatewayroutes/invalid_backendref.yaml b/tests/crdcel/testdata/aigatewayroutes/invalid_backendref.yaml
new file mode 100644
index 0000000..c589226
--- /dev/null
+++ b/tests/crdcel/testdata/aigatewayroutes/invalid_backendref.yaml
@@ -0,0 +1,26 @@
+# Copyright Envoy AI Gateway Authors
+# SPDX-License-Identifier: Apache-2.0
+# The full text of the Apache license is available in the LICENSE file at
+# the root of the repo.
+
+apiVersion: aigateway.envoyproxy.io/v1alpha1
+kind: AIGatewayRoute
+metadata:
+  name: invalid-backendref
+  namespace: default
+spec:
+  schema:
+    name: OpenAI
+  targetRefs:
+    - name: some-gateway
+      kind: Gateway
+      group: gateway.networking.k8s.io
+  rules:
+    - matches:
+        - headers:
+            - type: Exact
+              name: x-ai-eg-model
+              value: llama3-70b
+      backendRefs:
+        - name: kserve
+          kind: Foo
diff --git a/tests/e2e/e2e_test.go b/tests/e2e/e2e_test.go
index 3cdd6a8..eddfd43 100644
--- a/tests/e2e/e2e_test.go
+++ b/tests/e2e/e2e_test.go
@@ -60,11 +60,15 @@ func TestMain(m *testing.M) {
 	// They must be idempotent and can be run multiple times so that we can run the tests multiple times on
 	// failures.
 
+	run := false
 	defer func() {
 		// If the setup or some tests panic, try to collect the cluster logs
 		if r := recover(); r != nil {
 			cleanupKindCluster(true)
 		}
+		if !run {
+			panic("BUG: no tests were run. This is likely a bug during the setup")
+		}
 	}()
 
 	if err := initKindCluster(ctx); err != nil {
@@ -77,6 +81,11 @@ func TestMain(m *testing.M) {
 		panic(err)
 	}
 
+	if err := installInferenceExtensionCRD(ctx); err != nil {
+		cancel()
+		panic(err)
+	}
+
 	if err := initAIGateway(ctx); err != nil {
 		cancel()
 		panic(err)
@@ -93,6 +102,7 @@ func TestMain(m *testing.M) {
 	}
 
 	code := m.Run()
+	run = true
 	cancel()
 
 	cleanupKindCluster(code != 0)
@@ -157,6 +167,11 @@ func cleanupKindCluster(testsFailed bool) {
 	}
 }
 
+func installInferenceExtensionCRD(ctx context.Context) (err error) {
+	const infExtURL = "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.2.0/manifests.yaml"
+	return kubectlApplyManifest(ctx, infExtURL)
+}
+
 // initEnvoyGateway initializes the Envoy Gateway in the kind cluster following the quickstart guide:
 // https://gateway.envoyproxy.io/latest/tasks/quickstart/
 func initEnvoyGateway(ctx context.Context) (err error) {
@@ -201,6 +216,7 @@ func initAIGateway(ctx context.Context) (err error) {
 	}()
 	initLog("\tHelm Install")
 	helm := exec.CommandContext(ctx, "go", "tool", "helm", "upgrade", "-i", "ai-eg",
+		"--set", "controller.enableInferenceExtension=true",
 		"../../manifests/charts/ai-gateway-helm",
 		"-n", "envoy-ai-gateway-system", "--create-namespace")
 	helm.Stdout = os.Stdout
diff --git a/tests/e2e/inference_extension_test.go b/tests/e2e/inference_extension_test.go
new file mode 100644
index 0000000..8a6c25b
--- /dev/null
+++ b/tests/e2e/inference_extension_test.go
@@ -0,0 +1,46 @@
+// Copyright Envoy AI Gateway Authors
+// SPDX-License-Identifier: Apache-2.0
+// The full text of the Apache license is available in the LICENSE file at
+// the root of the repo.
+
+//go:build test_e2e
+
+package e2e
+
+import (
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/require"
+)
+
+func Test_Examples_InferenceExtension(t *testing.T) {
+	const manifest = "../../examples/inference_extension/inference_extension.yaml"
+	require.NoError(t, kubectlApplyManifest(t.Context(), manifest))
+
+	const egSelector = "gateway.envoyproxy.io/owning-gateway-name=inference-extension-example"
+	requireWaitForPodReady(t, egNamespace, egSelector)
+
+	fwd := requireNewHTTPPortForwarder(t, egNamespace, egSelector, egDefaultPort)
+	defer fwd.kill()
+	require.Eventually(t, func() bool {
+		requestBody := fmt.Sprintf(`{"messages":[{"role":"user","content":"Say this is a test"}],"model":"mistral:latest"}`)
+		req, err := http.NewRequest(http.MethodPut, fwd.address()+"/v1/chat/completions", strings.NewReader(requestBody))
+		require.NoError(t, err)
+		req.Header.Set("x-target-inference-extension", "yes")
+		resp, err := http.DefaultClient.Do(req)
+		require.NoError(t, err)
+		defer func() { _ = resp.Body.Close() }()
+
+		body, err := io.ReadAll(resp.Body)
+		t.Logf("response: status=%d; headers=%v; body=%s", resp.StatusCode, resp.Header, string(body))
+		if resp.StatusCode != http.StatusOK {
+			return false
+		}
+		return true
+	}, 5*time.Minute, 5*time.Second)
+}
diff --git a/tests/extproc/envoy.yaml b/tests/extproc/envoy.yaml
index 4e9c182..ff4fffb 100644
--- a/tests/extproc/envoy.yaml
+++ b/tests/extproc/envoy.yaml
@@ -96,6 +96,15 @@ static_resources:
                                   exact: testupstream
                           route:
                             cluster: testupstream
+                        - match:
+                            prefix: "/"
+                            headers:
+                              - name: x-selected-backend-name
+                                string_match:
+                                  exact: original_destination_cluster
+                          name: original_destination_cluster
+                          route:
+                            cluster: original_destination_cluster
                 http_filters:
                   - name: envoy.filters.http.ext_proc
                     typed_config:
@@ -204,6 +213,13 @@ static_resources:
         typed_config:
           "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
           sni: <azure_resource_name>.openai.azure.com   # Replace with your azure resource name
+    - name: original_destination_cluster
+      connectTimeout: 30s
+      type: ORIGINAL_DST
+      lbPolicy: CLUSTER_PROVIDED
+      originalDstLbConfig:
+        httpHeaderName: x-ai-eg-original-dst
+        useHttpHeader: true
 
 overload_manager:
   refresh_interval: 0.25s
diff --git a/tests/extproc/testupstream_test.go b/tests/extproc/testupstream_test.go
index 5115519..17110fa 100644
--- a/tests/extproc/testupstream_test.go
+++ b/tests/extproc/testupstream_test.go
@@ -25,6 +25,7 @@ import (
 
 	"github.com/envoyproxy/ai-gateway/filterapi"
 	"github.com/envoyproxy/ai-gateway/internal/apischema/openai"
+	internaltesting "github.com/envoyproxy/ai-gateway/internal/testing"
 	"github.com/envoyproxy/ai-gateway/tests/internal/testupstreamlib"
 )
 
@@ -375,6 +376,104 @@ data: [DONE]
 	})
 }
 
+func TestWithTestUpstreamDynamicLoadBalancing(t *testing.T) {
+	requireBinaries(t)
+	accessLogPath := t.TempDir() + "/access.log"
+	requireRunEnvoy(t, accessLogPath)
+	configPath := t.TempDir() + "/extproc-config.yaml"
+	requireTestUpstream(t)
+
+	const (
+		coolModelID = "cool-model"
+		routeHeader = "x-test-backend"
+	)
+	requireWriteFilterConfig(t, configPath, &filterapi.Config{
+		Schema:                   openAISchema,
+		SelectedBackendHeaderKey: "x-selected-backend-name",
+		ModelNameHeaderKey:       "x-model-name",
+		Rules: []filterapi.RouteRule{
+			{
+				Backends: []filterapi.Backend{{Name: "openai", Schema: openAISchema}},
+				Headers:  []filterapi.HeaderMatch{{Name: routeHeader, Value: "openai"}},
+			},
+			{
+				Backends: []filterapi.Backend{{
+					Name: "dynamic", Schema: openAISchema,
+					DynamicLoadBalancing: &filterapi.DynamicLoadBalancing{
+						Models: []filterapi.DynamicLoadBalancingModel{{Name: coolModelID}},
+						Backends: []filterapi.DynamicLoadBalancingBackend{
+							{
+								Backend:   filterapi.Backend{Name: "testupstream", Schema: openAISchema},
+								Hostnames: []string{"mylocal.io"},
+								Port:      8080, // We route to the testupstream.
+							},
+						},
+					},
+				}},
+				Headers: []filterapi.HeaderMatch{{Name: routeHeader, Value: "dynamic"}},
+			},
+		},
+	})
+
+	// Launch the external processor with the DNS_SERVER_ADDR environment variable set to a new DNS server.
+	requireExtProc(t, os.Stdout, extProcExecutablePath(), configPath,
+		"DNS_SERVER_ADDR="+internaltesting.RequireNewTestDNSServer(t))
+
+	// This ensures that dynamic load balancing can work in conjunction with the other normal backends.
+	t.Run("non dynamic route", func(t *testing.T) {
+		require.Eventually(t, func() bool {
+			req, err := http.NewRequest(http.MethodPost, listenerAddress+"/v1/chat/completions", strings.NewReader(fmt.Sprintf(`
+{"model":"%s","messages":[{"role":"system","content":"You are a chatbot."}], "stream": true}
+`, coolModelID)))
+			require.NoError(t, err)
+			req.Header.Set(routeHeader, "openai")
+
+			resp, err := http.DefaultClient.Do(req)
+			if err != nil {
+				t.Logf("error: %v", err)
+				return false
+			}
+			defer func() { _ = resp.Body.Close() }()
+			if resp.StatusCode != http.StatusUnauthorized {
+				t.Logf("unexpected status code: %d", resp.StatusCode)
+				return false
+			}
+			body, err := io.ReadAll(resp.Body)
+			require.NoError(t, err)
+			t.Logf("response: %s", string(body))
+			return true
+		}, 30*time.Second, 2*time.Second)
+	})
+
+	t.Run("dynamic route", func(t *testing.T) {
+		require.Eventually(t, func() bool {
+			req, err := http.NewRequest(http.MethodPost, listenerAddress+"/v1/chat/completions", strings.NewReader(fmt.Sprintf(`
+{"model":"%s","messages":[{"role":"system","content":"You are a chatbot."}], "stream": true}
+`, coolModelID)))
+			require.NoError(t, err)
+			req.Header.Set(routeHeader, "dynamic")
+
+			resp, err := http.DefaultClient.Do(req)
+			if err != nil {
+				t.Logf("error: %v", err)
+				return false
+			}
+			defer func() { _ = resp.Body.Close() }()
+			if resp.StatusCode != http.StatusOK {
+				t.Logf("unexpected status code: %d", resp.StatusCode)
+				return false
+			}
+			body, err := io.ReadAll(resp.Body)
+			require.NoError(t, err)
+			t.Logf("response: %s", string(body))
+			var openaiResp openai.ChatCompletionResponse
+			require.NoError(t, json.Unmarshal(body, &openaiResp))
+			require.Len(t, openaiResp.Choices, 1)
+			return true
+		}, 30*time.Second, 2*time.Second)
+	})
+}
+
 func checkModelsIgnoringTimestamps(want openai.ModelList) func(t require.TestingT, body []byte) {
 	return func(t require.TestingT, body []byte) {
 		var models openai.ModelList
diff --git a/tests/internal/envtest.go b/tests/internal/envtest.go
index 25ade75..ce1ecaf 100644
--- a/tests/internal/envtest.go
+++ b/tests/internal/envtest.go
@@ -60,6 +60,9 @@ func NewEnvTest(t *testing.T) (c client.Client, cfg *rest.Config, k kubernetes.I
 		crds = append(crds, requireThirdPartyCRDDownloaded(t, path, url))
 	}
 
+	const infExtURL = "https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.2.0/manifests.yaml"
+	crds = append(crds, requireThirdPartyCRDDownloaded(t, "inference_extension_for_tests.yaml", infExtURL))
+
 	env := &envtest.Environment{CRDDirectoryPaths: crds}
 	cfg, err = env.Start()
 	require.NoError(t, err)