Repository: kubernetes-sigs/gateway-api-inference-extension

Issue:
few minor issues in e2e tests
in addition to other e2e related open issue #619 and #486 there are few minor issues in e2e tests as specified by @danehans in one of the comments in PR review - https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/485#pullrequestreview-2682760174

- e2e test should perform cleanup when getting timeout. after further investigation I've noticed the curl command doesn't define a timeout and therefore if the command doesn't work it will hang until the test fails, but will not perform a cleanup since the curl doesn't return an error.
the fix to this issue should be straight forward by adding a timeout to the curl command.

- HF_TOKEN env var should be checked at the beginning of the test setup in order to fail fast in case it's not defined.

Discussion:
@nirrozenbaum: /assign

Solution patch:
diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES
index 6e8e0c5d..933fbe9c 100644
--- a/OWNERS_ALIASES
+++ b/OWNERS_ALIASES
@@ -11,6 +11,9 @@ aliases:
   gateway-api-inference-extension-reviewers:
   - liu-cong
   - robscott
+  - shaneutt
+  - nirrozenbaum
+
 
   wg-serving-leads:
   - ArangoGutierrez
diff --git a/README.md b/README.md
index 2ff00581..b74a13e9 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ It currently requires a version of vLLM that supports the necessary metrics to p
 
 ## Status
 
-This project is [alpha (0.2 release)](https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/tag/v0.2.0).  It should not be used in production yet.
+This project is [alpha (0.3 release)](https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/tag/v0.3.0).  It should not be used in production yet.
 
 ## Getting Started
 
diff --git a/config/charts/body-based-routing/README.md b/config/charts/body-based-routing/README.md
index 062f2b5c..d311b8c3 100644
--- a/config/charts/body-based-routing/README.md
+++ b/config/charts/body-based-routing/README.md
@@ -10,7 +10,7 @@ To install a body-based router named `body-based-router`, you can run the follow
 ```txt
 $ helm install body-based-router ./config/charts/body-based-routing \
     --set provider.name=[gke|istio] \
-    --set inference-gateway.name=inference-gateway
+    --set inferenceGateway.name=inference-gateway
 ```
 
 Note that the provider name is needed to ensure provider-specific manifests are also applied. If no provider is specified, then only
@@ -19,7 +19,7 @@ the deployment and service are deployed.
 To install via the latest published chart in staging  (--version v0 indicates latest dev version), you can run the following command:
 
 ```txt
-$ helm install body-based-router oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/body-based-router \ 
+$ helm install body-based-router oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/body-based-routing \ 
     --version v0
     --set provider.name=[gke|istio]
 ```
@@ -47,8 +47,8 @@ The following table list the configurable parameters of the chart.
 | `bbr.image.tag`              | Image tag.                                                                                                        |
 | `bbr.image.pullPolicy`       | Image pull policy for the container. Possible values: `Always`, `IfNotPresent`, or `Never`. Defaults to `Always`. |
 | `provider.name`              | Name of the Inference Gateway implementation being used. Possible values: `istio`, `gke`. Defaults to `none`.     |
-| `inference-gateway.name`     | The name of the Gateway. Defaults to `inference-gateway`.                                                         |                        
+| `inferenceGateway.name`      | The name of the Gateway. Defaults to `inference-gateway`.                                                         |                        
 
 ## Notes
 
-This chart should only be deployed once per Gateway.
\ No newline at end of file
+This chart should only be deployed once per Gateway.
diff --git a/config/charts/body-based-routing/templates/gke.yaml b/config/charts/body-based-routing/templates/gke.yaml
index 937bfa0b..77b776a4 100644
--- a/config/charts/body-based-routing/templates/gke.yaml
+++ b/config/charts/body-based-routing/templates/gke.yaml
@@ -9,7 +9,7 @@ spec:
   targetRefs:
   - group: "gateway.networking.k8s.io"
     kind: Gateway
-    name: {{ .Values.inference-gateway.name }}
+    name: {{ .Values.inferenceGateway.name }}
   extensionChains:
   - name: chain1
     extensions:
diff --git a/config/charts/body-based-routing/templates/istio.yaml b/config/charts/body-based-routing/templates/istio.yaml
index c4c1444f..6d4535cc 100644
--- a/config/charts/body-based-routing/templates/istio.yaml
+++ b/config/charts/body-based-routing/templates/istio.yaml
@@ -25,9 +25,9 @@ spec:
           processing_mode:
             request_header_mode: "SEND"
             response_header_mode: "SKIP"
-            request_body_mode: "BUFFERED"
+            request_body_mode: "FULL_DUPLEX_STREAMED"
             response_body_mode: "NONE"
-            request_trailer_mode: "SKIP"
+            request_trailer_mode: "SEND"
             response_trailer_mode: "SKIP"
           grpc_service:
             envoy_grpc:
diff --git a/config/charts/body-based-routing/values.yaml b/config/charts/body-based-routing/values.yaml
index b77d7542..0b88dc43 100644
--- a/config/charts/body-based-routing/values.yaml
+++ b/config/charts/body-based-routing/values.yaml
@@ -12,5 +12,5 @@ bbr:
 provider:
   name: none
 
-inference-gateway:
+inferenceGateway:
   name: inference-gateway
diff --git a/config/manifests/gateway/gke/healthcheck.yaml b/config/manifests/gateway/gke/healthcheck.yaml
index 95f4f2d2..93b6cd7f 100644
--- a/config/manifests/gateway/gke/healthcheck.yaml
+++ b/config/manifests/gateway/gke/healthcheck.yaml
@@ -7,7 +7,7 @@ spec:
   targetRef:
     group: "inference.networking.x-k8s.io"
     kind: InferencePool
-    name: vllm-llama2-7b
+    name: vllm-llama3-8b-instruct
   default:
     config:
       type: HTTP
diff --git a/config/manifests/gateway/httproute.yaml b/config/manifests/gateway/gke/httproute.yaml
similarity index 100%
rename from config/manifests/gateway/httproute.yaml
rename to config/manifests/gateway/gke/httproute.yaml
diff --git a/config/manifests/gateway/httproute-with-timeout.yaml b/config/manifests/gateway/istio/httproute.yaml
similarity index 100%
rename from config/manifests/gateway/httproute-with-timeout.yaml
rename to config/manifests/gateway/istio/httproute.yaml
diff --git a/config/manifests/gateway/kgateway/httproute.yaml b/config/manifests/gateway/kgateway/httproute.yaml
new file mode 100644
index 00000000..03967729
--- /dev/null
+++ b/config/manifests/gateway/kgateway/httproute.yaml
@@ -0,0 +1,21 @@
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: llm-route
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: inference-gateway
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: vllm-llama3-8b-instruct
+      port: 8000 # Remove when https://github.com/kgateway-dev/kgateway/issues/10987 is fixed.
+    matches:
+    - path:
+        type: PathPrefix
+        value: /
+    timeouts:
+      request: 300s
diff --git a/config/manifests/vllm/gpu-deployment.yaml b/config/manifests/vllm/gpu-deployment.yaml
index 3386a791..e7cb193e 100644
--- a/config/manifests/vllm/gpu-deployment.yaml
+++ b/config/manifests/vllm/gpu-deployment.yaml
@@ -77,7 +77,7 @@ spec:
               #exec:
               #  command:
               #  - /usr/bin/sleep
-              #  - 30
+              #  - "30"
           livenessProbe:
             httpGet:
               path: /health
@@ -133,7 +133,6 @@ spec:
               path: /health
               port: http
               scheme: HTTP
-
           resources:
             limits:
               nvidia.com/gpu: 1
@@ -235,12 +234,12 @@ spec:
           emptyDir: {}
         - name: config-volume
           configMap:
-            name: vllm-llama3-8b-adapters
+            name: vllm-llama3-8b-instruct-adapters
 ---
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: vllm-llama3-8b-adapters
+  name: vllm-llama3-8b-instruct-adapters
 data:
   configmap.yaml: |
       vLLMLoRAConfig:
diff --git a/hack/push-chart.sh b/hack/push-chart.sh
index e0938af4..36ed92cd 100755
--- a/hack/push-chart.sh
+++ b/hack/push-chart.sh
@@ -30,7 +30,7 @@ CHART=${CHART:-inferencepool}
 
 HELM=${HELM:-./bin/helm}
 
-readonly semver_regex='^v([0-9]+)(\.[0-9]+){1,2}$'
+readonly semver_regex='^v([0-9]+)(\.[0-9]+){1,2}(-rc.[0-9]+)?$'
 
 chart_version=${CHART_VERSION}
 if [[ ${EXTRA_TAG} =~ ${semver_regex} ]]
diff --git a/hack/release-quickstart.sh b/hack/release-quickstart.sh
index 832bd872..c2c0f74d 100755
--- a/hack/release-quickstart.sh
+++ b/hack/release-quickstart.sh
@@ -36,19 +36,26 @@ sed -i.bak -E "s|(releases/download/)v[0-9]+\.[0-9]+\.0-rc\.?[0-9]+|\1${RELEASE_
 sed -i.bak "s|kubectl apply -k https://github.com/kubernetes-sigs/gateway-api-inference-extension/config/crd|kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/${RELEASE_TAG}/manifests.yaml|g" "$README"
 
 # -----------------------------------------------------------------------------
-# Update config/manifests/ext_proc.yaml
+# Update image references
 # -----------------------------------------------------------------------------
-EXT_PROC="config/manifests/ext_proc.yaml"
-echo "Updating ${EXT_PROC} ..."
+EPP="config/manifests/inferencepool-resources.yaml"
+#TODO: Put all helm values files into an array to loop over
+EPP_HELM="config/charts/inferencepool/values.yaml"
+BBR_HELM="config/charts/body-based-routing/values.yaml"
+echo "Updating ${EPP} & ${EPP_HELM} ..."
 
-# Update the EPP container tag.
-sed -i.bak -E "s|(us-central1-docker\.pkg\.dev/k8s-staging-images/gateway-api-inference-extension/epp:)[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$EXT_PROC"
+# Update the container tag.
+sed -i.bak -E "s|(us-central1-docker\.pkg\.dev/k8s-staging-images/gateway-api-inference-extension/epp:)[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$EPP"
+sed -i.bak -E "s|(tag: )[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$EPP_HELM"
+sed -i.bak -E "s|(tag: )[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$BBR_HELM"
 
-# Update the EPP container image pull policy.
-sed -i.bak '/us-central1-docker.pkg.dev\/k8s-staging-images\/gateway-api-inference-extension\/epp/ { n; s/Always/IfNotPresent/ }' "$EXT_PROC"
+# Update the container image pull policy.
+sed -i.bak '/us-central1-docker.pkg.dev\/k8s-staging-images\/gateway-api-inference-extension\/epp/ { n; s/Always/IfNotPresent/ }' "$EPP"
 
-# Update the EPP container registry.
-sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$EXT_PROC"
+# Update the container registry.
+sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$EPP"
+sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$EPP_HELM"
+sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$BBR_HELM"
 
 # -----------------------------------------------------------------------------
 # Update config/manifests/vllm/gpu-deployment.yaml
@@ -65,8 +72,8 @@ sed -i.bak '/vllm\/vllm-openai/ { n; s/Always/IfNotPresent/ }' "$VLLM_DEPLOY"
 # -----------------------------------------------------------------------------
 # Stage the changes
 # -----------------------------------------------------------------------------
-echo "Staging $README $EXT_PROC $VLLM_DEPLOY files..."
-git add $README $EXT_PROC $VLLM_DEPLOY
+echo "Staging $README $EPP $EPP_HELM $BBR_HELM $VLLM_DEPLOY files..."
+git add $README $EPP $EPP_HELM $BBR_HELM $VLLM_DEPLOY
 
 # -----------------------------------------------------------------------------
 # Cleanup backup files and finish
diff --git a/mkdocs.yml b/mkdocs.yml
index 2dc4d2a1..b67cf8b4 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -10,7 +10,7 @@ theme:
   icon:
     repo: fontawesome/brands/git-alt
   logo: images/logo/logo-text-large-horizontal-white.png
-  favicon: images/k8s-favicon.png
+  favicon: images/favicon-64.png
   features:
     - search.highlight
     - navigation.tabs
diff --git a/pkg/body-based-routing/handlers/server.go b/pkg/body-based-routing/handlers/server.go
index 24664f98..484b3318 100644
--- a/pkg/body-based-routing/handlers/server.go
+++ b/pkg/body-based-routing/handlers/server.go
@@ -114,16 +114,16 @@ func (s *Server) processRequestBody(ctx context.Context, body *extProcPb.HttpBod
 
 	var requestBody map[string]interface{}
 	if s.streaming {
+		streamedBody.body = append(streamedBody.body, body.Body...)
 		// In the stream case, we can receive multiple request bodies.
-		if !body.EndOfStream {
-			streamedBody.body = append(streamedBody.body, body.Body...)
-			return nil, nil
-		} else {
+		if body.EndOfStream {
 			loggerVerbose.Info("Flushing stream buffer")
 			err := json.Unmarshal(streamedBody.body, &requestBody)
 			if err != nil {
 				logger.V(logutil.DEFAULT).Error(err, "Error unmarshaling request body")
 			}
+		} else {
+			return nil, nil
 		}
 	} else {
 		if err := json.Unmarshal(body.GetBody(), &requestBody); err != nil {
diff --git a/pkg/epp/backend/metrics/logger.go b/pkg/epp/backend/metrics/logger.go
index 8c73d488..d9a93027 100644
--- a/pkg/epp/backend/metrics/logger.go
+++ b/pkg/epp/backend/metrics/logger.go
@@ -32,6 +32,7 @@ const (
 	// Note currently the EPP treats stale metrics same as fresh.
 	// TODO: https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/336
 	metricsValidityPeriod = 5 * time.Second
+	debugPrintInterval    = 5 * time.Second
 )
 
 type Datastore interface {
@@ -46,16 +47,15 @@ type Datastore interface {
 // enabled; 2) flushes Prometheus metrics about the backend servers.
 func StartMetricsLogger(ctx context.Context, datastore Datastore, refreshPrometheusMetricsInterval time.Duration) {
 	logger := log.FromContext(ctx)
-
-	// Periodically flush prometheus metrics for inference pool
+	ticker := time.NewTicker(refreshPrometheusMetricsInterval)
 	go func() {
+		defer ticker.Stop()
 		for {
 			select {
 			case <-ctx.Done():
 				logger.V(logutil.DEFAULT).Info("Shutting down prometheus metrics thread")
 				return
-			default:
-				time.Sleep(refreshPrometheusMetricsInterval)
+			case <-ticker.C: // Periodically flush prometheus metrics for inference pool
 				flushPrometheusMetricsOnce(logger, datastore)
 			}
 		}
@@ -64,13 +64,14 @@ func StartMetricsLogger(ctx context.Context, datastore Datastore, refreshPrometh
 	// Periodically print out the pods and metrics for DEBUGGING.
 	if logger := logger.V(logutil.DEBUG); logger.Enabled() {
 		go func() {
+			ticker := time.NewTicker(debugPrintInterval)
+			defer ticker.Stop()
 			for {
 				select {
 				case <-ctx.Done():
 					logger.V(logutil.DEFAULT).Info("Shutting down metrics logger thread")
 					return
-				default:
-					time.Sleep(5 * time.Second)
+				case <-ticker.C:
 					podsWithFreshMetrics := datastore.PodList(func(pm PodMetrics) bool {
 						return time.Since(pm.GetMetrics().UpdateTime) <= metricsValidityPeriod
 					})
@@ -110,4 +111,5 @@ func flushPrometheusMetricsOnce(logger logr.Logger, datastore Datastore) {
 	podTotalCount := len(podMetrics)
 	metrics.RecordInferencePoolAvgKVCache(pool.Name, kvCacheTotal/float64(podTotalCount))
 	metrics.RecordInferencePoolAvgQueueSize(pool.Name, float64(queueTotal/podTotalCount))
+	metrics.RecordinferencePoolReadyPods(pool.Name, float64(podTotalCount))
 }
diff --git a/pkg/epp/backend/metrics/pod_metrics.go b/pkg/epp/backend/metrics/pod_metrics.go
index cfb6b138..c85d4d79 100644
--- a/pkg/epp/backend/metrics/pod_metrics.go
+++ b/pkg/epp/backend/metrics/pod_metrics.go
@@ -84,21 +84,19 @@ func (pm *podMetrics) startRefreshLoop() {
 	pm.once.Do(func() {
 		go func() {
 			pm.logger.V(logutil.DEFAULT).Info("Starting refresher", "pod", pm.GetPod())
+			ticker := time.NewTicker(pm.interval)
+			defer ticker.Stop()
 			for {
 				select {
 				case <-pm.done:
 					return
 				case <-pm.parentCtx.Done():
 					return
-				default:
+				case <-ticker.C: // refresh metrics periodically
+					if err := pm.refreshMetrics(); err != nil {
+						pm.logger.V(logutil.TRACE).Error(err, "Failed to refresh metrics", "pod", pm.GetPod())
+					}
 				}
-
-				err := pm.refreshMetrics()
-				if err != nil {
-					pm.logger.V(logutil.TRACE).Error(err, "Failed to refresh metrics", "pod", pm.GetPod())
-				}
-
-				time.Sleep(pm.interval)
 			}
 		}()
 	})
diff --git a/pkg/epp/handlers/server.go b/pkg/epp/handlers/server.go
index cd354c2f..a92f091c 100644
--- a/pkg/epp/handlers/server.go
+++ b/pkg/epp/handlers/server.go
@@ -228,6 +228,7 @@ type RequestContext struct {
 	ResponseSize              int
 	ResponseComplete          bool
 	ResponseStatusCode        string
+	RequestRunning            bool
 
 	RequestState         StreamRequestState
 	modelServerStreaming bool
diff --git a/pkg/epp/handlers/streamingserver.go b/pkg/epp/handlers/streamingserver.go
index d704578a..874dd734 100644
--- a/pkg/epp/handlers/streamingserver.go
+++ b/pkg/epp/handlers/streamingserver.go
@@ -81,13 +81,16 @@ func (s *StreamingServer) Process(srv extProcPb.ExternalProcessor_ProcessServer)
 	// error metrics. This doesn't cover the error "Cannot receive stream request" because
 	// such errors might happen even though response is processed.
 	var err error
-	defer func(error) {
+	defer func(error, *RequestContext) {
 		if reqCtx.ResponseStatusCode != "" {
 			metrics.RecordRequestErrCounter(reqCtx.Model, reqCtx.ResolvedTargetModel, reqCtx.ResponseStatusCode)
 		} else if err != nil {
 			metrics.RecordRequestErrCounter(reqCtx.Model, reqCtx.ResolvedTargetModel, errutil.CanonicalCode(err))
 		}
-	}(err)
+		if reqCtx.RequestRunning {
+			metrics.DecRunningRequests(reqCtx.Model)
+		}
+	}(err, reqCtx)
 
 	for {
 		select {
@@ -269,6 +272,8 @@ func (r *RequestContext) updateStateAndSendIfNeeded(srv extProcPb.ExternalProces
 			return status.Errorf(codes.Unknown, "failed to send response back to Envoy: %v", err)
 		}
 		r.RequestState = BodyRequestResponsesComplete
+		metrics.IncRunningRequests(r.Model)
+		r.RequestRunning = true
 		// Dump the response so a new stream message can begin
 		r.reqBodyResp = nil
 	}
diff --git a/pkg/epp/metrics/metrics.go b/pkg/epp/metrics/metrics.go
index e86ca901..434b8381 100644
--- a/pkg/epp/metrics/metrics.go
+++ b/pkg/epp/metrics/metrics.go
@@ -121,6 +121,16 @@ var (
 		[]string{"model_name", "target_model_name"},
 	)
 
+	runningRequests = compbasemetrics.NewGaugeVec(
+		&compbasemetrics.GaugeOpts{
+			Subsystem:      InferenceModelComponent,
+			Name:           "running_requests",
+			Help:           "Inference model number of running requests in each model.",
+			StabilityLevel: compbasemetrics.ALPHA,
+		},
+		[]string{"model_name"},
+	)
+
 	// Inference Pool Metrics
 	inferencePoolAvgKVCache = compbasemetrics.NewGaugeVec(
 		&compbasemetrics.GaugeOpts{
@@ -141,6 +151,16 @@ var (
 		},
 		[]string{"name"},
 	)
+
+	inferencePoolReadyPods = compbasemetrics.NewGaugeVec(
+		&compbasemetrics.GaugeOpts{
+			Subsystem:      InferencePoolComponent,
+			Name:           "ready_pods",
+			Help:           "The number of ready pods in the inference server pool.",
+			StabilityLevel: compbasemetrics.ALPHA,
+		},
+		[]string{"name"},
+	)
 )
 
 var registerMetrics sync.Once
@@ -155,9 +175,11 @@ func Register() {
 		legacyregistry.MustRegister(responseSizes)
 		legacyregistry.MustRegister(inputTokens)
 		legacyregistry.MustRegister(outputTokens)
+		legacyregistry.MustRegister(runningRequests)
 
 		legacyregistry.MustRegister(inferencePoolAvgKVCache)
 		legacyregistry.MustRegister(inferencePoolAvgQueueSize)
+		legacyregistry.MustRegister(inferencePoolReadyPods)
 	})
 }
 
@@ -209,6 +231,20 @@ func RecordOutputTokens(modelName, targetModelName string, size int) {
 	}
 }
 
+// IncRunningRequests increases the current running requests.
+func IncRunningRequests(modelName string) {
+	if modelName != "" {
+		runningRequests.WithLabelValues(modelName).Inc()
+	}
+}
+
+// DecRunningRequests decreases the current running requests.
+func DecRunningRequests(modelName string) {
+	if modelName != "" {
+		runningRequests.WithLabelValues(modelName).Dec()
+	}
+}
+
 func RecordInferencePoolAvgKVCache(name string, utilization float64) {
 	inferencePoolAvgKVCache.WithLabelValues(name).Set(utilization)
 }
@@ -216,3 +252,7 @@ func RecordInferencePoolAvgKVCache(name string, utilization float64) {
 func RecordInferencePoolAvgQueueSize(name string, queueSize float64) {
 	inferencePoolAvgQueueSize.WithLabelValues(name).Set(queueSize)
 }
+
+func RecordinferencePoolReadyPods(name string, runningPods float64) {
+	inferencePoolReadyPods.WithLabelValues(name).Set(runningPods)
+}
diff --git a/pkg/epp/server/controller_manager.go b/pkg/epp/server/controller_manager.go
index 41fe86a9..aaad8976 100644
--- a/pkg/epp/server/controller_manager.go
+++ b/pkg/epp/server/controller_manager.go
@@ -36,7 +36,7 @@ var scheme = runtime.NewScheme()
 
 func init() {
 	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
-	utilruntime.Must(v1alpha2.AddToScheme(scheme))
+	utilruntime.Must(v1alpha2.Install(scheme))
 }
 
 // DefaultManagerOptions returns the default options used to create the manager.
diff --git a/pkg/epp/server/runserver.go b/pkg/epp/server/runserver.go
index a6c9f1d3..7ed183be 100644
--- a/pkg/epp/server/runserver.go
+++ b/pkg/epp/server/runserver.go
@@ -76,6 +76,7 @@ func NewDefaultExtProcServerRunner() *ExtProcServerRunner {
 		PoolName:                                 DefaultPoolName,
 		PoolNamespace:                            DefaultPoolNamespace,
 		SecureServing:                            DefaultSecureServing,
+		RefreshPrometheusMetricsInterval:         DefaultRefreshPrometheusMetricsInterval,
 		// Datastore can be assigned later.
 	}
 }
diff --git a/site-src/guides/implementers.md b/site-src/guides/implementers.md
index 5d1c6267..7bfd536a 100644
--- a/site-src/guides/implementers.md
+++ b/site-src/guides/implementers.md
@@ -1,3 +1,113 @@
 # Implementer's Guide
 
-TODO
\ No newline at end of file
+This guide is intended for developers looking to implement support for the InferencePool custom resources within their Gateway API controller. It outlines how InferencePool fits into the existing resource model, discusses implementation options, explains how to interact with extensions, and provides guidance on testing.
+
+## InferencePool as a Gateway Backend
+Before we dive into the implementation, let’s recap how an InferencePool works. 
+
+<img src="/images/inference-overview.svg" alt="Overview of API integration" class="center" width="1000" />
+
+**InferencePool** represents a set of Inference-focused Pods and an extension that will be used to route to them. The InferencePool introduces a new type of backend within the Gateway API resource model. Instead of targeting Services, a Gateway can route traffic to an InferencePool. This InferencePool then becomes responsible for intelligent routing to the underlying model server pods based on the associated InferenceModel configurations. 
+
+Here is an example of how to route traffic to an InferencePool using an HTTPRoute:
+```
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: llm-route
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: inference-gateway
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: base-model
+    matches:
+    - path:
+        type: PathPrefix
+        value: /
+```
+
+Note that the `rules.backendRefs` describes which InferencePool should receive the forwarded traffic when the path matches the corresponding path prefix. This is very similar to how we configure a Gateway with an HTTPRoute that directs traffic to a Service (a way to select Pods and specify a port). By using the InferencePool, it provides an abstraction over a set of compute resources (model server pods), and allows the controller to implement specialized routing strategies for these inference workloads.
+
+## Building the Gateway controller 
+The general idea of implementing a Gateway controller supporting the InferencePool involves two major steps: 
+
+1. Tracking the endpoints for InferencePool backends 
+2. Callout to an extension to make intelligent routing decisions
+
+### Endpoint Tracking
+Consider a simple inference pool like this:
+```
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: vllm-llama3-8b-instruct
+spec:
+  targetPortNumber: 8000
+  selector:
+    app: vllm-llama3-8b-instruct
+  extensionRef:
+    name: vllm-llama3-8b-instruct-epp
+```
+
+There are mainly two options for how to treat the Inference Pool in your controller.
+
+**Option 1: Shadow Service Creation**
+
+If your Gateway controller already handles Service as a backend, you can choose to create a headless Service that mirrors the endpoints defined by the InferencePool, like this:
+
+```
+apiVersion: v1
+kind: Service
+metadata: 
+  name: vllm-llama3-8b-instruct-shadow-service
+spec:
+  ports:
+  - port: 54321
+    protocol: TCP
+    targetPort: 8000
+  selector:
+    app:  vllm-llama3-8b-instruct
+  type: ClusterIP
+  clusterIP: None
+```
+
+The gateway controller would then treat this shadow service just like any other backend service it routes traffic to. 
+
+This approach likely allows you to leverage existing service discovery, healthcheck infrastructure, and load balancing mechanisms that your controller already supports. However, it does come with the overhead of managing additional Service objects, and hence may affect the overall latency of the reconciliation of the Gateways.
+
+**Option 2: Tracking InferencePool Endpoints Separately**
+
+You can also choose to directly select and monitor the endpoints belonging to the InferencePool. For the simple inference pool example we have above, the controller would use the label `app: vllm-llama3-8b-instruct` to discover the pods matching the criteria, and get their endpoints (i.e. IP and port number). It would then need to monitor these pods for health and availability. 
+
+With this approach, you can tailor the endpoint tracking and routing logic specifically to the characteristics and requirements of your InferencePool.
+
+### Callout Extension
+
+The [Endpoint Picker](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp), or EPP, is a core component of the inference extension. The primary interaction for routing requests is defined between the proxy (e.g., Envoy) and the EPP using the Envoy [external processing service protocol](https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/ext_proc/v3/external_processor.proto). See the [Endpoint Picker Protocol](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/004-endpoint-picker-protocol) for more information.
+
+#### How to Callout to EPP
+
+For each HTTP request, the proxy CAN communicate the subset of endpoints the EPP MUST pick from by setting `x-gateway-destination-endpoint-subset` key in the filter metadata field of the ext-proc request. If this key is set, the EPP must select from this endpoint list. If the list is empty or no endpoints are eligible, it should return a 503 error. If the key isn't set, the EPP selects from the endpoints defined by the InferencePool selector.
+
+#### Response from the extension
+
+The EPP communicates the chosen endpoint to the proxy via the `x-gateway-destination-endpoint` HTTP header and the `dynamic_metadata` field of the ext-proc response. Failure to communicate the endpoint using both methods results in a 503 error if no endpoints are ready, or a 429 error if the request should be dropped. The header and metadata values must match. In addition to the chosen endpoint, a single fallback endpoint CAN be set using the key `x-gateway-destination-endpoint-fallback` in the same metadata namespace as one used for `x-gateway-destination-endpoint`.
+
+## Testing Tips
+
+Here are some tips for testing your controller end-to-end:
+
+- **Focus on Key Scenarios**: Add common scenarios like creating, updating, and deleting InferencePool resources, as well as different routing rules that target InferencePool backends.
+- **Verify Routing Behaviors**: Design more complex routing scenarios and verify that requests are correctly routed to the appropriate model server pods within the InferencePool based on the InferenceModel configuration.
+- **Test Error Handling**: Verify that the controller correctly handles scenarios like unsupported model names or resource constraints (if criticality-based shedding is implemented). Test with state transitions (such as constant requests while Pods behind EPP are being replaced and Pods behind InferencePool are being replaced) to ensure that the system is resilient to failures and can automatically recover by redirecting traffic to healthy Pods.
+- **Using Reference EPP Implementation + Echoserver**: You can use the [reference EPP implementation](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp) for testing your controller end-to-end. Instead of a full-fledged model server, a simple mock server (like the [echoserver](https://github.com/kubernetes-sigs/ingress-controller-conformance/tree/master/images/echoserver)) can be very useful for verifying routing to ensure the correct pod received the request. 
+- **Performance Test**: Run end-to-end [benchmarks](https://gateway-api-inference-extension.sigs.k8s.io/performance/benchmark/) to make sure that your inference gateway can achieve the latency target that is desired.
+
+### Conformance Tests
+
+A set of conformance tests will be developed soon to help verify that a controller is working as expected. This guide will be updated once we have more information. Stay tuned!
diff --git a/site-src/guides/index.md b/site-src/guides/index.md
index 4548d5cd..367ca902 100644
--- a/site-src/guides/index.md
+++ b/site-src/guides/index.md
@@ -7,11 +7,12 @@
 This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running! 
 
 ## **Prerequisites**
- - A cluster with:
-    - Support for services of type `LoadBalancer`. (This can be validated by ensuring your Envoy Gateway is up and running).
-   For example, with Kind, you can follow [these steps](https://kind.sigs.k8s.io/docs/user/loadbalancer).
-    - Support for [sidecar containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) (enabled by default since Kubernetes v1.29)
-   to run the model server deployment.
+
+- A cluster with:
+  - Support for services of type `LoadBalancer`. For kind clusters, follow [this guide](https://kind.sigs.k8s.io/docs/user/loadbalancer)
+  to get services of type LoadBalancer working.
+  - Support for [sidecar containers](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) (enabled by default since Kubernetes v1.29)
+  to run the model server deployment.
 
 ## **Steps**
 
@@ -105,6 +106,24 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          inference-gateway   inference-gateway   <MY_ADDRESS>    True         22s
          ```
 
+      3. Deploy the HTTPRoute
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/httproute.yaml
+         ```
+
+      4. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
+
+         ```bash
+         kubectl get httproute llm-route -o yaml
+         ```
+
+      5. Given that the default connection timeout may be insufficient for most inference workloads, it is recommended to configure a timeout appropriate for your intended use case.
+
+      ```bash
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gcp-backend-policy.yaml
+      ```
+
 === "Istio"
 
       Please note that this feature is currently in an experimental phase and is not intended for production use. 
@@ -114,7 +133,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
          - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
 
-      1. Install Istio
+      2. Install Istio
       
          ```
          TAG=1.26-alpha.80c74f7f43482c226f4f4b10b4dda6261b67a71f
@@ -131,19 +150,19 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          ./istioctl install --set tag=$TAG --set hub=gcr.io/istio-testing
          ```
 
-      1. If you run the Endpoint Picker (EPP) with the `--secureServing` flag set to `true` (the default mode), it is currently using a self-signed certificate. As a security measure, Istio does not trust self-signed certificates by default. As a temporary workaround, you can apply the destination rule to bypass TLS verification for EPP. A more secure TLS implementation in EPP is being discussed in [Issue 582](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/582).
+      3. If you run the Endpoint Picker (EPP) with the `--secureServing` flag set to `true` (the default mode), it is currently using a self-signed certificate. As a security measure, Istio does not trust self-signed certificates by default. As a temporary workaround, you can apply the destination rule to bypass TLS verification for EPP. A more secure TLS implementation in EPP is being discussed in [Issue 582](https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/582).
 
          ```bash
          kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml
          ```
 
-      1. Deploy Gateway
+      4. Deploy Gateway
 
          ```bash
          kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml
          ```
 
-      1. Label the gateway
+      5. Label the gateway
 
          ```bash
          kubectl label gateway llm-gateway istio.io/enable-inference-extproc=true
@@ -156,9 +175,21 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          inference-gateway   inference-gateway   <MY_ADDRESS>    True         22s
          ```
 
+      6. Deploy the HTTPRoute
+
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/httproute.yaml
+         ```
+
+      7. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
+
+         ```bash
+         kubectl get httproute llm-route -o yaml
+         ```
+
 === "Kgateway"
 
-      [Kgateway](https://kgateway.dev/) v2.0.0 adds support for inference extension as a **technical preview**. This means do not
+      [Kgateway](https://kgateway.dev/) recently added support for inference extension as a **technical preview**. This means do not
       run Kgateway with inference extension in production environments. Refer to [Issue 10411](https://github.com/kgateway-dev/kgateway/issues/10411)
       for the list of caveats, supported features, etc.
 
@@ -167,20 +198,20 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          - [Helm](https://helm.sh/docs/intro/install/) installed.
          - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
 
-      1. Install Kgateway CRDs
+      2. Set the Kgateway version and install the Kgateway CRDs.
 
          ```bash
-         helm upgrade -i --create-namespace --namespace kgateway-system --version $VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
+         KGTW_VERSION=v2.0.0
+         helm upgrade -i --create-namespace --namespace kgateway-system --version $KGTW_VERSION kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds
          ```
 
-      1. Install Kgateway
+      3. Install Kgateway
 
          ```bash
-         helm upgrade -i --namespace kgateway-system --version $VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway
-         --set inferenceExtension.enabled=true
+         helm upgrade -i --namespace kgateway-system --version $KGTW_VERSION kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --set inferenceExtension.enabled=true
          ```
 
-      1. Deploy Gateway
+      4. Deploy the Gateway
 
          ```bash
          kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml
@@ -193,50 +224,50 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          inference-gateway   kgateway            <MY_ADDRESS>    True         22s
          ```
 
-### Deploy the HTTPRoute
+      5. Deploy the HTTPRoute
 
-   ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute.yaml
-   ```
+         ```bash
+         kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/httproute.yaml
+         ```
 
-### Configure Timeouts
+      6. Confirm that the HTTPRoute status conditions include `Accepted=True` and `ResolvedRefs=True`:
 
-   Given that default timeouts for above implementations may be insufficient for most inference workloads, it is recommended to configure a timeout appropriate for your intended use case.
+         ```bash
+         kubectl get httproute llm-route -o yaml
+         ```
 
-=== "GKE"
+### Try it out
 
-      ```bash
-      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gcp-backend-policy.yaml
-      ```
+   Wait until the gateway is ready.
 
-=== "Istio"
+=== "GPU-Based Model Server"
 
       ```bash
-      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute-with-timeout.yaml
+      IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
+      PORT=80
+
+      curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
+      "model": "food-review",
+      "prompt": "Write as if you were a critic: San Francisco",
+      "max_tokens": 100,
+      "temperature": 0
+      }'
       ```
 
-=== "Kgateway"
+=== "CPU-Based Model Server"
 
       ```bash
-      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute-with-timeout.yaml
+      IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
+      PORT=80
+
+      curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
+      "model": "Qwen/Qwen2.5-1.5B-Instruct",
+      "prompt": "Write as if you were a critic: San Francisco",
+      "max_tokens": 100,
+      "temperature": 0
+      }'
       ```
 
-### Try it out
-
-   Wait until the gateway is ready.
-
-   ```bash
-   IP=$(kubectl get gateway/inference-gateway -o jsonpath='{.status.addresses[0].value}')
-   PORT=80
-
-   curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
-   "model": "food-review",
-   "prompt": "Write as if you were a critic: San Francisco",
-   "max_tokens": 100,
-   "temperature": 0
-   }'
-   ```
-
 ### Cleanup
 
    The following cleanup assumes you would like to clean ALL resources that were created in this quickstart guide.  
@@ -258,10 +289,12 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gateway.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/healthcheck.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/gcp-backend-policy.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/gke/httproute.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/gateway.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/destination-rule.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/istio/httproute.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/gateway.yaml --ignore-not-found
-      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/httproute.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/gateway/kgateway/httproute.yaml --ignore-not-found
       ```
 
    1. Uninstall the CRDs
diff --git a/site-src/guides/metrics.md b/site-src/guides/metrics.md
index fca43dd6..a781f721 100644
--- a/site-src/guides/metrics.md
+++ b/site-src/guides/metrics.md
@@ -30,8 +30,10 @@ curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
 | inference_model_response_sizes               | Distribution     | Distribution of response size in bytes.                           | `model_name`=&lt;model-name&gt; <br> `target_model_name`=&lt;target-model-name&gt; | ALPHA       |
 | inference_model_input_tokens                 | Distribution     | Distribution of input token count.                                | `model_name`=&lt;model-name&gt; <br> `target_model_name`=&lt;target-model-name&gt; | ALPHA       |
 | inference_model_output_tokens                | Distribution     | Distribution of output token count.                               | `model_name`=&lt;model-name&gt; <br> `target_model_name`=&lt;target-model-name&gt; | ALPHA       |
+| inference_model_running_requests                | Gauge     | Number of running requests for each model.             | `model_name`=&lt;model-name&gt;  | ALPHA       |
 | inference_pool_average_kv_cache_utilization  | Gauge            | The average kv cache utilization for an inference server pool.    | `name`=&lt;inference-pool-name&gt;                                                 | ALPHA       |
 | inference_pool_average_queue_size            | Gauge            | The average number of requests pending in the model server queue. | `name`=&lt;inference-pool-name&gt;                                                 | ALPHA       |
+| inference_pool_ready_pods                    | Gauge            | The number of ready pods for an inference server pool.            | `name`=&lt;inference-pool-name&gt;                                                 | ALPHA       |
 
 ## Scrape Metrics
 
diff --git a/site-src/images/favicon-64.png b/site-src/images/favicon-64.png
new file mode 100644
index 00000000..f2bd3d64
Binary files /dev/null and b/site-src/images/favicon-64.png differ
diff --git a/site-src/images/logo/logo-text-xl-dark.png b/site-src/images/logo/logo-text-xl-dark.png
new file mode 100644
index 00000000..4d878e5c
Binary files /dev/null and b/site-src/images/logo/logo-text-xl-dark.png differ
diff --git a/site-src/implementations.md b/site-src/implementations.md
index 89acb436..8a95119d 100644
--- a/site-src/implementations.md
+++ b/site-src/implementations.md
@@ -54,3 +54,12 @@ Issue](https://github.com/GoogleCloudPlatform/gke-gateway-api/issues/20).
 [gke-gateway]:https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api
 [gke-gateway-deploy]:https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways
 [gke-multi-cluster-gateway]:https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-multi-cluster-gateways
+
+## Istio
+
+[Istio](https://istio.io/) is an open source service mesh and gateway implementation.
+It provides a fully compliant implementation of the Kubernetes Gateway API for cluster ingress traffic control. 
+For service mesh users, Istio also fully supports east-west (including [GAMMA](https://gateway-api.sigs.k8s.io/mesh/)) traffic management within the mesh.
+
+Gateway API Inference Extension support is being tracked by this [GitHub
+Issue](https://github.com/istio/istio/issues/55768).
diff --git a/tools/dynamic-lora-sidecar/README.md b/tools/dynamic-lora-sidecar/README.md
index 4e85fd92..65dc0d78 100644
--- a/tools/dynamic-lora-sidecar/README.md
+++ b/tools/dynamic-lora-sidecar/README.md
@@ -83,7 +83,7 @@ In this example, both adapters will use `meta-llama/Llama-3.1-8B-Instruct` as th
 apiVersion: v1
 kind: ConfigMap
 metadata:
-  name: vllm-llama3-8b-adapters
+  name: vllm-llama3-8b-instruct-adapters
 data:
   configmap.yaml: |
       vLLMLoRAConfig:



Test patch:
diff --git a/pkg/epp/controller/inferencemodel_reconciler_test.go b/pkg/epp/controller/inferencemodel_reconciler_test.go
index cd1ff1fb..57dc2469 100644
--- a/pkg/epp/controller/inferencemodel_reconciler_test.go
+++ b/pkg/epp/controller/inferencemodel_reconciler_test.go
@@ -178,7 +178,7 @@ func TestInferenceModelReconciler(t *testing.T) {
 		t.Run(test.name, func(t *testing.T) {
 			// Create a fake client with no InferenceModel objects.
 			scheme := runtime.NewScheme()
-			_ = v1alpha2.AddToScheme(scheme)
+			_ = v1alpha2.Install(scheme)
 			initObjs := []client.Object{}
 			if test.model != nil {
 				initObjs = append(initObjs, test.model)
diff --git a/pkg/epp/controller/inferencepool_reconciler_test.go b/pkg/epp/controller/inferencepool_reconciler_test.go
index 27c4238e..7e5d4801 100644
--- a/pkg/epp/controller/inferencepool_reconciler_test.go
+++ b/pkg/epp/controller/inferencepool_reconciler_test.go
@@ -77,7 +77,7 @@ func TestInferencePoolReconciler(t *testing.T) {
 	// Set up the scheme.
 	scheme := runtime.NewScheme()
 	_ = clientgoscheme.AddToScheme(scheme)
-	_ = v1alpha2.AddToScheme(scheme)
+	_ = v1alpha2.Install(scheme)
 
 	// Create a fake client with the pool and the pods.
 	initialObjects := []client.Object{pool1, pool2}
diff --git a/pkg/epp/metrics/metrics_test.go b/pkg/epp/metrics/metrics_test.go
index c2436bab..dc4c7044 100644
--- a/pkg/epp/metrics/metrics_test.go
+++ b/pkg/epp/metrics/metrics_test.go
@@ -36,6 +36,7 @@ const (
 	ResponseSizesMetric     = InferenceModelComponent + "_response_sizes"
 	InputTokensMetric       = InferenceModelComponent + "_input_tokens"
 	OutputTokensMetric      = InferenceModelComponent + "_output_tokens"
+	RunningRequestsMetric   = InferenceModelComponent + "_running_requests"
 	KVCacheAvgUsageMetric   = InferencePoolComponent + "_average_kv_cache_utilization"
 	QueueAvgSizeMetric      = InferencePoolComponent + "_average_queue_size"
 )
@@ -345,6 +346,66 @@ func TestRecordResponseMetrics(t *testing.T) {
 	}
 }
 
+func TestRunningRequestsMetrics(t *testing.T) {
+	type request struct {
+		modelName string
+		complete  bool // true -> request is completed, false -> running request
+	}
+
+	scenarios := []struct {
+		name     string
+		requests []request
+	}{
+		{
+			name: "basic test",
+			requests: []request{
+				{
+					modelName: "m1",
+					complete:  false,
+				},
+				{
+					modelName: "m1",
+					complete:  false,
+				},
+				{
+					modelName: "m1",
+					complete:  true,
+				},
+				{
+					modelName: "m2",
+					complete:  false,
+				},
+			},
+		},
+	}
+
+	Register()
+	for _, scenario := range scenarios {
+		t.Run(scenario.name, func(t *testing.T) {
+			for _, req := range scenario.requests {
+				if req.complete {
+					DecRunningRequests(req.modelName)
+				} else {
+					IncRunningRequests(req.modelName)
+				}
+			}
+
+			wantRunningRequests, err := os.Open("testdata/running_requests_metrics")
+			defer func() {
+				if err := wantRunningRequests.Close(); err != nil {
+					t.Error(err)
+				}
+			}()
+			if err != nil {
+				t.Fatal(err)
+			}
+			if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, wantRunningRequests, RunningRequestsMetric); err != nil {
+				t.Error(err)
+			}
+		})
+	}
+}
+
 func TestInferencePoolMetrics(t *testing.T) {
 	scenarios := []struct {
 		name         string
diff --git a/pkg/epp/metrics/testdata/running_requests_metrics b/pkg/epp/metrics/testdata/running_requests_metrics
new file mode 100644
index 00000000..a880e499
--- /dev/null
+++ b/pkg/epp/metrics/testdata/running_requests_metrics
@@ -0,0 +1,4 @@
+# HELP inference_model_running_requests [ALPHA] Inference model number of running requests in each model.
+# TYPE inference_model_running_requests gauge
+inference_model_running_requests{model_name="m1"} 1
+inference_model_running_requests{model_name="m2"} 1
diff --git a/test/e2e/epp/e2e_suite_test.go b/test/e2e/epp/e2e_suite_test.go
index 643bbf75..61ee2540 100644
--- a/test/e2e/epp/e2e_suite_test.go
+++ b/test/e2e/epp/e2e_suite_test.go
@@ -49,6 +49,8 @@ const (
 	defaultReadyTimeout = 3 * time.Minute
 	// defaultModelReadyTimeout is the default timeout for the model server deployment to report a ready state.
 	defaultModelReadyTimeout = 10 * time.Minute
+	// defaultCurlTimeout is the default timeout for the curl command to get a response.
+	defaultCurlTimeout = 30 * time.Second
 	// defaultInterval is the default interval to check if a resource exists or ready conditions.
 	defaultInterval = time.Millisecond * 250
 	// defaultCurlInterval is the default interval to run the test curl command.
@@ -107,7 +109,11 @@ var _ = ginkgo.BeforeSuite(func() {
 })
 
 func setupInfra() {
-	modelServerManifest := readModelServerManifestPath()
+	modelServerManifestPath := readModelServerManifestPath()
+	modelServerManifestArray := getYamlsFromModelServerManifest(modelServerManifestPath)
+	if strings.Contains(modelServerManifestArray[0], "hf-token") {
+		createHfSecret(cli, modelServerSecretManifest)
+	}
 	crds := map[string]string{
 		"inferencepools.inference.networking.x-k8s.io":  inferPoolManifest,
 		"inferencemodels.inference.networking.x-k8s.io": inferModelManifest,
@@ -117,7 +123,7 @@ func setupInfra() {
 	createClient(cli, clientManifest)
 	createEnvoy(cli, envoyManifest)
 	// Run this step last, as it requires additional time for the model server to become ready.
-	createModelServer(cli, modelServerSecretManifest, modelServerManifest)
+	createModelServer(cli, modelServerManifestArray, modelServerManifestPath)
 }
 
 var _ = ginkgo.AfterSuite(func() {
@@ -137,7 +143,7 @@ func setupSuite() {
 	err = apiextv1.AddToScheme(scheme)
 	gomega.ExpectWithOffset(1, err).NotTo(gomega.HaveOccurred())
 
-	err = infextv1a2.AddToScheme(scheme)
+	err = infextv1a2.Install(scheme)
 	gomega.ExpectWithOffset(1, err).NotTo(gomega.HaveOccurred())
 
 	cli, err = client.New(cfg, client.Options{Scheme: scheme})
@@ -171,6 +177,7 @@ var (
 	existsTimeout     = getTimeout("EXISTS_TIMEOUT", defaultExistsTimeout)
 	readyTimeout      = getTimeout("READY_TIMEOUT", defaultReadyTimeout)
 	modelReadyTimeout = getTimeout("MODEL_READY_TIMEOUT", defaultModelReadyTimeout)
+	curlTimeout       = getTimeout("CURL_TIMEOUT", defaultCurlTimeout)
 	interval          = defaultInterval
 	curlInterval      = defaultCurlInterval
 )
@@ -191,6 +198,13 @@ func readModelServerManifestPath() string {
 	return modelServerManifestFilepath
 }
 
+func getYamlsFromModelServerManifest(modelServerManifestPath string) []string {
+	ginkgo.By("Ensuring the model server manifest points to an existing file")
+	modelServerManifestArray := readYaml(modelServerManifestPath)
+	gomega.Expect(modelServerManifestArray).NotTo(gomega.BeEmpty())
+	return modelServerManifestArray
+}
+
 // createCRDs creates the Inference Extension CRDs used for testing.
 func createCRDs(k8sClient client.Client, crds map[string]string) {
 	for name, path := range crds {
@@ -224,15 +238,7 @@ func createClient(k8sClient client.Client, filePath string) {
 }
 
 // createModelServer creates the model server resources used for testing from the given filePaths.
-func createModelServer(k8sClient client.Client, secretPath, deployPath string) {
-	ginkgo.By("Ensuring the model server manifest points to an existing file")
-	modelServerManifestArray := readYaml(deployPath)
-	gomega.Expect(modelServerManifestArray).NotTo(gomega.BeEmpty())
-	modelServerManifestYaml := modelServerManifestArray[0]
-	if strings.Contains(modelServerManifestYaml, "hf-token") {
-		createHfSecret(k8sClient, secretPath)
-	}
-
+func createModelServer(k8sClient client.Client, modelServerManifestArray []string, deployPath string) {
 	ginkgo.By("Creating model server resources from manifest: " + deployPath)
 	createObjsFromYaml(k8sClient, modelServerManifestArray)
 
diff --git a/test/e2e/epp/e2e_test.go b/test/e2e/epp/e2e_test.go
index 09c8835a..7240cebc 100644
--- a/test/e2e/epp/e2e_test.go
+++ b/test/e2e/epp/e2e_test.go
@@ -18,7 +18,9 @@ package epp
 
 import (
 	"fmt"
+	"strconv"
 	"strings"
+	"time"
 
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
@@ -53,7 +55,7 @@ var _ = ginkgo.Describe("InferencePool", func() {
 			}, existsTimeout, interval).Should(gomega.Succeed())
 
 			ginkgo.By("Verifying connectivity through the inference extension")
-			curlCmd := getCurlCommand(envoyName, nsName, envoyPort, modelName)
+			curlCmd := getCurlCommand(envoyName, nsName, envoyPort, modelName, curlTimeout)
 
 			// Ensure the expected responses include the inferencemodel target model names.
 			var expected []string
@@ -94,11 +96,11 @@ var _ = ginkgo.Describe("InferencePool", func() {
 func newInferenceModel(ns string) *v1alpha2.InferenceModel {
 	targets := []v1alpha2.TargetModel{
 		{
-			Name:   modelName + "-0",
+			Name:   modelName,
 			Weight: ptr.To(int32(50)),
 		},
 		{
-			Name:   modelName + "-1",
+			Name:   "cad-fabricator",
 			Weight: ptr.To(int32(50)),
 		},
 	}
@@ -112,10 +114,12 @@ func newInferenceModel(ns string) *v1alpha2.InferenceModel {
 
 // getCurlCommand returns the command, as a slice of strings, for curl'ing
 // the test model server at the given name, namespace, port, and model name.
-func getCurlCommand(name, ns, port, model string) []string {
+func getCurlCommand(name, ns, port, model string, timeout time.Duration) []string {
 	return []string{
 		"curl",
 		"-i",
+		"--max-time",
+		strconv.Itoa((int)(timeout.Seconds())),
 		fmt.Sprintf("%s.%s.svc:%s/v1/completions", name, ns, port),
 		"-H",
 		"Content-Type: application/json",
diff --git a/test/integration/bbr/hermetic_test.go b/test/integration/bbr/hermetic_test.go
index 718bfedf..02d412ab 100644
--- a/test/integration/bbr/hermetic_test.go
+++ b/test/integration/bbr/hermetic_test.go
@@ -19,20 +19,19 @@ package bbr
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"testing"
 	"time"
 
 	configPb "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
 	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
-	"github.com/go-logr/logr"
 	"github.com/google/go-cmp/cmp"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials/insecure"
 	"google.golang.org/protobuf/testing/protocmp"
 	runserver "sigs.k8s.io/gateway-api-inference-extension/pkg/body-based-routing/server"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
+	integrationutils "sigs.k8s.io/gateway-api-inference-extension/test/integration"
 )
 
 var logger = logutil.NewTestLogger().V(logutil.VERBOSE)
@@ -46,7 +45,7 @@ func TestBodyBasedRouting(t *testing.T) {
 	}{
 		{
 			name: "success adding model parameter to header",
-			req:  generateRequest(logger, "llama"),
+			req:  integrationutils.GenerateRequest(logger, "test", "llama"),
 			wantHeaders: []*configPb.HeaderValueOption{
 				{
 					Header: &configPb.HeaderValue{
@@ -59,7 +58,7 @@ func TestBodyBasedRouting(t *testing.T) {
 		},
 		{
 			name:        "no model parameter",
-			req:         generateRequest(logger, ""),
+			req:         integrationutils.GenerateRequest(logger, "test1", ""),
 			wantHeaders: []*configPb.HeaderValueOption{},
 			wantErr:     false,
 		},
@@ -67,7 +66,7 @@ func TestBodyBasedRouting(t *testing.T) {
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			client, cleanup := setUpHermeticServer()
+			client, cleanup := setUpHermeticServer(false)
 			t.Cleanup(cleanup)
 
 			want := &extProcPb.ProcessingResponse{}
@@ -88,7 +87,7 @@ func TestBodyBasedRouting(t *testing.T) {
 				}
 			}
 
-			res, err := sendRequest(t, client, test.req)
+			res, err := integrationutils.SendRequest(t, client, test.req)
 			if err != nil && !test.wantErr {
 				t.Errorf("Unexpected error, got: %v, want error: %v", err, test.wantErr)
 			}
@@ -99,12 +98,171 @@ func TestBodyBasedRouting(t *testing.T) {
 	}
 }
 
-func setUpHermeticServer() (client extProcPb.ExternalProcessor_ProcessClient, cleanup func()) {
+func TestFullDuplexStreamed_BodyBasedRouting(t *testing.T) {
+	tests := []struct {
+		name          string
+		reqs          []*extProcPb.ProcessingRequest
+		wantResponses []*extProcPb.ProcessingResponse
+		wantErr       bool
+	}{
+		{
+			name: "success adding model parameter to header",
+			reqs: integrationutils.GenerateStreamedRequestSet(logger, "test", "foo"),
+			wantResponses: []*extProcPb.ProcessingResponse{
+				{
+					Response: &extProcPb.ProcessingResponse_RequestHeaders{
+						RequestHeaders: &extProcPb.HeadersResponse{
+							Response: &extProcPb.CommonResponse{
+								ClearRouteCache: true,
+								HeaderMutation: &extProcPb.HeaderMutation{
+									SetHeaders: []*configPb.HeaderValueOption{
+										{
+											Header: &configPb.HeaderValue{
+												Key:      "X-Gateway-Model-Name",
+												RawValue: []byte("foo"),
+											},
+										},
+									}},
+							},
+						},
+					},
+				},
+				{
+					Response: &extProcPb.ProcessingResponse_RequestBody{
+						RequestBody: &extProcPb.BodyResponse{
+							Response: &extProcPb.CommonResponse{
+								BodyMutation: &extProcPb.BodyMutation{
+									Mutation: &extProcPb.BodyMutation_StreamedResponse{
+										StreamedResponse: &extProcPb.StreamedBodyResponse{
+											Body:        []byte("{\"max_tokens\":100,\"model\":\"foo\",\"prompt\":\"test\",\"temperature\":0}"),
+											EndOfStream: true,
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			name: "success adding model parameter to header with multiple body chunks",
+			reqs: []*extProcPb.ProcessingRequest{
+				{
+					Request: &extProcPb.ProcessingRequest_RequestHeaders{
+						RequestHeaders: &extProcPb.HttpHeaders{
+							Headers: &configPb.HeaderMap{
+								Headers: []*configPb.HeaderValue{
+									{
+										Key:   "hi",
+										Value: "mom",
+									},
+								},
+							},
+						},
+					},
+				},
+				{
+					Request: &extProcPb.ProcessingRequest_RequestBody{
+						RequestBody: &extProcPb.HttpBody{Body: []byte("{\"max_tokens\":100,\"model\":\"sql-lo"), EndOfStream: false},
+					},
+				},
+				{
+					Request: &extProcPb.ProcessingRequest_RequestBody{
+						RequestBody: &extProcPb.HttpBody{Body: []byte("ra-sheddable\",\"prompt\":\"test\",\"temperature\":0}"), EndOfStream: true},
+					},
+				},
+			},
+			wantResponses: []*extProcPb.ProcessingResponse{
+				{
+					Response: &extProcPb.ProcessingResponse_RequestHeaders{
+						RequestHeaders: &extProcPb.HeadersResponse{
+							Response: &extProcPb.CommonResponse{
+								ClearRouteCache: true,
+								HeaderMutation: &extProcPb.HeaderMutation{
+									SetHeaders: []*configPb.HeaderValueOption{
+										{
+											Header: &configPb.HeaderValue{
+												Key:      "X-Gateway-Model-Name",
+												RawValue: []byte("sql-lora-sheddable"),
+											},
+										},
+									}},
+							},
+						},
+					},
+				},
+				{
+					Response: &extProcPb.ProcessingResponse_RequestBody{
+						RequestBody: &extProcPb.BodyResponse{
+							Response: &extProcPb.CommonResponse{
+								BodyMutation: &extProcPb.BodyMutation{
+									Mutation: &extProcPb.BodyMutation_StreamedResponse{
+										StreamedResponse: &extProcPb.StreamedBodyResponse{
+											Body:        []byte("{\"max_tokens\":100,\"model\":\"sql-lora-sheddable\",\"prompt\":\"test\",\"temperature\":0}"),
+											EndOfStream: true,
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			name: "no model parameter",
+			reqs: integrationutils.GenerateStreamedRequestSet(logger, "test", ""),
+			wantResponses: []*extProcPb.ProcessingResponse{
+				{
+					Response: &extProcPb.ProcessingResponse_RequestHeaders{
+						RequestHeaders: &extProcPb.HeadersResponse{},
+					},
+				},
+				{
+					Response: &extProcPb.ProcessingResponse_RequestBody{
+						RequestBody: &extProcPb.BodyResponse{
+							Response: &extProcPb.CommonResponse{
+								BodyMutation: &extProcPb.BodyMutation{
+									Mutation: &extProcPb.BodyMutation_StreamedResponse{
+										StreamedResponse: &extProcPb.StreamedBodyResponse{
+											Body:        []byte("{\"max_tokens\":100,\"prompt\":\"test\",\"temperature\":0}"),
+											EndOfStream: true,
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			client, cleanup := setUpHermeticServer(true)
+			t.Cleanup(cleanup)
+
+			responses, err := integrationutils.StreamedRequest(t, client, test.reqs, len(test.wantResponses))
+			if err != nil && !test.wantErr {
+				t.Errorf("Unexpected error, got: %v, want error: %v", err, test.wantErr)
+			}
+
+			if diff := cmp.Diff(test.wantResponses, responses, protocmp.Transform()); diff != "" {
+				t.Errorf("Unexpected response, (-want +got): %v", diff)
+			}
+		})
+	}
+}
+
+func setUpHermeticServer(streaming bool) (client extProcPb.ExternalProcessor_ProcessClient, cleanup func()) {
 	port := 9004
 
 	serverCtx, stopServer := context.WithCancel(context.Background())
 	serverRunner := runserver.NewDefaultExtProcServerRunner(port, false)
 	serverRunner.SecureServing = false
+	serverRunner.Streaming = streaming
 
 	go func() {
 		if err := serverRunner.AsRunnable(logger.WithName("ext-proc")).Start(serverCtx); err != nil {
@@ -133,41 +291,3 @@ func setUpHermeticServer() (client extProcPb.ExternalProcessor_ProcessClient, cl
 		time.Sleep(5 * time.Second)
 	}
 }
-
-func generateRequest(logger logr.Logger, model string) *extProcPb.ProcessingRequest {
-	j := map[string]interface{}{
-		"prompt":      "test1",
-		"max_tokens":  100,
-		"temperature": 0,
-	}
-	if model != "" {
-		j["model"] = model
-	}
-
-	llmReq, err := json.Marshal(j)
-	if err != nil {
-		logutil.Fatal(logger, err, "Failed to unmarshal LLM request")
-	}
-	req := &extProcPb.ProcessingRequest{
-		Request: &extProcPb.ProcessingRequest_RequestBody{
-			RequestBody: &extProcPb.HttpBody{Body: llmReq},
-		},
-	}
-	return req
-}
-
-func sendRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessClient, req *extProcPb.ProcessingRequest) (*extProcPb.ProcessingResponse, error) {
-	t.Logf("Sending request: %v", req)
-	if err := client.Send(req); err != nil {
-		t.Logf("Failed to send request %+v: %v", req, err)
-		return nil, err
-	}
-
-	res, err := client.Recv()
-	if err != nil {
-		t.Logf("Failed to receive: %v", err)
-		return nil, err
-	}
-	t.Logf("Received request %+v", res)
-	return res, err
-}
diff --git a/test/integration/epp/hermetic_test.go b/test/integration/epp/hermetic_test.go
index 8e02aca4..1c5eca18 100644
--- a/test/integration/epp/hermetic_test.go
+++ b/test/integration/epp/hermetic_test.go
@@ -66,7 +66,8 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
 	runserver "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
-	utiltesting "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/testing"
+	epptestutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/testing"
+	integrationutils "sigs.k8s.io/gateway-api-inference-extension/test/integration"
 	"sigs.k8s.io/yaml"
 )
 
@@ -104,7 +105,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 	}{
 		{
 			name: "select lower queue and kv cache, no active lora",
-			req:  utiltesting.GenerateRequest(logger, "test1", "my-model"),
+			req:  integrationutils.GenerateRequest(logger, "test1", "my-model"),
 			// pod-1 will be picked because it has relatively low queue size and low KV cache.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
 				fakePod(0): {
@@ -145,7 +146,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name: "select active lora, low queue",
-			req:  utiltesting.GenerateRequest(logger, "test2", "sql-lora"),
+			req:  integrationutils.GenerateRequest(logger, "test2", "sql-lora"),
 			// pod-1 will be picked because it has relatively low queue size, with the requested
 			// model being active, and has low KV cache.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
@@ -199,7 +200,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name: "select no lora despite active model, avoid excessive queue size",
-			req:  utiltesting.GenerateRequest(logger, "test3", "sql-lora"),
+			req:  integrationutils.GenerateRequest(logger, "test3", "sql-lora"),
 			// pod-2 will be picked despite it NOT having the requested model being active
 			// as it's above the affinity for queue size. Also is critical, so we should
 			// still honor request despite all queues > 5
@@ -253,7 +254,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name: "noncritical and all models past threshold, shed request",
-			req:  utiltesting.GenerateRequest(logger, "test4", "sql-lora-sheddable"),
+			req:  integrationutils.GenerateRequest(logger, "test4", "sql-lora-sheddable"),
 			// no pods will be picked as all models are either above kv threshold,
 			// queue threshold, or both.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
@@ -296,7 +297,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name: "noncritical, but one server has capacity, do not shed",
-			req:  utiltesting.GenerateRequest(logger, "test5", "sql-lora-sheddable"),
+			req:  integrationutils.GenerateRequest(logger, "test5", "sql-lora-sheddable"),
 			// pod 0 will be picked as all other models are above threshold
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
 				fakePod(0): {
@@ -370,7 +371,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 				},
 				DynamicMetadata: test.wantMetadata,
 			}
-			res, err := sendRequest(t, client, test.req)
+			res, err := integrationutils.SendRequest(t, client, test.req)
 
 			if err != nil && !test.wantErr {
 				t.Errorf("Unexpected error, got: %v, want error: %v", err, test.wantErr)
@@ -410,7 +411,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		// Request flow tests
 		{
 			name:     "select lower queue and kv cache, no active lora",
-			requests: utiltesting.GenerateStreamedRequestSet(logger, "test1", "my-model"),
+			requests: integrationutils.GenerateStreamedRequestSet(logger, "test1", "my-model"),
 			// pod-1 will be picked because it has relatively low queue size and low KV cache.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
 				fakePod(0): {
@@ -430,7 +431,13 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 					# HELP inference_model_request_total [ALPHA] Counter of inference model requests broken out for each model and target model.
 					# TYPE inference_model_request_total counter
 					inference_model_request_total{model_name="my-model",target_model_name="my-model-12345"} 1
-					`},
+					`,
+				`inference_pool_ready_pods`: `
+					# HELP inference_pool_ready_pods [ALPHA] The number of ready pods in the inference server pool.
+					# TYPE inference_pool_ready_pods gauge
+					inference_pool_ready_pods{name="vllm-llama3-8b-instruct-pool"} 3
+					`,
+			},
 			wantErr: false,
 			wantResponses: []*extProcPb.ProcessingResponse{
 				{
@@ -478,7 +485,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name:     "select active lora, low queue",
-			requests: utiltesting.GenerateStreamedRequestSet(logger, "test2", "sql-lora"),
+			requests: integrationutils.GenerateStreamedRequestSet(logger, "test2", "sql-lora"),
 			// pod-1 will be picked because it has relatively low queue size, with the requested
 			// model being active, and has low KV cache.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
@@ -559,7 +566,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name:     "select no lora despite active model, avoid excessive queue size",
-			requests: utiltesting.GenerateStreamedRequestSet(logger, "test3", "sql-lora"),
+			requests: integrationutils.GenerateStreamedRequestSet(logger, "test3", "sql-lora"),
 			// pod-2 will be picked despite it NOT having the requested model being active
 			// as it's above the affinity for queue size. Also is critical, so we should
 			// still honor request despite all queues > 5
@@ -640,7 +647,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name:     "noncritical and all models past threshold, shed request",
-			requests: utiltesting.GenerateStreamedRequestSet(logger, "test4", "sql-lora-sheddable"),
+			requests: integrationutils.GenerateStreamedRequestSet(logger, "test4", "sql-lora-sheddable"),
 			// no pods will be picked as all models are either above kv threshold,
 			// queue threshold, or both.
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
@@ -686,7 +693,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		},
 		{
 			name:     "noncritical, but one server has capacity, do not shed",
-			requests: utiltesting.GenerateStreamedRequestSet(logger, "test5", "sql-lora-sheddable"),
+			requests: integrationutils.GenerateStreamedRequestSet(logger, "test5", "sql-lora-sheddable"),
 			// pod 0 will be picked as all other models are above threshold
 			pods: map[backendmetrics.Pod]*backendmetrics.Metrics{
 				fakePod(0): {
@@ -1465,6 +1472,11 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 					},
 				},
 			},
+			wantMetrics: map[string]string{`inference_pool_ready_pods`: `
+				# HELP inference_pool_ready_pods [ALPHA] The number of ready pods in the inference server pool.
+				# TYPE inference_pool_ready_pods gauge
+				inference_pool_ready_pods{name="vllm-llama3-8b-instruct-pool"} 1
+				`},
 		},
 	}
 
@@ -1472,7 +1484,7 @@ func TestFullDuplexStreamed_KubeInferenceModelRequest(t *testing.T) {
 		t.Run(test.name, func(t *testing.T) {
 			client, cleanup := setUpHermeticServer(t, test.pods, true)
 			t.Cleanup(cleanup)
-			responses, err := streamedRequest(t, client, test.requests, len(test.wantResponses))
+			responses, err := integrationutils.StreamedRequest(t, client, test.requests, len(test.wantResponses))
 
 			if err != nil && !test.wantErr {
 				t.Errorf("Unexpected error, got: %v, want error: %v", err, test.wantErr)
@@ -1511,7 +1523,7 @@ func setUpHermeticServer(t *testing.T, podAndMetrics map[backendmetrics.Pod]*bac
 	}
 
 	for pod := range podAndMetrics {
-		pod := utiltesting.MakePod(pod.NamespacedName.Name).
+		pod := epptestutil.MakePod(pod.NamespacedName.Name).
 			Namespace(pod.NamespacedName.Namespace).
 			ReadyCondition().
 			Labels(podLabels).
@@ -1536,6 +1548,8 @@ func setUpHermeticServer(t *testing.T, podAndMetrics map[backendmetrics.Pod]*bac
 		}
 	}()
 
+	time.Sleep(serverRunner.RefreshPrometheusMetricsInterval) // wait for metrics to get available before running tests that rely on these metrics
+
 	// check if all pods are synced to datastore
 	assert.EventuallyWithT(t, func(t *assert.CollectT) {
 		assert.Len(t, serverRunner.Datastore.PodGetAll(), len(podAndMetrics), "Datastore not synced")
@@ -1560,7 +1574,7 @@ func setUpHermeticServer(t *testing.T, podAndMetrics map[backendmetrics.Pod]*bac
 
 		// clear created pods
 		for pod := range podAndMetrics {
-			pod := utiltesting.MakePod(pod.NamespacedName.Name).
+			pod := epptestutil.MakePod(pod.NamespacedName.Name).
 				Namespace(pod.NamespacedName.Namespace).Complete().ObjRef()
 
 			if err := k8sClient.Delete(context.Background(), pod); err != nil {
@@ -1590,7 +1604,7 @@ func BeforeSuite() func() {
 	}
 
 	utilruntime.Must(clientgoscheme.AddToScheme(scheme))
-	utilruntime.Must(v1alpha2.AddToScheme(scheme))
+	utilruntime.Must(v1alpha2.Install(scheme))
 
 	k8sClient, err = k8sclient.New(cfg, k8sclient.Options{Scheme: scheme})
 	if err != nil {
@@ -1677,55 +1691,6 @@ func BeforeSuite() func() {
 	}
 }
 
-func sendRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessClient, req *extProcPb.ProcessingRequest) (*extProcPb.ProcessingResponse, error) {
-	t.Logf("Sending request: %v", req)
-	if err := client.Send(req); err != nil {
-		t.Logf("Failed to send request %+v: %v", req, err)
-		return nil, err
-	}
-
-	res, err := client.Recv()
-	if err != nil {
-		t.Logf("Failed to receive: %v", err)
-		return nil, err
-	}
-	t.Logf("Received request %+v", res)
-	return res, err
-}
-
-func streamedRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessClient, requests []*extProcPb.ProcessingRequest, expectedResponses int) ([]*extProcPb.ProcessingResponse, error) {
-	for _, req := range requests {
-		t.Logf("Sending request: %v", req)
-		if err := client.Send(req); err != nil {
-			t.Logf("Failed to send request %+v: %v", req, err)
-			return nil, err
-		}
-	}
-	responses := []*extProcPb.ProcessingResponse{}
-
-	// Make an incredible simple timeout func in the case where
-	// there is less than the expected amount of responses; bail and fail.
-	var simpleTimeout bool
-	go func() {
-		time.Sleep(10 * time.Second)
-		simpleTimeout = true
-	}()
-
-	for range expectedResponses {
-		if simpleTimeout {
-			break
-		}
-		res, err := client.Recv()
-		if err != nil && err != io.EOF {
-			t.Logf("Failed to receive: %v", err)
-			return nil, err
-		}
-		t.Logf("Received request %+v", res)
-		responses = append(responses, res)
-	}
-	return responses, nil
-}
-
 // readDocuments reads documents from file.
 func readDocuments(fp string) ([][]byte, error) {
 	b, err := os.ReadFile(fp)
diff --git a/pkg/epp/util/testing/request.go b/test/integration/util.go
similarity index 56%
rename from pkg/epp/util/testing/request.go
rename to test/integration/util.go
index 30772ad5..5fcc9d18 100644
--- a/pkg/epp/util/testing/request.go
+++ b/test/integration/util.go
@@ -14,10 +14,13 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package testing
+package integration
 
 import (
 	"encoding/json"
+	"io"
+	"testing"
+	"time"
 
 	envoyCorev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
 	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
@@ -25,13 +28,64 @@ import (
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
+func SendRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessClient, req *extProcPb.ProcessingRequest) (*extProcPb.ProcessingResponse, error) {
+	t.Logf("Sending request: %v", req)
+	if err := client.Send(req); err != nil {
+		t.Logf("Failed to send request %+v: %v", req, err)
+		return nil, err
+	}
+
+	res, err := client.Recv()
+	if err != nil {
+		t.Logf("Failed to receive: %v", err)
+		return nil, err
+	}
+	t.Logf("Received response %+v", res)
+	return res, err
+}
+
+func StreamedRequest(t *testing.T, client extProcPb.ExternalProcessor_ProcessClient, requests []*extProcPb.ProcessingRequest, expectedResponses int) ([]*extProcPb.ProcessingResponse, error) {
+	for _, req := range requests {
+		t.Logf("Sending request: %v", req)
+		if err := client.Send(req); err != nil {
+			t.Logf("Failed to send request %+v: %v", req, err)
+			return nil, err
+		}
+	}
+	responses := []*extProcPb.ProcessingResponse{}
+
+	// Make an incredible simple timeout func in the case where
+	// there is less than the expected amount of responses; bail and fail.
+	var simpleTimeout bool
+	go func() {
+		time.Sleep(10 * time.Second)
+		simpleTimeout = true
+	}()
+
+	for range expectedResponses {
+		if simpleTimeout {
+			break
+		}
+		res, err := client.Recv()
+		if err != nil && err != io.EOF {
+			t.Logf("Failed to receive: %v", err)
+			return nil, err
+		}
+		t.Logf("Received response %+v", res)
+		responses = append(responses, res)
+	}
+	return responses, nil
+}
+
 func GenerateRequest(logger logr.Logger, prompt, model string) *extProcPb.ProcessingRequest {
 	j := map[string]interface{}{
-		"model":       model,
 		"prompt":      prompt,
 		"max_tokens":  100,
 		"temperature": 0,
 	}
+	if model != "" {
+		j["model"] = model
+	}
 
 	llmReq, err := json.Marshal(j)
 	if err != nil {
diff --git a/test/testdata/envoy.yaml b/test/testdata/envoy.yaml
index fc32b5aa..62e6b4c5 100644
--- a/test/testdata/envoy.yaml
+++ b/test/testdata/envoy.yaml
@@ -104,10 +104,11 @@ data:
                             timeout: 10s
                           processing_mode:
                             request_header_mode: SEND
-                            response_header_mode: SKIP
-                            request_body_mode: BUFFERED
-                            request_trailer_mode: SKIP
-                            response_trailer_mode: SKIP
+                            response_header_mode: SEND
+                            request_body_mode: FULL_DUPLEX_STREAMED
+                            response_body_mode: FULL_DUPLEX_STREAMED
+                            request_trailer_mode: SEND
+                            response_trailer_mode: SEND
                           message_timeout: 1000s
                         # Mark it as disabled if needed for troubleshooting:
                         # disabled: true
@@ -221,7 +222,7 @@ spec:
     spec:
       containers:
       - name: envoy
-        image: docker.io/envoyproxy/envoy:distroless-v1.32.2
+        image: docker.io/envoyproxy/envoy:distroless-v1.33.2
         args:
           - "--service-cluster" 
           - "default/inference-gateway"