Repository: kubernetes-sigs/gateway-api-inference-extension

Issue:
The normalized_time_per_output_token_seconds metrics is not recorded
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

**What happened**:
As [this guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/metrics/) says, the `normalized_time_per_output_token_seconds` metric is supported by EPP, which means "Distribution of ntpot (response latency per output token)".

However, this metric is not actually being recorded in the latest EPP code. The `RecordNormalizedTimePerOutputToken`  function here is only called by unit test.
https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/c300d268ad7e90f3b49a08c68092bd81fedd9a02/pkg/epp/metrics/metrics.go#L361
**What you expected to happen**:

`normalized_time_per_output_token_seconds` should be recored and exposed by EPP after generating streaming response.

**How to reproduce it (as minimally and precisely as possible)**:

I actually discovered this issue while providing e2e testing for metrics. You can refer to my branch (https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/938). Remove the line marked with TODO and run the e2e tests to observe the problem.

**Anything else we need to know?**:

I find the [guidance documentation](https://gateway-api-inference-extension.sigs.k8s.io/guides/metrics/) for the metrics to also be problematic.
1. I think the `normalized_time_per_output_token_seconds` metric mentioned in the document should actually be the `inference_model_normalized_time_per_output_token_seconds` metric. The prefix for the subsystem is not included here.
2. The doc says: 

>  To have response metrics, ensure the body mode is set to Buffered or Streamed (this should be the default behavior for all implementations)."

Is the description here somewhat outdated? As far as I know, EPP now exclusively uses `FULL_DUPLEX_STREAMED`.

**Environment**:
- Kubernetes version (use `kubectl version`):
- Inference extension version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- Install tools:
- Others:


Discussion:
@KrishnaSindhur: /assign@dharaneeshvrd: /assign

Solution patch:
diff --git a/.github/ISSUE_TEMPLATE/new-release.md b/.github/ISSUE_TEMPLATE/new-release.md
index 27e8378..16ff164 100644
--- a/.github/ISSUE_TEMPLATE/new-release.md
+++ b/.github/ISSUE_TEMPLATE/new-release.md
@@ -35,10 +35,14 @@ This document defines the process for releasing Gateway API Inference Extension.
    export RC=1
    ```
 
-4. The vLLM image tag defaults to `v0.7.2` for a release. Set the `VLLM` environment variable if a newer [tag][vllm-tag] has been published. For example:
+4. Refer to the [release-quickstart script][release-quickstart] for the default image tags used
+   by the vLLM deployment manifests. If a newer [GPU][vllm-gpu-tag], [CPU][vllm-cpu-tag], or [Simulator][vllm-sim-tag]
+   tag has been published, set the appropriate environment variable or update the script. For example:
 
    ```shell
-   export VLLM=0.7.3
+   export VLLM_GPU=0.9.2
+   export VLLM_CPU=0.9.3
+   export VLLM_SIM=0.1.2
    ```
 
 ## Release Process
@@ -159,4 +163,7 @@ Use the following steps to announce the release.
 [k8s.io]: https://github.com/kubernetes/k8s.io
 [yaml]: https://github.com/kubernetes/k8s.io/blob/main/registry.k8s.io/images/k8s-staging-gateway-api-inference-extension/images.yaml
 [issue]: https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/new/choose
-[vllm-tag]: https://hub.docker.com/r/vllm/vllm-openai/tags
+[vllm-gpu-tag]: https://hub.docker.com/r/vllm/vllm-openai/tags
+[vllm-cpu-tag]: https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
+[vllm-sim-tag]: https://github.com/llm-d/llm-d-inference-sim/pkgs/container/llm-d-inference-sim
+[release-quickstart]: https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/hack/release-quickstart.sh
diff --git a/Dockerfile b/Dockerfile
index e49a21f..5512518 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -17,12 +17,12 @@ COPY go.mod go.sum ./
 RUN go mod download
 
 # Sources
-COPY cmd/epp ./cmd
+COPY cmd/epp ./cmd/epp
 COPY pkg/epp ./pkg/epp
 COPY conformance/testing-epp ./conformance/testing-epp
 COPY internal ./internal
 COPY api ./api
-WORKDIR /src/cmd
+WORKDIR /src/cmd/epp
 RUN go build -ldflags="-X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.CommitSHA=${COMMIT_SHA} -X sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics.BuildRef=${BUILD_REF}" -o /epp
 
 ## Multistage deploy
diff --git a/Makefile b/Makefile
index d3eeaad..52fe9fd 100644
--- a/Makefile
+++ b/Makefile
@@ -137,8 +137,8 @@ test-integration: ## Run integration tests.
 	KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./test/integration/epp/... -race -coverprofile cover.out
 
 .PHONY: test-e2e
-test-e2e: ## Run end-to-end tests against an existing Kubernetes cluster. When using default configuration, the tests need at least 3 available GPUs.
-	MANIFEST_PATH=$(PROJECT_DIR)/$(E2E_MANIFEST_PATH) go test ./test/e2e/epp/ -v -ginkgo.v
+test-e2e: ## Run end-to-end tests against an existing Kubernetes cluster.
+	MANIFEST_PATH=$(PROJECT_DIR)/$(E2E_MANIFEST_PATH) ./hack/run-e2es.sh
 
 .PHONY: lint
 lint: golangci-lint ## Run golangci-lint linter
@@ -337,7 +337,7 @@ release: artifacts release-quickstart verify test # Create a release.
 ## Location to install dependencies to
 LOCALBIN ?= $(shell pwd)/bin
 $(LOCALBIN):
-	mkdir -p $(LOCALBIN)
+	[ -d $@ ] || mkdir -p $@
 
 ## Tool Binaries
 KUBECTL ?= kubectl
diff --git a/OWNERS_ALIASES b/OWNERS_ALIASES
index 933fbe9..32a0493 100644
--- a/OWNERS_ALIASES
+++ b/OWNERS_ALIASES
@@ -5,14 +5,13 @@ aliases:
   gateway-api-inference-extension-maintainers:
   - ahg-g
   - danehans
-  - jeffwan
+  - nirrozenbaum
   - kfswain
 
   gateway-api-inference-extension-reviewers:
   - liu-cong
   - robscott
   - shaneutt
-  - nirrozenbaum
 
 
   wg-serving-leads:
diff --git a/PROJECT b/PROJECT
index 75c9c9c..c049fc8 100644
--- a/PROJECT
+++ b/PROJECT
@@ -24,4 +24,12 @@ resources:
   kind: InferenceModel
   path: sigs.k8s.io/gateway-api-inference-extension/api/v1alpha1
   version: v1alpha1
+- api:
+    crdVersion: v1
+    namespaced: true
+  domain: x-k8s.io
+  group: inference
+  kind: EndpointPickerConfig
+  path: sigs.k8s.io/gateway-api-inference-extension/api/config/v1alpha1
+  version: v1alpha1
 version: "3"
diff --git a/api/config/v1alpha1/defaults.go b/api/config/v1alpha1/defaults.go
new file mode 100644
index 0000000..f59bc7c
--- /dev/null
+++ b/api/config/v1alpha1/defaults.go
@@ -0,0 +1,29 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+// SetDefaults_EndpointPickerConfig sets default values in a
+// EndpointPickerConfig struct.
+//
+// This naming convension is required by the defalter-gen code.
+func SetDefaults_EndpointPickerConfig(cfg *EndpointPickerConfig) {
+	for idx, pluginConfig := range cfg.Plugins {
+		if pluginConfig.Name == "" {
+			cfg.Plugins[idx].Name = pluginConfig.PluginName
+		}
+	}
+}
diff --git a/api/config/v1alpha1/doc.go b/api/config/v1alpha1/doc.go
new file mode 100644
index 0000000..122c3b9
--- /dev/null
+++ b/api/config/v1alpha1/doc.go
@@ -0,0 +1,22 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Package v1alpha1 contains API Schema definitions for the
+// inference.networking.x-k8s.io API group.
+//
+// +kubebuilder:object:generate=true
+// +groupName=inference.networking.x-k8s.io
+package v1alpha1
diff --git a/api/config/v1alpha1/endpointpickerconfig_types.go b/api/config/v1alpha1/endpointpickerconfig_types.go
new file mode 100644
index 0000000..86f5ab7
--- /dev/null
+++ b/api/config/v1alpha1/endpointpickerconfig_types.go
@@ -0,0 +1,92 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package v1alpha1
+
+import (
+	"encoding/json"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// +k8s:defaulter-gen=true
+// +kubebuilder:object:root=true
+
+// EndpointPickerConfig is the Schema for the endpointpickerconfigs API
+type EndpointPickerConfig struct {
+	metav1.TypeMeta `json:",inline"`
+
+	// +required
+	// +kubebuilder:validation:Required
+	// Plugins is the list of plugins that will be instantiated.
+	Plugins []PluginSpec `json:"plugins"`
+
+	// +required
+	// +kubebuilder:validation:Required
+	// SchedulingProfiles is the list of named SchedulingProfiles
+	// that will be created.
+	SchedulingProfiles []SchedulingProfile `json:"schedulingProfiles"`
+}
+
+// PluginSpec contains the information that describes a plugin that
+// will be instantiated.
+type PluginSpec struct {
+	// +optional
+	// Name provides a name for plugin entries to reference. If
+	// omitted, the value of the PluginName field will be used.
+	Name string `json:"name"`
+
+	// +required
+	// +kubebuilder:validation:Required
+	// PluginName specifies the plugin to be instantiated.
+	PluginName string `json:"pluginName"`
+
+	// +optional
+	// Parameters are the set of parameters to be passed to the plugin's
+	// factory function. The factory function is responsible
+	// to parse the parameters.
+	Parameters json.RawMessage `json:"parameters"`
+}
+
+// SchedulingProfile contains the information to create a SchedulingProfile
+// entry to be used by the scheduler.
+type SchedulingProfile struct {
+	// +kubebuilder:validation:Required
+	// Name specifies the name of this SchedulingProfile
+	Name string `json:"name"`
+
+	// +required
+	// +kubebuilder:validation:Required
+	// Plugins is the list of plugins for this SchedulingProfile. They are assigned
+	// to the appropriate "slots" based on their type.
+	Plugins []SchedulingPlugin `json:"plugins"`
+}
+
+// SchedulingPlugin describes a plugin that will be associated with a
+// SchedulingProfile entry.
+type SchedulingPlugin struct {
+	// +required
+	// +kubebuilder:validation:Required
+	// PluginRef specifies a partiular Plugin instance to be associated with
+	// this SchedulingProfile. The reference is to the name of an
+	// entry of the Plugins defined in the configuration's Plugins
+	// section
+	PluginRef string `json:"pluginRef"`
+
+	// +optional
+	// Weight is the weight fo be used if this plugin is a Scorer.
+	Weight *int `json:"weight"`
+}
diff --git a/api/config/v1alpha1/zz_generated.deepcopy.go b/api/config/v1alpha1/zz_generated.deepcopy.go
new file mode 100644
index 0000000..1326f35
--- /dev/null
+++ b/api/config/v1alpha1/zz_generated.deepcopy.go
@@ -0,0 +1,126 @@
+//go:build !ignore_autogenerated
+
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by controller-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"encoding/json"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+)
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *EndpointPickerConfig) DeepCopyInto(out *EndpointPickerConfig) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	if in.Plugins != nil {
+		in, out := &in.Plugins, &out.Plugins
+		*out = make([]PluginSpec, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.SchedulingProfiles != nil {
+		in, out := &in.SchedulingProfiles, &out.SchedulingProfiles
+		*out = make([]SchedulingProfile, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new EndpointPickerConfig.
+func (in *EndpointPickerConfig) DeepCopy() *EndpointPickerConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(EndpointPickerConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *EndpointPickerConfig) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *PluginSpec) DeepCopyInto(out *PluginSpec) {
+	*out = *in
+	if in.Parameters != nil {
+		in, out := &in.Parameters, &out.Parameters
+		*out = make(json.RawMessage, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PluginSpec.
+func (in *PluginSpec) DeepCopy() *PluginSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(PluginSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *SchedulingPlugin) DeepCopyInto(out *SchedulingPlugin) {
+	*out = *in
+	if in.Weight != nil {
+		in, out := &in.Weight, &out.Weight
+		*out = new(int)
+		**out = **in
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SchedulingPlugin.
+func (in *SchedulingPlugin) DeepCopy() *SchedulingPlugin {
+	if in == nil {
+		return nil
+	}
+	out := new(SchedulingPlugin)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *SchedulingProfile) DeepCopyInto(out *SchedulingProfile) {
+	*out = *in
+	if in.Plugins != nil {
+		in, out := &in.Plugins, &out.Plugins
+		*out = make([]SchedulingPlugin, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new SchedulingProfile.
+func (in *SchedulingProfile) DeepCopy() *SchedulingProfile {
+	if in == nil {
+		return nil
+	}
+	out := new(SchedulingProfile)
+	in.DeepCopyInto(out)
+	return out
+}
diff --git a/api/config/v1alpha1/zz_generated.defaults.go b/api/config/v1alpha1/zz_generated.defaults.go
new file mode 100644
index 0000000..1cf0c42
--- /dev/null
+++ b/api/config/v1alpha1/zz_generated.defaults.go
@@ -0,0 +1,38 @@
+//go:build !ignore_autogenerated
+// +build !ignore_autogenerated
+
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by defaulter-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	runtime "k8s.io/apimachinery/pkg/runtime"
+)
+
+// RegisterDefaults adds defaulters functions to the given scheme.
+// Public to allow building arbitrary schemes.
+// All generated defaulters are covering - they call all nested defaulters.
+func RegisterDefaults(scheme *runtime.Scheme) error {
+	scheme.AddTypeDefaultingFunc(&EndpointPickerConfig{}, func(obj interface{}) { SetObjectDefaults_EndpointPickerConfig(obj.(*EndpointPickerConfig)) })
+	return nil
+}
+
+func SetObjectDefaults_EndpointPickerConfig(in *EndpointPickerConfig) {
+	SetDefaults_EndpointPickerConfig(in)
+}
diff --git a/api/config/v1alpha1/zz_generated.register.go b/api/config/v1alpha1/zz_generated.register.go
new file mode 100644
index 0000000..3c48d0c
--- /dev/null
+++ b/api/config/v1alpha1/zz_generated.register.go
@@ -0,0 +1,69 @@
+//go:build !ignore_autogenerated
+// +build !ignore_autogenerated
+
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+// Code generated by register-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+)
+
+// GroupName specifies the group name used to register the objects.
+const GroupName = "inference.networking.x-k8s.io"
+
+// GroupVersion specifies the group and the version used to register the objects.
+var GroupVersion = v1.GroupVersion{Group: GroupName, Version: "v1alpha1"}
+
+// SchemeGroupVersion is group version used to register these objects
+// Deprecated: use GroupVersion instead.
+var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1alpha1"}
+
+// Resource takes an unqualified resource and returns a Group qualified GroupResource
+func Resource(resource string) schema.GroupResource {
+	return SchemeGroupVersion.WithResource(resource).GroupResource()
+}
+
+var (
+	// localSchemeBuilder and AddToScheme will stay in k8s.io/kubernetes.
+	SchemeBuilder      runtime.SchemeBuilder
+	localSchemeBuilder = &SchemeBuilder
+	// Deprecated: use Install instead
+	AddToScheme = localSchemeBuilder.AddToScheme
+	Install     = localSchemeBuilder.AddToScheme
+)
+
+func init() {
+	// We only register manually written functions here. The registration of the
+	// generated functions takes place in the generated files. The separation
+	// makes the code compile even when the generated files are missing.
+	localSchemeBuilder.Register(addKnownTypes)
+}
+
+// Adds the list of known types to Scheme.
+func addKnownTypes(scheme *runtime.Scheme) error {
+	scheme.AddKnownTypes(SchemeGroupVersion,
+		&EndpointPickerConfig{},
+	)
+	// AddToGroupVersion allows the serialization of client types like ListOptions.
+	v1.AddToGroupVersion(scheme, SchemeGroupVersion)
+	return nil
+}
diff --git a/api/v1alpha2/inferencemodel_types.go b/api/v1alpha2/inferencemodel_types.go
index 7cd98a7..28d2b58 100644
--- a/api/v1alpha2/inferencemodel_types.go
+++ b/api/v1alpha2/inferencemodel_types.go
@@ -67,7 +67,7 @@ type InferenceModelSpec struct {
 	// ModelNames must be unique for a referencing InferencePool
 	// (names can be reused for a different pool in the same cluster).
 	// The modelName with the oldest creation timestamp is retained, and the incoming
-	// InferenceModel is sets the Ready status to false with a corresponding reason.
+	// InferenceModel's Ready status is set to false with a corresponding reason.
 	// In the rare case of a race condition, one Model will be selected randomly to be considered valid, and the other rejected.
 	// Names can be reserved without an underlying model configured in the pool.
 	// This can be done by specifying a target model and setting the weight to zero,
diff --git a/api/v1alpha2/inferencepool_types.go b/api/v1alpha2/inferencepool_types.go
index 7018ba2..fc7cfa3 100644
--- a/api/v1alpha2/inferencepool_types.go
+++ b/api/v1alpha2/inferencepool_types.go
@@ -31,7 +31,11 @@ type InferencePool struct {
 	metav1.TypeMeta   `json:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty"`
 
-	Spec   InferencePoolSpec   `json:"spec,omitempty"`
+	Spec InferencePoolSpec `json:"spec,omitempty"`
+
+	// Status defines the observed state of InferencePool.
+	//
+	// +kubebuilder:default={parent: {{parentRef: {kind: "Status", name: "default"}, conditions: {{type: "Accepted", status: "Unknown", reason: "Pending", message: "Waiting for controller", lastTransitionTime: "1970-01-01T00:00:00Z"}}}}}
 	Status InferencePoolStatus `json:"status,omitempty"`
 }
 
@@ -80,14 +84,18 @@ type EndpointPickerConfig struct {
 
 // Extension specifies how to configure an extension that runs the endpoint picker.
 type Extension struct {
-	// Reference is a reference to a service extension.
+	// Reference is a reference to a service extension. When ExtensionReference is invalid,
+	// a 5XX status code MUST be returned for the request that would have otherwise been routed
+	// to the invalid backend.
 	ExtensionReference `json:",inline"`
 
 	// ExtensionConnection configures the connection between the gateway and the extension.
 	ExtensionConnection `json:",inline"`
 }
 
-// ExtensionReference is a reference to the extension deployment.
+// ExtensionReference is a reference to the extension deployment. When ExtensionReference is invalid,
+// a 5XX status code MUST be returned for the request that would have otherwise been routed to the
+// invalid backend.
 type ExtensionReference struct {
 	// Group is the group of the referent.
 	// The default value is "", representing the Core API group.
@@ -146,14 +154,19 @@ const (
 	FailClose ExtensionFailureMode = "FailClose"
 )
 
-// InferencePoolStatus defines the observed state of InferencePool
+// InferencePoolStatus defines the observed state of InferencePool.
 type InferencePoolStatus struct {
 	// Parents is a list of parent resources (usually Gateways) that are
-	// associated with the route, and the status of the InferencePool with respect to
+	// associated with the InferencePool, and the status of the InferencePool with respect to
 	// each parent.
 	//
-	// A maximum of 32 Gateways will be represented in this list. An empty list
-	// means the route has not been attached to any Gateway.
+	// A maximum of 32 Gateways will be represented in this list. When the list contains
+	// `kind: Status, name: default`, it indicates that the InferencePool is not
+	// associated with any Gateway and a controller must perform the following:
+	//
+	//  - Remove the parent when setting the "Accepted" condition.
+	//  - Add the parent when the controller will no longer manage the InferencePool
+	//    and no other parents exist.
 	//
 	// +kubebuilder:validation:MaxItems=32
 	Parents []PoolStatus `json:"parent,omitempty"`
@@ -186,7 +199,7 @@ type InferencePoolConditionType string
 type InferencePoolReason string
 
 const (
-	// This condition indicates whether the route has been accepted or rejected
+	// This condition indicates whether the InferencePool has been accepted or rejected
 	// by a Gateway, and why.
 	//
 	// Possible reasons for this condition to be True are:
@@ -205,7 +218,7 @@ const (
 	// prefer to use the reasons listed above to improve interoperability.
 	InferencePoolConditionAccepted InferencePoolConditionType = "Accepted"
 
-	// This reason is used with the "Accepted" condition when the Route has been
+	// This reason is used with the "Accepted" condition when the InferencePool has been
 	// accepted by the Gateway.
 	InferencePoolReasonAccepted InferencePoolReason = "Accepted"
 
@@ -215,7 +228,7 @@ const (
 	InferencePoolReasonNotSupportedByGateway InferencePoolReason = "NotSupportedByGateway"
 
 	// This reason is used with the "Accepted" when a controller has not yet
-	// reconciled the route.
+	// reconciled the InferencePool.
 	InferencePoolReasonPending InferencePoolReason = "Pending"
 )
 
@@ -229,7 +242,7 @@ const (
 	//
 	// Possible reasons for this condition to be False are:
 	//
-	// * "InvalidExtnesionRef"
+	// * "InvalidExtensionRef"
 	//
 	// Controllers MAY raise this condition with other reasons, but should
 	// prefer to use the reasons listed above to improve interoperability.
diff --git a/cmd/bbr/health.go b/cmd/bbr/health.go
index 7d1b5fd..f42ca2b 100644
--- a/cmd/bbr/health.go
+++ b/cmd/bbr/health.go
@@ -19,6 +19,7 @@ package main
 import (
 	"context"
 
+	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
 	"github.com/go-logr/logr"
 	"google.golang.org/grpc/codes"
 	healthPb "google.golang.org/grpc/health/grpc_health_v1"
@@ -31,10 +32,31 @@ type healthServer struct {
 }
 
 func (s *healthServer) Check(ctx context.Context, in *healthPb.HealthCheckRequest) (*healthPb.HealthCheckResponse, error) {
+	// TODO: we're accepting ANY service name for now as a temporary hack in alignment with
+	// upstream issues. See https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/788
+	// if in.Service != extProcPb.ExternalProcessor_ServiceDesc.ServiceName {
+	// 	s.logger.V(logutil.DEFAULT).Info("gRPC health check requested unknown service", "available-services", []string{extProcPb.ExternalProcessor_ServiceDesc.ServiceName}, "requested-service", in.Service)
+	// 	return &healthPb.HealthCheckResponse{Status: healthPb.HealthCheckResponse_SERVICE_UNKNOWN}, nil
+	// }
+
 	s.logger.V(logutil.VERBOSE).Info("gRPC health check serving", "service", in.Service)
 	return &healthPb.HealthCheckResponse{Status: healthPb.HealthCheckResponse_SERVING}, nil
 }
 
+func (s *healthServer) List(ctx context.Context, _ *healthPb.HealthListRequest) (*healthPb.HealthListResponse, error) {
+	// currently only the ext_proc service is provided
+	serviceHealthResponse, err := s.Check(ctx, &healthPb.HealthCheckRequest{Service: extProcPb.ExternalProcessor_ServiceDesc.ServiceName})
+	if err != nil {
+		return nil, err
+	}
+
+	return &healthPb.HealthListResponse{
+		Statuses: map[string]*healthPb.HealthCheckResponse{
+			extProcPb.ExternalProcessor_ServiceDesc.ServiceName: serviceHealthResponse,
+		},
+	}, nil
+}
+
 func (s *healthServer) Watch(in *healthPb.HealthCheckRequest, srv healthPb.Health_WatchServer) error {
 	return status.Error(codes.Unimplemented, "Watch is not implemented")
 }
diff --git a/cmd/epp/main.go b/cmd/epp/main.go
index cef2649..b5e0617 100644
--- a/cmd/epp/main.go
+++ b/cmd/epp/main.go
@@ -17,322 +17,20 @@ limitations under the License.
 package main
 
 import (
-	"flag"
-	"fmt"
 	"os"
 
-	"github.com/go-logr/logr"
-	"github.com/prometheus/client_golang/prometheus"
-	uberzap "go.uber.org/zap"
-	"go.uber.org/zap/zapcore"
-	"google.golang.org/grpc"
-	healthPb "google.golang.org/grpc/health/grpc_health_v1"
-	"k8s.io/apimachinery/pkg/types"
 	ctrl "sigs.k8s.io/controller-runtime"
-	"sigs.k8s.io/controller-runtime/pkg/log"
-	"sigs.k8s.io/controller-runtime/pkg/log/zap"
-	"sigs.k8s.io/controller-runtime/pkg/manager"
-	"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
-	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
 
-	conformance_epp "sigs.k8s.io/gateway-api-inference-extension/conformance/testing-epp"
-	"sigs.k8s.io/gateway-api-inference-extension/internal/runnable"
-	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/datastore"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics/collectors"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/requestcontrol"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/saturationdetector"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
-	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
-	runserver "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
-	envutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/env"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
+	"sigs.k8s.io/gateway-api-inference-extension/cmd/epp/runner"
 )
 
-var (
-	grpcPort = flag.Int(
-		"grpcPort",
-		runserver.DefaultGrpcPort,
-		"The gRPC port used for communicating with Envoy proxy")
-	grpcHealthPort = flag.Int(
-		"grpcHealthPort",
-		9003,
-		"The port used for gRPC liveness and readiness probes")
-	metricsPort = flag.Int(
-		"metricsPort", 9090, "The metrics port")
-	destinationEndpointHintKey = flag.String(
-		"destinationEndpointHintKey",
-		runserver.DefaultDestinationEndpointHintKey,
-		"Header and response metadata key used by Envoy to route to the appropriate pod. This must match Envoy configuration.")
-	destinationEndpointHintMetadataNamespace = flag.String(
-		"DestinationEndpointHintMetadataNamespace",
-		runserver.DefaultDestinationEndpointHintMetadataNamespace,
-		"The key for the outer namespace struct in the metadata field of the extproc response that is used to wrap the"+
-			"target endpoint. If not set, then an outer namespace struct should not be created.")
-	poolName = flag.String(
-		"poolName",
-		runserver.DefaultPoolName,
-		"Name of the InferencePool this Endpoint Picker is associated with.")
-	poolNamespace = flag.String(
-		"poolNamespace",
-		runserver.DefaultPoolNamespace,
-		"Namespace of the InferencePool this Endpoint Picker is associated with.")
-	refreshMetricsInterval = flag.Duration(
-		"refreshMetricsInterval",
-		runserver.DefaultRefreshMetricsInterval,
-		"interval to refresh metrics")
-	refreshPrometheusMetricsInterval = flag.Duration(
-		"refreshPrometheusMetricsInterval",
-		runserver.DefaultRefreshPrometheusMetricsInterval,
-		"interval to flush prometheus metrics")
-	logVerbosity  = flag.Int("v", logging.DEFAULT, "number for the log level verbosity")
-	secureServing = flag.Bool(
-		"secureServing", runserver.DefaultSecureServing, "Enables secure serving. Defaults to true.")
-	certPath = flag.String(
-		"certPath", "", "The path to the certificate for secure serving. The certificate and private key files "+
-			"are assumed to be named tls.crt and tls.key, respectively. If not set, and secureServing is enabled, "+
-			"then a self-signed certificate is used.")
-	// metric flags
-	totalQueuedRequestsMetric = flag.String("totalQueuedRequestsMetric",
-		"vllm:num_requests_waiting",
-		"Prometheus metric for the number of queued requests.")
-	kvCacheUsagePercentageMetric = flag.String("kvCacheUsagePercentageMetric",
-		"vllm:gpu_cache_usage_perc",
-		"Prometheus metric for the fraction of KV-cache blocks currently in use (from 0 to 1).")
-	// LoRA metrics
-	loraInfoMetric = flag.String("loraInfoMetric",
-		"vllm:lora_requests_info",
-		"Prometheus metric for the LoRA info metrics (must be in vLLM label format).")
-
-	setupLog = ctrl.Log.WithName("setup")
-
-	// Environment variables
-	schedulerV2                       = envutil.GetEnvBool("EXPERIMENTAL_USE_SCHEDULER_V2", false, setupLog)
-	prefixCacheScheduling             = envutil.GetEnvBool("ENABLE_PREFIX_CACHE_SCHEDULING", false, setupLog)
-	reqHeaderBasedSchedulerForTesting = envutil.GetEnvBool("ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING", false, setupLog)
-)
-
-func loadPrefixCacheConfig() prefix.Config {
-	baseLogger := log.Log.WithName("env-config")
-
-	return prefix.Config{
-		HashBlockSize:          envutil.GetEnvInt("PREFIX_CACHE_HASH_BLOCK_SIZE", prefix.DefaultHashBlockSize, baseLogger),
-		MaxPrefixBlocksToMatch: envutil.GetEnvInt("PREFIX_CACHE_MAX_PREFIX_BLOCKS", prefix.DefaultMaxPrefixBlocks, baseLogger),
-		LRUIndexerCapacity:     envutil.GetEnvInt("PREFIX_CACHE_LRU_CAPACITY", prefix.DefaultLRUIndexerCapacity, baseLogger),
-	}
-}
-
 func main() {
-	if err := run(); err != nil {
-		os.Exit(1)
-	}
-}
-
-func run() error {
-	opts := zap.Options{
-		Development: true,
-	}
-	opts.BindFlags(flag.CommandLine)
-	flag.Parse()
-	initLogging(&opts)
-
-	// Validate flags
-	if err := validateFlags(); err != nil {
-		setupLog.Error(err, "Failed to validate flags")
-		return err
-	}
-
-	// Print all flag values
-	flags := make(map[string]any)
-	flag.VisitAll(func(f *flag.Flag) {
-		flags[f.Name] = f.Value
-	})
-	setupLog.Info("Flags processed", "flags", flags)
-
-	// --- Load Configurations from Environment Variables ---
-	sdConfig := saturationdetector.LoadConfigFromEnv()
-
-	// --- Get Kubernetes Config ---
-	cfg, err := ctrl.GetConfig()
-	if err != nil {
-		setupLog.Error(err, "Failed to get Kubernetes rest config")
-		return err
-	}
-
-	// --- Setup Datastore ---
-	mapping, err := backendmetrics.NewMetricMapping(
-		*totalQueuedRequestsMetric,
-		*kvCacheUsagePercentageMetric,
-		*loraInfoMetric,
-	)
-	if err != nil {
-		setupLog.Error(err, "Failed to create metric mapping from flags.")
-		return err
-	}
-	verifyMetricMapping(*mapping, setupLog)
-	pmf := backendmetrics.NewPodMetricsFactory(&backendmetrics.PodMetricsClientImpl{MetricMapping: mapping}, *refreshMetricsInterval)
-	ctx := ctrl.SetupSignalHandler()
-	datastore := datastore.NewDatastore(ctx, pmf)
-
-	// --- Setup Metrics Server ---
-	customCollectors := []prometheus.Collector{collectors.NewInferencePoolMetricsCollector(datastore)}
-	metrics.Register(customCollectors...)
-	metrics.RecordInferenceExtensionInfo()
-	// Register metrics handler.
-	// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
-	// More info:
-	// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/server
-	// - https://book.kubebuilder.io/reference/metrics.html
-	metricsServerOptions := metricsserver.Options{
-		BindAddress:    fmt.Sprintf(":%d", *metricsPort),
-		FilterProvider: filters.WithAuthenticationAndAuthorization,
-	}
-
-	poolNamespacedName := types.NamespacedName{
-		Name:      *poolName,
-		Namespace: *poolNamespace,
-	}
-	mgr, err := runserver.NewDefaultManager(poolNamespacedName, cfg, metricsServerOptions)
-	if err != nil {
-		setupLog.Error(err, "Failed to create controller manager")
-		return err
-	}
-
-	// --- Initialize Core EPP Components ---
-	scheduler := scheduling.NewScheduler(datastore)
-	if schedulerV2 {
-		queueScorerWeight := envutil.GetEnvInt("QUEUE_SCORE_WEIGHT", scorer.DefaultQueueScorerWeight, setupLog)
-		kvCacheScorerWeight := envutil.GetEnvInt("KV_CACHE_SCORE_WEIGHT", scorer.DefaultKVCacheScorerWeight, setupLog)
-
-		schedulerProfile := framework.NewSchedulerProfile().
-			WithScorers(framework.NewWeightedScorer(&scorer.QueueScorer{}, queueScorerWeight),
-				framework.NewWeightedScorer(&scorer.KVCacheScorer{}, kvCacheScorerWeight)).
-			WithPicker(picker.NewMaxScorePicker())
-
-		if prefixCacheScheduling {
-			prefixScorerWeight := envutil.GetEnvInt("PREFIX_CACHE_SCORE_WEIGHT", prefix.DefaultScorerWeight, setupLog)
-			if err := schedulerProfile.AddPlugins(framework.NewWeightedScorer(prefix.New(loadPrefixCacheConfig()), prefixScorerWeight)); err != nil {
-				setupLog.Error(err, "Failed to register scheduler plugins")
-				return err
-			}
-		}
-
-		schedulerConfig := scheduling.NewSchedulerConfig(profilepicker.NewAllProfilesPicker(), map[string]*framework.SchedulerProfile{"schedulerv2": schedulerProfile})
-		scheduler = scheduling.NewSchedulerWithConfig(datastore, schedulerConfig)
-	}
+	// Register all known plugin factories
+	runner.RegisterAllPlugins()
+	// For adding out-of-tree plugins to the plugins registry, use the following:
+	// plugins.Register(my-out-of-tree-plugin-name, my-out-of-tree-plugin-factory-function)
 
-	if reqHeaderBasedSchedulerForTesting {
-		scheduler = conformance_epp.NewReqHeaderBasedScheduler(datastore)
-	}
-
-	saturationDetector := saturationdetector.NewDetector(sdConfig, datastore, ctrl.Log)
-
-	director := requestcontrol.NewDirector(datastore, scheduler, saturationDetector) // can call "director.WithPostResponsePlugins" to add post response plugins
-
-	// --- Setup ExtProc Server Runner ---
-	serverRunner := &runserver.ExtProcServerRunner{
-		GrpcPort:                                 *grpcPort,
-		DestinationEndpointHintMetadataNamespace: *destinationEndpointHintMetadataNamespace,
-		DestinationEndpointHintKey:               *destinationEndpointHintKey,
-		PoolNamespacedName:                       poolNamespacedName,
-		Datastore:                                datastore,
-		SecureServing:                            *secureServing,
-		CertPath:                                 *certPath,
-		RefreshPrometheusMetricsInterval:         *refreshPrometheusMetricsInterval,
-		Director:                                 director,
-		SaturationDetector:                       saturationDetector,
-	}
-	if err := serverRunner.SetupWithManager(ctx, mgr); err != nil {
-		setupLog.Error(err, "Failed to setup EPP controllers")
-		return err
-	}
-
-	// --- Add Runnables to Manager ---
-	// Register health server.
-	if err := registerHealthServer(mgr, ctrl.Log.WithName("health"), datastore, *grpcHealthPort); err != nil {
-		return err
-	}
-
-	// Register ext-proc server.
-	if err := registerExtProcServer(mgr, serverRunner, ctrl.Log.WithName("ext-proc")); err != nil {
-		return err
-	}
-
-	// --- Start Manager ---
-	// This blocks until a signal is received.
-	setupLog.Info("Controller manager starting")
-	if err := mgr.Start(ctx); err != nil {
-		setupLog.Error(err, "Error starting controller manager")
-		return err
-	}
-	setupLog.Info("Controller manager terminated")
-	return nil
-}
-
-func initLogging(opts *zap.Options) {
-	// Unless -zap-log-level is explicitly set, use -v
-	useV := true
-	flag.Visit(func(f *flag.Flag) {
-		if f.Name == "zap-log-level" {
-			useV = false
-		}
-	})
-	if useV {
-		// See https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/log/zap#Options.Level
-		lvl := -1 * (*logVerbosity)
-		opts.Level = uberzap.NewAtomicLevelAt(zapcore.Level(int8(lvl)))
-	}
-
-	logger := zap.New(zap.UseFlagOptions(opts), zap.RawZapOpts(uberzap.AddCaller()))
-	ctrl.SetLogger(logger)
-}
-
-// registerExtProcServer adds the ExtProcServerRunner as a Runnable to the manager.
-func registerExtProcServer(mgr manager.Manager, runner *runserver.ExtProcServerRunner, logger logr.Logger) error {
-	if err := mgr.Add(runner.AsRunnable(logger)); err != nil {
-		setupLog.Error(err, "Failed to register ext-proc gRPC server runnable")
-		return err
-	}
-	setupLog.Info("ExtProc server runner added to manager.")
-	return nil
-}
-
-// registerHealthServer adds the Health gRPC server as a Runnable to the given manager.
-func registerHealthServer(mgr manager.Manager, logger logr.Logger, ds datastore.Datastore, port int) error {
-	srv := grpc.NewServer()
-	healthPb.RegisterHealthServer(srv, &healthServer{
-		logger:    logger,
-		datastore: ds,
-	})
-	if err := mgr.Add(
-		runnable.NoLeaderElection(runnable.GRPCServer("health", srv, port))); err != nil {
-		setupLog.Error(err, "Failed to register health server")
-		return err
-	}
-	return nil
-}
-
-func validateFlags() error {
-	if *poolName == "" {
-		return fmt.Errorf("required %q flag not set", "poolName")
-	}
-
-	return nil
-}
-
-func verifyMetricMapping(mapping backendmetrics.MetricMapping, logger logr.Logger) {
-	if mapping.TotalQueuedRequests == nil {
-		logger.Info("Not scraping metric: TotalQueuedRequests")
-	}
-	if mapping.KVCacheUtilization == nil {
-		logger.Info("Not scraping metric: KVCacheUtilization")
-	}
-	if mapping.LoraRequestInfo == nil {
-		logger.Info("Not scraping metric: LoraRequestInfo")
+	if err := runner.NewRunner().Run(ctrl.SetupSignalHandler()); err != nil {
+		os.Exit(1)
 	}
 }
diff --git a/cmd/epp/health.go b/cmd/epp/runner/health.go
similarity index 58%
rename from cmd/epp/health.go
rename to cmd/epp/runner/health.go
index 9369700..283541e 100644
--- a/cmd/epp/health.go
+++ b/cmd/epp/runner/health.go
@@ -14,11 +14,12 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-package main
+package runner
 
 import (
 	"context"
 
+	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
 	"github.com/go-logr/logr"
 	"google.golang.org/grpc/codes"
 	healthPb "google.golang.org/grpc/health/grpc_health_v1"
@@ -33,6 +34,13 @@ type healthServer struct {
 }
 
 func (s *healthServer) Check(ctx context.Context, in *healthPb.HealthCheckRequest) (*healthPb.HealthCheckResponse, error) {
+	// TODO: we're accepting ANY service name for now as a temporary hack in alignment with
+	// upstream issues. See https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/788
+	// if in.Service != extProcPb.ExternalProcessor_ServiceDesc.ServiceName {
+	// 	s.logger.V(logutil.DEFAULT).Info("gRPC health check requested unknown service", "available-services", []string{extProcPb.ExternalProcessor_ServiceDesc.ServiceName}, "requested-service", in.Service)
+	// 	return &healthPb.HealthCheckResponse{Status: healthPb.HealthCheckResponse_SERVICE_UNKNOWN}, nil
+	// }
+
 	if !s.datastore.PoolHasSynced() {
 		s.logger.V(logutil.DEFAULT).Info("gRPC health check not serving", "service", in.Service)
 		return &healthPb.HealthCheckResponse{Status: healthPb.HealthCheckResponse_NOT_SERVING}, nil
@@ -41,6 +49,20 @@ func (s *healthServer) Check(ctx context.Context, in *healthPb.HealthCheckReques
 	return &healthPb.HealthCheckResponse{Status: healthPb.HealthCheckResponse_SERVING}, nil
 }
 
+func (s *healthServer) List(ctx context.Context, _ *healthPb.HealthListRequest) (*healthPb.HealthListResponse, error) {
+	// currently only the ext_proc service is provided
+	serviceHealthResponse, err := s.Check(ctx, &healthPb.HealthCheckRequest{Service: extProcPb.ExternalProcessor_ServiceDesc.ServiceName})
+	if err != nil {
+		return nil, err
+	}
+
+	return &healthPb.HealthListResponse{
+		Statuses: map[string]*healthPb.HealthCheckResponse{
+			extProcPb.ExternalProcessor_ServiceDesc.ServiceName: serviceHealthResponse,
+		},
+	}, nil
+}
+
 func (s *healthServer) Watch(in *healthPb.HealthCheckRequest, srv healthPb.Health_WatchServer) error {
 	return status.Error(codes.Unimplemented, "Watch is not implemented")
 }
diff --git a/cmd/epp/runner/register.go b/cmd/epp/runner/register.go
new file mode 100644
index 0000000..e2ce601
--- /dev/null
+++ b/cmd/epp/runner/register.go
@@ -0,0 +1,87 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package runner
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
+)
+
+// RegisterAllPlugins registers the factory functions of all known plugins
+func RegisterAllPlugins() {
+	plugins.Register(filter.LeastKVCacheFilterType, filter.LeastKVCacheFilterFactory)
+	plugins.Register(filter.LeastQueueFilterType, filter.LeastQueueFilterFactory)
+	plugins.Register(filter.LoraAffinityFilterType, filter.LoraAffinityFilterFactory)
+	plugins.Register(filter.LowQueueFilterType, filter.LowQueueFilterFactory)
+	plugins.Register(prefix.PrefixCachePluginType, prefix.PrefixCachePluginFactory)
+	plugins.Register(picker.MaxScorePickerType, picker.MaxScorePickerFactory)
+	plugins.Register(picker.RandomPickerType, picker.RandomPickerFactory)
+	plugins.Register(profile.SingleProfileHandlerType, profile.SingleProfileHandlerFactory)
+	plugins.Register(scorer.KvCacheScorerType, scorer.KvCacheScorerFactory)
+	plugins.Register(scorer.QueueScorerType, scorer.QueueScorerFactory)
+}
+
+// eppHandle is an implementation of the interface plugins.Handle
+type eppHandle struct {
+	plugins plugins.HandlePlugins
+}
+
+// Plugins returns the sub-handle for working with instantiated plugins
+func (h *eppHandle) Plugins() plugins.HandlePlugins {
+	return h.plugins
+}
+
+// eppHandlePlugins implements the set of APIs to work with instantiated plugins
+type eppHandlePlugins struct {
+	thePlugins map[string]plugins.Plugin
+}
+
+// Plugin returns the named plugin instance
+func (h *eppHandlePlugins) Plugin(name string) plugins.Plugin {
+	return h.thePlugins[name]
+}
+
+// AddPlugin adds a plugin to the set of known plugin instances
+func (h *eppHandlePlugins) AddPlugin(name string, plugin plugins.Plugin) {
+	h.thePlugins[name] = plugin
+}
+
+// GetAllPlugins returns all of the known plugins
+func (h *eppHandlePlugins) GetAllPlugins() []plugins.Plugin {
+	result := make([]plugins.Plugin, 0)
+	for _, plugin := range h.thePlugins {
+		result = append(result, plugin)
+	}
+	return result
+}
+
+// GetAllPluginsWithNames returns al of the known plugins with their names
+func (h *eppHandlePlugins) GetAllPluginsWithNames() map[string]plugins.Plugin {
+	return h.thePlugins
+}
+
+func newEppHandle() *eppHandle {
+	return &eppHandle{
+		plugins: &eppHandlePlugins{
+			thePlugins: map[string]plugins.Plugin{},
+		},
+	}
+}
diff --git a/cmd/epp/runner/runner.go b/cmd/epp/runner/runner.go
new file mode 100644
index 0000000..7bb55b7
--- /dev/null
+++ b/cmd/epp/runner/runner.go
@@ -0,0 +1,403 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package runner
+
+import (
+	"context"
+	"flag"
+	"fmt"
+
+	"github.com/go-logr/logr"
+	"github.com/prometheus/client_golang/prometheus"
+	uberzap "go.uber.org/zap"
+	"go.uber.org/zap/zapcore"
+	"google.golang.org/grpc"
+	healthPb "google.golang.org/grpc/health/grpc_health_v1"
+	"k8s.io/apimachinery/pkg/types"
+	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/log/zap"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
+	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
+
+	conformance_epp "sigs.k8s.io/gateway-api-inference-extension/conformance/testing-epp"
+	"sigs.k8s.io/gateway-api-inference-extension/internal/runnable"
+	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/common/config/loader"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/datastore"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics/collectors"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/requestcontrol"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/saturationdetector"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
+	runserver "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
+	envutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/env"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
+)
+
+var (
+	grpcPort = flag.Int(
+		"grpcPort",
+		runserver.DefaultGrpcPort,
+		"The gRPC port used for communicating with Envoy proxy")
+	grpcHealthPort = flag.Int(
+		"grpcHealthPort",
+		9003,
+		"The port used for gRPC liveness and readiness probes")
+	metricsPort = flag.Int(
+		"metricsPort", 9090, "The metrics port")
+	destinationEndpointHintKey = flag.String(
+		"destinationEndpointHintKey",
+		runserver.DefaultDestinationEndpointHintKey,
+		"Header and response metadata key used by Envoy to route to the appropriate pod. This must match Envoy configuration.")
+	destinationEndpointHintMetadataNamespace = flag.String(
+		"DestinationEndpointHintMetadataNamespace",
+		runserver.DefaultDestinationEndpointHintMetadataNamespace,
+		"The key for the outer namespace struct in the metadata field of the extproc response that is used to wrap the"+
+			"target endpoint. If not set, then an outer namespace struct should not be created.")
+	poolName = flag.String(
+		"poolName",
+		runserver.DefaultPoolName,
+		"Name of the InferencePool this Endpoint Picker is associated with.")
+	poolNamespace = flag.String(
+		"poolNamespace",
+		runserver.DefaultPoolNamespace,
+		"Namespace of the InferencePool this Endpoint Picker is associated with.")
+	refreshMetricsInterval = flag.Duration(
+		"refreshMetricsInterval",
+		runserver.DefaultRefreshMetricsInterval,
+		"interval to refresh metrics")
+	refreshPrometheusMetricsInterval = flag.Duration(
+		"refreshPrometheusMetricsInterval",
+		runserver.DefaultRefreshPrometheusMetricsInterval,
+		"interval to flush prometheus metrics")
+	logVerbosity  = flag.Int("v", logging.DEFAULT, "number for the log level verbosity")
+	secureServing = flag.Bool(
+		"secureServing", runserver.DefaultSecureServing, "Enables secure serving. Defaults to true.")
+	healthChecking = flag.Bool("healthChecking", runserver.DefaultHealthChecking, "Enables health checking")
+	certPath       = flag.String(
+		"certPath", "", "The path to the certificate for secure serving. The certificate and private key files "+
+			"are assumed to be named tls.crt and tls.key, respectively. If not set, and secureServing is enabled, "+
+			"then a self-signed certificate is used.")
+	// metric flags
+	totalQueuedRequestsMetric = flag.String("totalQueuedRequestsMetric",
+		"vllm:num_requests_waiting",
+		"Prometheus metric for the number of queued requests.")
+	kvCacheUsagePercentageMetric = flag.String("kvCacheUsagePercentageMetric",
+		"vllm:gpu_cache_usage_perc",
+		"Prometheus metric for the fraction of KV-cache blocks currently in use (from 0 to 1).")
+	// LoRA metrics
+	loraInfoMetric = flag.String("loraInfoMetric",
+		"vllm:lora_requests_info",
+		"Prometheus metric for the LoRA info metrics (must be in vLLM label format).")
+	// configuration flags
+	configFile = flag.String("configFile", "", "The path to the configuration file")
+	configText = flag.String("configText", "", "The configuration specified as text, in lieu of a file")
+
+	setupLog = ctrl.Log.WithName("setup")
+
+	// Environment variables
+	schedulerV2                       = envutil.GetEnvBool("EXPERIMENTAL_USE_SCHEDULER_V2", false, setupLog)
+	prefixCacheScheduling             = envutil.GetEnvBool("ENABLE_PREFIX_CACHE_SCHEDULING", false, setupLog)
+	reqHeaderBasedSchedulerForTesting = envutil.GetEnvBool("ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING", false, setupLog)
+)
+
+// NewRunner initializes a new EPP Runner and returns its pointer.
+func NewRunner() *Runner {
+	return &Runner{
+		requestControlConfig: requestcontrol.NewConfig(), // default requestcontrol config has empty plugin list
+	}
+}
+
+// Runner is used to run epp with its plugins
+type Runner struct {
+	requestControlConfig *requestcontrol.Config
+	schedulerConfig      *scheduling.SchedulerConfig
+}
+
+func (r *Runner) WithRequestControlConfig(requestControlConfig *requestcontrol.Config) *Runner {
+	r.requestControlConfig = requestControlConfig
+	return r
+}
+
+func (r *Runner) WithSchedulerConfig(schedulerConfig *scheduling.SchedulerConfig) *Runner {
+	r.schedulerConfig = schedulerConfig
+	return r
+}
+
+func (r *Runner) Run(ctx context.Context) error {
+	opts := zap.Options{
+		Development: true,
+	}
+	opts.BindFlags(flag.CommandLine)
+	flag.Parse()
+	initLogging(&opts)
+
+	// Validate flags
+	if err := validateFlags(); err != nil {
+		setupLog.Error(err, "Failed to validate flags")
+		return err
+	}
+
+	// Print all flag values
+	flags := make(map[string]any)
+	flag.VisitAll(func(f *flag.Flag) {
+		flags[f.Name] = f.Value
+	})
+	setupLog.Info("Flags processed", "flags", flags)
+
+	// --- Load Configurations from Environment Variables ---
+	sdConfig := saturationdetector.LoadConfigFromEnv()
+
+	// --- Get Kubernetes Config ---
+	cfg, err := ctrl.GetConfig()
+	if err != nil {
+		setupLog.Error(err, "Failed to get Kubernetes rest config")
+		return err
+	}
+
+	// --- Setup Datastore ---
+	mapping, err := backendmetrics.NewMetricMapping(
+		*totalQueuedRequestsMetric,
+		*kvCacheUsagePercentageMetric,
+		*loraInfoMetric,
+	)
+	if err != nil {
+		setupLog.Error(err, "Failed to create metric mapping from flags.")
+		return err
+	}
+	verifyMetricMapping(*mapping, setupLog)
+	pmf := backendmetrics.NewPodMetricsFactory(&backendmetrics.PodMetricsClientImpl{MetricMapping: mapping}, *refreshMetricsInterval)
+
+	datastore := datastore.NewDatastore(ctx, pmf)
+
+	// --- Setup Metrics Server ---
+	customCollectors := []prometheus.Collector{collectors.NewInferencePoolMetricsCollector(datastore)}
+	metrics.Register(customCollectors...)
+	metrics.RecordInferenceExtensionInfo()
+	// Register metrics handler.
+	// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
+	// More info:
+	// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/server
+	// - https://book.kubebuilder.io/reference/metrics.html
+	metricsServerOptions := metricsserver.Options{
+		BindAddress:    fmt.Sprintf(":%d", *metricsPort),
+		FilterProvider: filters.WithAuthenticationAndAuthorization,
+	}
+
+	poolNamespacedName := types.NamespacedName{
+		Name:      *poolName,
+		Namespace: *poolNamespace,
+	}
+	mgr, err := runserver.NewDefaultManager(poolNamespacedName, cfg, metricsServerOptions)
+	if err != nil {
+		setupLog.Error(err, "Failed to create controller manager")
+		return err
+	}
+
+	if len(*configText) != 0 || len(*configFile) != 0 {
+		theConfig, err := loader.LoadConfig([]byte(*configText), *configFile)
+		if err != nil {
+			setupLog.Error(err, "Failed to load the configuration")
+			return err
+		}
+
+		epp := newEppHandle()
+
+		err = loader.LoadPluginReferences(theConfig.Plugins, epp)
+		if err != nil {
+			setupLog.Error(err, "Failed to instantiate the plugins")
+			return err
+		}
+
+		r.schedulerConfig, err = loader.LoadSchedulerConfig(theConfig.SchedulingProfiles, epp)
+		if err != nil {
+			setupLog.Error(err, "Failed to create Scheduler configuration")
+			return err
+		}
+
+		// Add requestControl plugins
+		r.requestControlConfig.AddPlugins(epp.Plugins().GetAllPlugins()...)
+	}
+
+	// --- Initialize Core EPP Components ---
+	scheduler, err := r.initializeScheduler()
+	if err != nil {
+		setupLog.Error(err, "Failed to create scheduler")
+		return err
+	}
+
+	saturationDetector := saturationdetector.NewDetector(sdConfig, datastore, ctrl.Log)
+
+	director := requestcontrol.NewDirectorWithConfig(datastore, scheduler, saturationDetector, r.requestControlConfig)
+
+	// --- Setup ExtProc Server Runner ---
+	serverRunner := &runserver.ExtProcServerRunner{
+		GrpcPort:                                 *grpcPort,
+		DestinationEndpointHintMetadataNamespace: *destinationEndpointHintMetadataNamespace,
+		DestinationEndpointHintKey:               *destinationEndpointHintKey,
+		PoolNamespacedName:                       poolNamespacedName,
+		Datastore:                                datastore,
+		SecureServing:                            *secureServing,
+		HealthChecking:                           *healthChecking,
+		CertPath:                                 *certPath,
+		RefreshPrometheusMetricsInterval:         *refreshPrometheusMetricsInterval,
+		Director:                                 director,
+		SaturationDetector:                       saturationDetector,
+	}
+	if err := serverRunner.SetupWithManager(ctx, mgr); err != nil {
+		setupLog.Error(err, "Failed to setup EPP controllers")
+		return err
+	}
+
+	// --- Add Runnables to Manager ---
+	// Register health server.
+	if err := registerHealthServer(mgr, ctrl.Log.WithName("health"), datastore, *grpcHealthPort); err != nil {
+		return err
+	}
+
+	// Register ext-proc server.
+	if err := registerExtProcServer(mgr, serverRunner, ctrl.Log.WithName("ext-proc")); err != nil {
+		return err
+	}
+
+	// --- Start Manager ---
+	// This blocks until a signal is received.
+	setupLog.Info("Controller manager starting")
+	if err := mgr.Start(ctx); err != nil {
+		setupLog.Error(err, "Error starting controller manager")
+		return err
+	}
+	setupLog.Info("Controller manager terminated")
+	return nil
+}
+
+func (r *Runner) initializeScheduler() (*scheduling.Scheduler, error) {
+	if r.schedulerConfig != nil {
+		return scheduling.NewSchedulerWithConfig(r.schedulerConfig), nil
+	}
+
+	// otherwise, no one configured from outside scheduler config. use existing configuration
+	scheduler := scheduling.NewScheduler()
+	if schedulerV2 {
+		queueScorerWeight := envutil.GetEnvInt("QUEUE_SCORE_WEIGHT", scorer.DefaultQueueScorerWeight, setupLog)
+		kvCacheScorerWeight := envutil.GetEnvInt("KV_CACHE_SCORE_WEIGHT", scorer.DefaultKVCacheScorerWeight, setupLog)
+
+		schedulerProfile := framework.NewSchedulerProfile().
+			WithScorers(framework.NewWeightedScorer(scorer.NewQueueScorer(), queueScorerWeight),
+				framework.NewWeightedScorer(scorer.NewKVCacheScorer(), kvCacheScorerWeight)).
+			WithPicker(picker.NewMaxScorePicker())
+
+		if prefixCacheScheduling {
+			prefixScorerWeight := envutil.GetEnvInt("PREFIX_CACHE_SCORE_WEIGHT", prefix.DefaultScorerWeight, setupLog)
+			if err := schedulerProfile.AddPlugins(framework.NewWeightedScorer(prefix.New(loadPrefixCacheConfig()), prefixScorerWeight)); err != nil {
+				return nil, fmt.Errorf("Failed to register scheduler plugins - %w", err)
+			}
+		}
+
+		schedulerConfig := scheduling.NewSchedulerConfig(profile.NewSingleProfileHandler(), map[string]*framework.SchedulerProfile{"schedulerv2": schedulerProfile})
+		scheduler = scheduling.NewSchedulerWithConfig(schedulerConfig)
+	}
+
+	if reqHeaderBasedSchedulerForTesting {
+		scheduler = conformance_epp.NewReqHeaderBasedScheduler()
+	}
+
+	return scheduler, nil
+}
+
+func initLogging(opts *zap.Options) {
+	// Unless -zap-log-level is explicitly set, use -v
+	useV := true
+	flag.Visit(func(f *flag.Flag) {
+		if f.Name == "zap-log-level" {
+			useV = false
+		}
+	})
+	if useV {
+		// See https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/log/zap#Options.Level
+		lvl := -1 * (*logVerbosity)
+		opts.Level = uberzap.NewAtomicLevelAt(zapcore.Level(int8(lvl)))
+	}
+
+	logger := zap.New(zap.UseFlagOptions(opts), zap.RawZapOpts(uberzap.AddCaller()))
+	ctrl.SetLogger(logger)
+}
+
+func loadPrefixCacheConfig() prefix.Config {
+	baseLogger := log.Log.WithName("env-config")
+
+	return prefix.Config{
+		HashBlockSize:          envutil.GetEnvInt("PREFIX_CACHE_HASH_BLOCK_SIZE", prefix.DefaultHashBlockSize, baseLogger),
+		MaxPrefixBlocksToMatch: envutil.GetEnvInt("PREFIX_CACHE_MAX_PREFIX_BLOCKS", prefix.DefaultMaxPrefixBlocks, baseLogger),
+		LRUCapacityPerServer:   envutil.GetEnvInt("PREFIX_CACHE_LRU_CAPACITY_PER_SERVER", prefix.DefaultLRUCapacityPerServer, baseLogger),
+	}
+}
+
+// registerExtProcServer adds the ExtProcServerRunner as a Runnable to the manager.
+func registerExtProcServer(mgr manager.Manager, runner *runserver.ExtProcServerRunner, logger logr.Logger) error {
+	if err := mgr.Add(runner.AsRunnable(logger)); err != nil {
+		setupLog.Error(err, "Failed to register ext-proc gRPC server runnable")
+		return err
+	}
+	setupLog.Info("ExtProc server runner added to manager.")
+	return nil
+}
+
+// registerHealthServer adds the Health gRPC server as a Runnable to the given manager.
+func registerHealthServer(mgr manager.Manager, logger logr.Logger, ds datastore.Datastore, port int) error {
+	srv := grpc.NewServer()
+	healthPb.RegisterHealthServer(srv, &healthServer{
+		logger:    logger,
+		datastore: ds,
+	})
+	if err := mgr.Add(
+		runnable.NoLeaderElection(runnable.GRPCServer("health", srv, port))); err != nil {
+		setupLog.Error(err, "Failed to register health server")
+		return err
+	}
+	return nil
+}
+
+func validateFlags() error {
+	if *poolName == "" {
+		return fmt.Errorf("required %q flag not set", "poolName")
+	}
+	if len(*configText) != 0 && len(*configFile) != 0 {
+		return fmt.Errorf("both the %s and %s flags can not be set at the same time", "configText", "configFile")
+	}
+
+	return nil
+}
+
+func verifyMetricMapping(mapping backendmetrics.MetricMapping, logger logr.Logger) {
+	if mapping.TotalQueuedRequests == nil {
+		logger.Info("Not scraping metric: TotalQueuedRequests")
+	}
+	if mapping.KVCacheUtilization == nil {
+		logger.Info("Not scraping metric: KVCacheUtilization")
+	}
+	if mapping.LoraRequestInfo == nil {
+		logger.Info("Not scraping metric: LoraRequestInfo")
+	}
+}
diff --git a/config/charts/body-based-routing/templates/istio.yaml b/config/charts/body-based-routing/templates/istio.yaml
index 6d4535c..2c744e2 100644
--- a/config/charts/body-based-routing/templates/istio.yaml
+++ b/config/charts/body-based-routing/templates/istio.yaml
@@ -31,7 +31,7 @@ spec:
             response_trailer_mode: "SKIP"
           grpc_service:
             envoy_grpc:
-              cluster_name: outbound|{{ .Values.bbr.port }}||{{ .Values.bbr.name }}.default.svc.cluster.local
+              cluster_name: outbound|{{ .Values.bbr.port }}||{{ .Values.bbr.name }}.{{ .Release.Namespace }}.svc.cluster.local
 ---
 apiVersion: networking.istio.io/v1
 kind: DestinationRule
@@ -39,7 +39,7 @@ metadata:
   name: {{ .Values.bbr.name }}
   namespace: {{ .Release.Namespace }}
 spec:
-  host: {{ .Values.bbr.name }}.default.svc.cluster.local
+  host: {{ .Values.bbr.name }}.{{ .Release.Namespace }}.svc.cluster.local
   trafficPolicy:
       tls:
         mode: SIMPLE
diff --git a/config/charts/inferencepool/templates/rbac.yaml b/config/charts/inferencepool/templates/rbac.yaml
index cdd50c6..4cb5d7b 100644
--- a/config/charts/inferencepool/templates/rbac.yaml
+++ b/config/charts/inferencepool/templates/rbac.yaml
@@ -33,6 +33,7 @@ subjects:
   name: {{ include "gateway-api-inference-extension.name" . }}
   namespace: {{ .Release.Namespace }}
 roleRef:
+  apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: {{ include "gateway-api-inference-extension.name" . }}
 ---
diff --git a/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml b/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml
index 2880509..d76c93f 100644
--- a/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml
+++ b/config/crd/bases/inference.networking.x-k8s.io_inferencemodels.yaml
@@ -88,7 +88,7 @@ spec:
                   ModelNames must be unique for a referencing InferencePool
                   (names can be reused for a different pool in the same cluster).
                   The modelName with the oldest creation timestamp is retained, and the incoming
-                  InferenceModel is sets the Ready status to false with a corresponding reason.
+                  InferenceModel's Ready status is set to false with a corresponding reason.
                   In the rare case of a race condition, one Model will be selected randomly to be considered valid, and the other rejected.
                   Names can be reserved without an underlying model configured in the pool.
                   This can be done by specifying a target model and setting the weight to zero,
diff --git a/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml b/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml
index 8386db8..db342d0 100644
--- a/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml
+++ b/config/crd/bases/inference.networking.x-k8s.io_inferencepools.yaml
@@ -134,16 +134,32 @@ spec:
             - targetPortNumber
             type: object
           status:
-            description: InferencePoolStatus defines the observed state of InferencePool
+            default:
+              parent:
+              - conditions:
+                - lastTransitionTime: "1970-01-01T00:00:00Z"
+                  message: Waiting for controller
+                  reason: Pending
+                  status: Unknown
+                  type: Accepted
+                parentRef:
+                  kind: Status
+                  name: default
+            description: Status defines the observed state of InferencePool.
             properties:
               parent:
                 description: |-
                   Parents is a list of parent resources (usually Gateways) that are
-                  associated with the route, and the status of the InferencePool with respect to
+                  associated with the InferencePool, and the status of the InferencePool with respect to
                   each parent.
 
-                  A maximum of 32 Gateways will be represented in this list. An empty list
-                  means the route has not been attached to any Gateway.
+                  A maximum of 32 Gateways will be represented in this list. When the list contains
+                  `kind: Status, name: default`, it indicates that the InferencePool is not
+                  associated with any Gateway and a controller must perform the following:
+
+                   - Remove the parent when setting the "Accepted" condition.
+                   - Add the parent when the controller will no longer manage the InferencePool
+                     and no other parents exist.
                 items:
                   description: PoolStatus defines the observed state of InferencePool
                     from a Gateway.
diff --git a/config/manifests/gateway/istio/destination-rule.yaml b/config/manifests/gateway/istio/destination-rule.yaml
index f9cd0c3..12ca982 100644
--- a/config/manifests/gateway/istio/destination-rule.yaml
+++ b/config/manifests/gateway/istio/destination-rule.yaml
@@ -3,7 +3,7 @@ kind: DestinationRule
 metadata:
   name: epp-insecure-tls
 spec:
-  host: vllm-llama2-7b-epp
+  host: vllm-llama3-8b-instruct-epp
   trafficPolicy:
       tls:
         mode: SIMPLE
diff --git a/config/manifests/inferencepool-resources.yaml b/config/manifests/inferencepool-resources.yaml
index 37099c4..34aacfa 100644
--- a/config/manifests/inferencepool-resources.yaml
+++ b/config/manifests/inferencepool-resources.yaml
@@ -117,5 +117,6 @@ subjects:
   name: default
   namespace: default
 roleRef:
+  apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: pod-read
diff --git a/config/manifests/vllm/cpu-deployment.yaml b/config/manifests/vllm/cpu-deployment.yaml
index 827f215..485d44a 100644
--- a/config/manifests/vllm/cpu-deployment.yaml
+++ b/config/manifests/vllm/cpu-deployment.yaml
@@ -14,7 +14,7 @@ spec:
     spec:
       containers:
         - name: lora
-          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.8.0" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
+          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.9.1" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
           imagePullPolicy: Always
           command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
           args:
diff --git a/config/manifests/vllm/sim-deployment.yaml b/config/manifests/vllm/sim-deployment.yaml
index d9706ea..1862efe 100644
--- a/config/manifests/vllm/sim-deployment.yaml
+++ b/config/manifests/vllm/sim-deployment.yaml
@@ -14,7 +14,7 @@ spec:
     spec:
       containers:
       - name: vllm-sim
-        image: ghcr.io/llm-d/llm-d-inference-sim:v0.1.1
+        image: ghcr.io/llm-d/llm-d-inference-sim:latest
         imagePullPolicy: Always
         args:
         - --model
diff --git a/conformance/conformance.go b/conformance/conformance.go
index 66b8c09..406124b 100644
--- a/conformance/conformance.go
+++ b/conformance/conformance.go
@@ -45,7 +45,6 @@ import (
 	// Import necessary types and utilities from the core Gateway API conformance suite.
 	gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"            // Import core Gateway API types
 	confapis "sigs.k8s.io/gateway-api/conformance/apis/v1" // Report struct definition
-	confconfig "sigs.k8s.io/gateway-api/conformance/utils/config"
 	confflags "sigs.k8s.io/gateway-api/conformance/utils/flags"
 	apikubernetes "sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
 	confsuite "sigs.k8s.io/gateway-api/conformance/utils/suite"
@@ -68,8 +67,8 @@ import (
 
 // Constants for the shared Gateway
 const (
-	SharedGatewayName      = "conformance-gateway"       // Name of the Gateway in manifests.yaml
-	SharedGatewayNamespace = "gateway-conformance-infra" // Namespace of the Gateway
+	SharedGatewayName      = "conformance-primary-gateway" // Name of the Gateway in manifests.yaml
+	SharedGatewayNamespace = "gateway-conformance-infra"   // Namespace of the Gateway
 )
 
 // GatewayLayerProfileName defines the name for the conformance profile that tests
@@ -88,6 +87,7 @@ const SupportInferencePool features.FeatureName = "SupportInferencePool"
 // of the "Gateway" profile for the Inference Extension MUST support.
 var InferenceCoreFeatures = sets.New(
 	features.SupportGateway, // This is needed to ensure manifest gets applied during setup.
+	features.SupportHTTPRoute,
 	SupportInferencePool,
 )
 
@@ -134,6 +134,9 @@ func DefaultOptions(t *testing.T) confsuite.ConformanceOptions {
 
 	exemptFeatures := confsuite.ParseSupportedFeatures(*confflags.ExemptFeatures)
 	skipTests := confsuite.ParseSkipTests(*confflags.SkipTests)
+	namespaceLabels := confsuite.ParseKeyValuePairs(*confflags.NamespaceLabels)
+	namespaceAnnotations := confsuite.ParseKeyValuePairs(*confflags.NamespaceAnnotations)
+
 	// Initially, run the GatewayLayerProfile. This will expand as other profiles
 	// (EPP, ModelServer) are added and can be selected via flags in future iterations.
 	conformanceProfiles := sets.New(GatewayLayerProfileName)
@@ -163,7 +166,7 @@ func DefaultOptions(t *testing.T) confsuite.ConformanceOptions {
 		Debug:                *confflags.ShowDebug,
 		CleanupBaseResources: *confflags.CleanupBaseResources,
 		SupportedFeatures:    sets.New[features.FeatureName](),
-		TimeoutConfig:        confconfig.DefaultTimeoutConfig(),
+		TimeoutConfig:        inferenceconfig.DefaultInferenceExtensionTimeoutConfig().TimeoutConfig,
 		SkipTests:            skipTests,
 		ExemptFeatures:       exemptFeatures,
 		RunTest:              *confflags.RunTest,
@@ -173,6 +176,9 @@ func DefaultOptions(t *testing.T) confsuite.ConformanceOptions {
 		ManifestFS:           []fs.FS{&Manifests},
 		ReportOutputPath:     *confflags.ReportOutput,
 		SkipProvisionalTests: *confflags.SkipProvisionalTests,
+		AllowCRDsMismatch:    *confflags.AllowCRDsMismatch,
+		NamespaceLabels:      namespaceLabels,
+		NamespaceAnnotations: namespaceAnnotations,
 		// TODO: Add the inference extension specific fields to ConformanceOptions struct if needed,
 		// or handle them during report generation.
 		// GatewayAPIInferenceExtensionChannel: inferenceExtensionChannel,
diff --git a/conformance/reports/README.md b/conformance/reports/README.md
index 81652b1..ba0278d 100644
--- a/conformance/reports/README.md
+++ b/conformance/reports/README.md
@@ -6,6 +6,7 @@ This directory stores conformance reports submitted by various implementations o
 
 This folder stores conformance reports organized first by the version of the Gateway API Inference Extension specification they were tested against, and then by the specific conformance profile (e.g., Gateway, EPP, Model Server):
 
+```
 |-- conformance/reports
 |   |-- v0.3.0  # Example extension version
 |   |   |-- gateway  # Conformance profile/category
@@ -22,6 +23,7 @@ This folder stores conformance reports organized first by the version of the Gat
 |   |   |   |-- ...
 |   |-- v0.4.0 # Future extension version
 |   |   |-- ...
+```
 
 ## Implementation Submissions
 
diff --git a/conformance/resources/manifests/manifests.yaml b/conformance/resources/manifests/manifests.yaml
index e25c294..5b1f2d4 100644
--- a/conformance/resources/manifests/manifests.yaml
+++ b/conformance/resources/manifests/manifests.yaml
@@ -4,7 +4,6 @@
 # individual test files or other resource directories (e.g., sample_backends).
 
 ---
-# Namespace for core infrastructure like Gateways.
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -12,8 +11,6 @@ metadata:
   labels:
     gateway-conformance: infra
 ---
-# Namespace for application backends (potentially simulating model servers
-# or where InferencePools might reside in some tests).
 apiVersion: v1
 kind: Namespace
 metadata:
@@ -21,40 +18,26 @@ metadata:
   labels:
     gateway-conformance: backend
 ---
-# Namespace for simple web server backends. This is expected by
-# the upstream conformance suite's Setup method.
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: gateway-conformance-web-backend
-  labels:
-    gateway-conformance: web-backend
----
 # A basic Gateway resource that allows HTTPRoutes from the same namespace.
 # Tests can use this as a parent reference for routes that target InferencePools.
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
 metadata:
-  name: conformance-gateway
+  name: conformance-primary-gateway
   namespace: gateway-conformance-infra
 spec:
-  # The conformance suite runner will replace this placeholder
-  # with the actual GatewayClass name provided via flags.
   gatewayClassName: "{GATEWAY_CLASS_NAME}"
   listeners:
-  - name: http # Standard listener name
+  - name: http
     port: 80
     protocol: HTTP
     allowedRoutes:
       namespaces:
         from: All
       kinds:
-      # Allows HTTPRoutes to attach, which can then reference InferencePools.
       - group: gateway.networking.k8s.io
         kind: HTTPRoute
 ---
-# --- Conformance Secondary Gateway Definition ---
-# A second generic Gateway resource for tests requiring multiple Gateways.
 apiVersion: gateway.networking.k8s.io/v1
 kind: Gateway
 metadata:
@@ -66,7 +49,305 @@ spec:
   - name: http
     port: 80
     protocol: HTTP
-    hostname: "secondary.example.com" # Distinct hostname to differentiate from conformance-gateway
+    hostname: "secondary.example.com"
     allowedRoutes:
       namespaces:
         from: All
+
+### The following defines the essential resources for the gateway conformance test.
+### All resources are created in the 'gateway-conformance-app-backend' namespace.
+---
+# Deploys a mock backend service to act as a model server.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: primary-inference-model-server-deployment
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: primary-inference-model-server
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: primary-inference-model-server
+  template:
+    metadata:
+      labels:
+        app: primary-inference-model-server
+    spec:
+      containers:
+      - name: echoserver
+        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
+        ports:
+        - containerPort: 3000
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 3000
+          initialDelaySeconds: 3
+          periodSeconds: 5
+          failureThreshold: 2
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+---
+# Deploys a secondary mock backend service to act as a model server.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: secondary-inference-model-server-deployment
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: secondary-inference-model-server
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: secondary-inference-model-server
+  template:
+    metadata:
+      labels:
+        app: secondary-inference-model-server
+    spec:
+      containers:
+      - name: echoserver
+        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
+        ports:
+        - containerPort: 3000
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 3000
+          initialDelaySeconds: 3
+          periodSeconds: 5
+          failureThreshold: 2
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+        - name: POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+---
+# --- Primary InferencePool Definition ---
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: primary-inference-pool
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: primary-inference-model-server
+  targetPortNumber: 3000
+  extensionRef:
+    name: primary-endpoint-picker-svc
+---
+# --- Primary Conformance EPP service Definition ---
+apiVersion: v1
+kind: Service
+metadata:
+  name: primary-endpoint-picker-svc
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: primary-app-backend-epp
+  ports:
+    - protocol: TCP
+      port: 9002
+      targetPort: 9002
+      appProtocol: http2
+  type: ClusterIP
+---
+# --- Primary Conformance EPP Deployment ---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: primary-app-endpoint-picker
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: primary-app-backend-epp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: primary-app-backend-epp
+  template:
+    metadata:
+      labels:
+        app: primary-app-backend-epp
+    spec:
+      # Conservatively, this timeout should mirror the longest grace period of the pods within the pool
+      terminationGracePeriodSeconds: 130
+      containers:
+      - name: epp
+        image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:main
+        imagePullPolicy: Always
+        args:
+        - -poolName
+        - "primary-inference-pool"
+        - -poolNamespace
+        - "gateway-conformance-app-backend"
+        - -v
+        - "4"
+        - --zap-encoder
+        - "json"
+        - -grpcPort
+        - "9002"
+        - -grpcHealthPort
+        - "9003"
+        env:
+        - name: USE_STREAMING
+          value: "true"
+        - name: ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING # Used for conformance test.
+          value: "true"
+        ports:
+        - containerPort: 9002
+        - containerPort: 9003
+        - name: metrics
+          containerPort: 9090
+        livenessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10
+        readinessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10
+---
+# --- Secondary InferencePool Definition ---
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: secondary-inference-pool
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: secondary-inference-model-server
+  targetPortNumber: 3000
+  extensionRef:
+    name: secondary-endpoint-picker-svc
+    failureMode: FailOpen
+---
+# --- Secondary Conformance EPP service Definition ---
+apiVersion: v1
+kind: Service
+metadata:
+  name: secondary-endpoint-picker-svc
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: secondary-app-backend-epp
+  ports:
+    - protocol: TCP
+      port: 9002
+      targetPort: 9002
+      appProtocol: http2
+  type: ClusterIP
+---
+# --- Secondary Conformance EPP Deployment ---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: secondary-app-endpoint-picker
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: secondary-app-backend-epp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: secondary-app-backend-epp
+  template:
+    metadata:
+      labels:
+        app: secondary-app-backend-epp
+    spec:
+      # Conservatively, this timeout should mirror the longest grace period of the pods within the pool
+      terminationGracePeriodSeconds: 130
+      containers:
+      - name: epp
+        image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:main
+        imagePullPolicy: Always
+        args:
+        - -poolName
+        - "secondary-inference-pool"
+        - -poolNamespace
+        - "gateway-conformance-app-backend"
+        - -v
+        - "4"
+        - --zap-encoder
+        - "json"
+        - -grpcPort
+        - "9002"
+        - -grpcHealthPort
+        - "9003"
+        env:
+        - name: USE_STREAMING
+          value: "true"
+        - name: ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING # Used for conformance test.
+          value: "true"
+        ports:
+        - containerPort: 9002
+        - containerPort: 9003
+        - name: metrics
+          containerPort: 9090
+        livenessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10
+        readinessProbe:
+          grpc:
+            port: 9003
+            service: inference-extension
+          initialDelaySeconds: 5
+          periodSeconds: 10
+---
+# --- Required Role and RoleBinding for Conformance Test for EPP ---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: inference-model-reader
+  namespace: gateway-conformance-app-backend
+rules:
+- apiGroups: ["inference.networking.x-k8s.io"]
+  resources: ["inferencemodels", "inferencepools"]
+  verbs: ["get", "list", "watch"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: epp-to-inference-model-reader
+  namespace: gateway-conformance-app-backend
+subjects:
+- kind: ServiceAccount
+  name: default
+  namespace: gateway-conformance-app-backend
+roleRef:
+  kind: Role
+  name: inference-model-reader
+  apiGroup: rbac.authorization.k8s.io
diff --git a/conformance/utils/config/timing.go b/conformance/utils/config/timing.go
index f5d4eeb..861eb0f 100644
--- a/conformance/utils/config/timing.go
+++ b/conformance/utils/config/timing.go
@@ -20,6 +20,7 @@ import (
 	"time"
 
 	// Import the upstream Gateway API timeout config
+
 	gatewayconfig "sigs.k8s.io/gateway-api/conformance/utils/config"
 )
 
@@ -29,8 +30,8 @@ type InferenceExtensionTimeoutConfig struct {
 	// All fields from gatewayconfig.TimeoutConfig will be available directly.
 	gatewayconfig.TimeoutConfig
 
-	// InferencePoolMustHaveConditionTimeout represents the maximum time to wait for an InferencePool to have a specific condition.
-	InferencePoolMustHaveConditionTimeout time.Duration
+	// GeneralMustHaveConditionTimeout represents the maximum time to wait for an InferencePool, HttpRoute or other assets to have a specific condition.
+	GeneralMustHaveConditionTimeout time.Duration
 
 	// InferencePoolMustHaveConditionInterval represents the polling interval for checking an InferencePool's condition.
 	InferencePoolMustHaveConditionInterval time.Duration
@@ -38,16 +39,19 @@ type InferenceExtensionTimeoutConfig struct {
 	// GatewayObjectPollInterval is the polling interval used when waiting for a Gateway object to appear.
 	GatewayObjectPollInterval time.Duration
 
-	// HTTPRouteDeletionReconciliationTimeout is the time to wait for controllers to reconcile
-	// state after an HTTPRoute is deleted, before checking dependent resources or traffic.
+	// HTTPRouteConditionTimeout represents the maximum time to wait for an HTTPRoute to have a specific condition.
 	HTTPRouteDeletionReconciliationTimeout time.Duration
 }
 
 // DefaultInferenceExtensionTimeoutConfig returns a new InferenceExtensionTimeoutConfig with default values.
 func DefaultInferenceExtensionTimeoutConfig() InferenceExtensionTimeoutConfig {
+	config := gatewayconfig.DefaultTimeoutConfig()
+	config.HTTPRouteMustHaveCondition = 300 * time.Second
+	config.MaxTimeToConsistency = 200 * time.Second
+	config.DefaultTestTimeout = 600 * time.Second
 	return InferenceExtensionTimeoutConfig{
-		TimeoutConfig:                          gatewayconfig.DefaultTimeoutConfig(), // Initialize embedded struct
-		InferencePoolMustHaveConditionTimeout:  300 * time.Second,
+		TimeoutConfig:                          config, // Initialize embedded struct
+		GeneralMustHaveConditionTimeout:        300 * time.Second,
 		InferencePoolMustHaveConditionInterval: 10 * time.Second,
 		GatewayObjectPollInterval:              5 * time.Second,
 		HTTPRouteDeletionReconciliationTimeout: 5 * time.Second,
diff --git a/conformance/utils/kubernetes/helpers.go b/conformance/utils/kubernetes/helpers.go
index 2e866ca..2fcaa0c 100644
--- a/conformance/utils/kubernetes/helpers.go
+++ b/conformance/utils/kubernetes/helpers.go
@@ -23,20 +23,20 @@ import (
 	"fmt"
 	"reflect"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/require"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
-	// Import the Inference Extension API types
-	inferenceapi "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2" // Adjust if your API version is different
-
-	// Import local config for Inference Extension
+	inferenceapi "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	"sigs.k8s.io/gateway-api-inference-extension/conformance/utils/config"
-	// Import necessary utilities from the core Gateway API conformance suite
 	gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
 	gatewayapiconfig "sigs.k8s.io/gateway-api/conformance/utils/config"
 	gatewayk8sutils "sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
@@ -81,7 +81,7 @@ func InferencePoolMustHaveCondition(t *testing.T, c client.Client, poolNN types.
 	waitErr := wait.PollUntilContextTimeout(
 		context.Background(),
 		timeoutConfig.InferencePoolMustHaveConditionInterval,
-		timeoutConfig.InferencePoolMustHaveConditionTimeout,
+		timeoutConfig.GeneralMustHaveConditionTimeout,
 		true, func(ctx context.Context) (bool, error) {
 			pool := &inferenceapi.InferencePool{} // This is the type instance used for Get
 			err := c.Get(ctx, poolNN, pool)
@@ -172,7 +172,7 @@ func InferencePoolMustHaveNoParents(t *testing.T, c client.Client, poolNN types.
 		ctx,
 
 		timeoutConfig.InferencePoolMustHaveConditionInterval,
-		timeoutConfig.InferencePoolMustHaveConditionTimeout,
+		timeoutConfig.GeneralMustHaveConditionTimeout,
 		true,
 		func(pollCtx context.Context) (bool, error) {
 			pool := &inferenceapi.InferencePool{}
@@ -257,6 +257,43 @@ func InferencePoolMustBeAcceptedByParent(t *testing.T, c client.Client, poolNN t
 	t.Logf("InferencePool %s is Accepted by a parent Gateway (Reason: %s)", poolNN.String(), gatewayv1.GatewayReasonAccepted)
 }
 
+// InferencePoolMustBeRouteAccepted waits for the specified InferencePool resource
+// to exist and report an Accepted condition with Type=RouteConditionAccepted,
+// Status=True, and Reason=RouteReasonAccepted within one of its parent statuses.
+func InferencePoolMustBeRouteAccepted(t *testing.T, c client.Client, poolNN types.NamespacedName) {
+	t.Helper()
+
+	expectedPoolCondition := metav1.Condition{
+		Type:   string(gatewayv1.RouteConditionAccepted),
+		Status: metav1.ConditionTrue,
+		Reason: string(gatewayv1.RouteReasonAccepted),
+	}
+
+	// Call the existing generic helper with the predefined condition
+	InferencePoolMustHaveCondition(t, c, poolNN, expectedPoolCondition)
+	t.Logf("InferencePool %s successfully verified with RouteAccepted condition (Type: %s, Status: %s, Reason: %s).",
+		poolNN.String(), expectedPoolCondition.Type, expectedPoolCondition.Status, expectedPoolCondition.Reason)
+}
+
+// HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted waits for the specified HTTPRoute
+// to be Accepted and have its references resolved by the specified Gateway,
+// AND for the specified InferencePool to be "RouteAccepted" using the specific
+// RouteConditionAccepted criteria.
+func HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted(
+	t *testing.T,
+	c client.Client,
+	routeNN types.NamespacedName,
+	gatewayNN types.NamespacedName,
+	poolNN types.NamespacedName) {
+	t.Helper()
+	var timeoutConfig config.InferenceExtensionTimeoutConfig = config.DefaultInferenceExtensionTimeoutConfig()
+
+	HTTPRouteMustBeAcceptedAndResolved(t, c, timeoutConfig.TimeoutConfig, routeNN, gatewayNN)
+	InferencePoolMustBeRouteAccepted(t, c, poolNN)
+	t.Logf("Successfully verified: HTTPRoute %s (Gateway %s) is Accepted & Resolved, and InferencePool %s is RouteAccepted.",
+		routeNN.String(), gatewayNN.String(), poolNN.String())
+}
+
 // GetGatewayEndpoint waits for the specified Gateway to have at least one address
 // and returns the address in "host:port" format.
 // It leverages the upstream Gateway API's WaitForGatewayAddress.
@@ -271,3 +308,104 @@ func GetGatewayEndpoint(t *testing.T, k8sClient client.Client, timeoutConfig gat
 	t.Logf("Gateway %s/%s has address: %s", gatewayNN.Namespace, gatewayNN.Name, gwAddr)
 	return gwAddr
 }
+
+// GetPodsWithLabel retrieves a list of Pods.
+// It finds pods matching the given labels in a specific namespace.
+func GetPodsWithLabel(t *testing.T, c client.Client, namespace string, labels map[string]string) ([]corev1.Pod, error) {
+	t.Helper()
+
+	podList := &corev1.PodList{}
+	listOptions := []client.ListOption{
+		client.InNamespace(namespace),
+		client.MatchingLabels(labels),
+	}
+
+	t.Logf("Searching for Pods with labels %v in namespace %s", labels, namespace)
+	if err := c.List(context.Background(), podList, listOptions...); err != nil {
+		return nil, fmt.Errorf("failed to list pods with labels '%v' in namespace '%s': %w", labels, namespace, err)
+	}
+
+	if len(podList.Items) == 0 {
+		return nil, fmt.Errorf("no pods found with labels '%v' in namespace '%s'", labels, namespace)
+	}
+	return podList.Items, nil
+}
+
+// GetPod waits for a Pod matching the specified labels to exist in the given
+// namespace and have an IP address assigned. This function returns the first
+// matching Pod found if there are multiple matches. It fails the on timeout or error.
+// TODO(#1003) combline with GetPodsWithLabel that is being introduced in PR #961
+func GetPod(t *testing.T, c client.Client, namespace string, selector labels.Selector, timeout time.Duration) *corev1.Pod {
+	t.Helper()
+
+	var pods corev1.PodList
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+
+	waitErr := wait.PollUntilContextTimeout(ctx, 1*time.Second, timeout, true, func(ctx context.Context) (bool, error) {
+		if err := c.List(ctx, &pods, &client.ListOptions{
+			LabelSelector: selector,
+			Namespace:     namespace,
+		}); err != nil {
+			t.Logf("Error listing pods with selector %s: %v. Retrying.", selector.String(), err)
+			return false, nil
+		}
+
+		if len(pods.Items) > 0 {
+			pod := pods.Items[0]
+			if pod.Status.PodIP != "" && pod.Status.Phase == corev1.PodRunning {
+				return true, nil
+			}
+			t.Logf("Pod %s found, but not yet running or has no IP. Current phase: %s, IP: '%s'. Retrying.", pod.Name, pod.Status.Phase, pod.Status.PodIP)
+		} else {
+			t.Logf("No pods found with selector %s yet. Retrying.", selector.String())
+		}
+		return false, nil
+	})
+	require.NoErrorf(t, waitErr, "timed out waiting for Pod with selector %s in namespace %s to be ready", selector.String(), namespace)
+	require.NotEmpty(t, pods.Items, "expected at least one pod for selector %s in namespace %s, but found none", selector.String(), namespace)
+
+	pod := &pods.Items[0]
+	t.Logf("Successfully found ready Pod %s with IP %s for selector %s", pod.Name, pod.Status.PodIP, selector.String())
+	return pod
+}
+
+// DeleteDeployment deletes the specified Deployment and waits until it is no longer
+// present in the cluster.
+func DeleteDeployment(t *testing.T, c client.Client, timeoutConfig gatewayapiconfig.TimeoutConfig, deploymentRef types.NamespacedName) error {
+	t.Helper()
+
+	deploymentToDelete := &appsv1.Deployment{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      deploymentRef.Name,
+			Namespace: deploymentRef.Namespace,
+		},
+	}
+
+	t.Logf("Deleting Deployment %s/%s...", deploymentRef.Namespace, deploymentRef.Name)
+	if err := c.Delete(context.Background(), deploymentToDelete); err != nil {
+		// If the resource is already gone, we don't consider it an error.
+		if !apierrors.IsNotFound(err) {
+			return fmt.Errorf("failed to delete Deployment %s/%s: %w", deploymentRef.Namespace, deploymentRef.Name, err)
+		}
+	}
+
+	// Wait for the Deployment to be fully removed.
+	waitErr := wait.PollUntilContextTimeout(context.Background(), 1*time.Second, timeoutConfig.DeleteTimeout, true, func(ctx context.Context) (bool, error) {
+		var dep appsv1.Deployment
+		err := c.Get(ctx, deploymentRef, &dep)
+		if apierrors.IsNotFound(err) {
+			return true, nil
+		}
+		if err != nil {
+			return false, fmt.Errorf("error waiting for Deployment %s/%s to be deleted: %w", deploymentRef.Namespace, deploymentRef.Name, err)
+		}
+		return false, nil
+	})
+
+	if waitErr != nil {
+		return fmt.Errorf("timed out waiting for Deployment %s/%s to be deleted: %w", deploymentRef.Namespace, deploymentRef.Name, waitErr)
+	}
+	t.Logf("Successfully deleted Deployment %s/%s", deploymentRef.Namespace, deploymentRef.Name)
+	return nil
+}
diff --git a/conformance/utils/traffic/traffic.go b/conformance/utils/traffic/traffic.go
index e65f45f..e277223 100644
--- a/conformance/utils/traffic/traffic.go
+++ b/conformance/utils/traffic/traffic.go
@@ -17,14 +17,47 @@ limitations under the License.
 package traffic
 
 import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
 	"net/http"
+	"net/http/httputil"
+	"regexp"
+	"strings"
 	"testing"
+	"time"
 
+	corev1 "k8s.io/api/core/v1"
 	gwconfig "sigs.k8s.io/gateway-api/conformance/utils/config"
 	gwhttp "sigs.k8s.io/gateway-api/conformance/utils/http"
 	"sigs.k8s.io/gateway-api/conformance/utils/roundtripper"
+	"sigs.k8s.io/gateway-api/conformance/utils/tlog"
 )
 
+// Request defines the parameters for a single HTTP test request and its expected outcome.
+type Request struct {
+	// Host is the hostname to use in the HTTP request.
+	Host string
+	// Path is the path to request.
+	Path string
+	// Method is the HTTP method to use. Defaults to "GET" if empty.
+	Method string
+	// Headers are the HTTP headers to include in the request.
+	Headers map[string]string
+	// Body is the request body.
+	Body string
+
+	// ExpectedStatusCode is the HTTP status code expected in the response.
+	ExpectedStatusCode int
+	// Backend is the name of the backend service expected to handle the request.
+	// This is not checked for non-200 responses.
+	Backend string
+	// Namespace is the namespace of the backend service.
+	Namespace string
+}
+
+// Deprecated: this will be replaced by letting caller to construct the Request type.
 // BuildExpectedHTTPResponse constructs a gwhttp.ExpectedResponse for common test scenarios.
 // For 200 OK responses, it sets up ExpectedRequest to check Host and Path.
 // For other status codes (like 404), ExpectedRequest is nil as detailed backend checks are usually skipped by CompareRequest.
@@ -60,6 +93,8 @@ func BuildExpectedHTTPResponse(
 	return resp
 }
 
+// Deprecated: please use MakeRequestWithRequestParamAndExpectSuccess instead.
+// https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/983
 // MakeRequestAndExpectSuccess is a helper function that builds an expected success (200 OK) response
 // and then calls MakeRequestAndExpectEventuallyConsistentResponse.
 func MakeRequestAndExpectSuccess(
@@ -83,6 +118,8 @@ func MakeRequestAndExpectSuccess(
 	gwhttp.MakeRequestAndExpectEventuallyConsistentResponse(t, r, timeoutConfig, gatewayAddress, expectedResponse)
 }
 
+// Deprecated: please use MakeRequestAndExpectEventuallyConsistentResponse instead and specify the ExpectedStatusCode in Request.
+// https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/983
 // MakeRequestAndExpectNotFound is a helper function that builds an expected not found (404) response
 // and then calls MakeRequestAndExpectEventuallyConsistentResponse.
 func MakeRequestAndExpectNotFound(
@@ -103,3 +140,262 @@ func MakeRequestAndExpectNotFound(
 	)
 	gwhttp.MakeRequestAndExpectEventuallyConsistentResponse(t, r, timeoutConfig, gatewayAddress, expectedResponse)
 }
+
+// MakeRequestAndExpectEventuallyConsistentResponse makes a request using the parameters
+// from the Request struct and waits for the response to consistently match the expectations.
+func MakeRequestAndExpectEventuallyConsistentResponse(
+	t *testing.T,
+	r roundtripper.RoundTripper,
+	timeoutConfig gwconfig.TimeoutConfig,
+	gatewayAddress string,
+	req Request,
+) {
+	t.Helper()
+
+	expectedResponse := makeExpectedResponse(t, req)
+	waitForConvergeToExpected(t, r, timeoutConfig, gatewayAddress, req.Body, expectedResponse)
+}
+
+// MakeRequestWithRequestParamAndExpectSuccess is a convenience wrapper for requests that are
+// expected to succeed with a 200 OK status.
+func MakeRequestWithRequestParamAndExpectSuccess(
+	t *testing.T,
+	r roundtripper.RoundTripper,
+	timeoutConfig gwconfig.TimeoutConfig,
+	gatewayAddress string,
+	req Request,
+) {
+	t.Helper()
+	req.ExpectedStatusCode = http.StatusOK
+	MakeRequestAndExpectEventuallyConsistentResponse(t, r, timeoutConfig, gatewayAddress, req)
+}
+
+// MakeRequestAndExpectResponseFromPod sends a request to the specified path by IP address and
+// uses a special "test-epp-endpoint-selection" header to target a specific backend Pod.
+// It then verifies that the response was served by that Pod.
+func MakeRequestAndExpectResponseFromPod(t *testing.T, r roundtripper.RoundTripper, timeoutConfig gwconfig.TimeoutConfig, gwAddr, path string, targetPod *corev1.Pod) {
+	t.Helper()
+
+	const (
+		eppSelectionHeader = "test-epp-endpoint-selection"
+		backendPort        = 3000
+	)
+
+	expectedResponse := gwhttp.ExpectedResponse{
+		Request: gwhttp.Request{
+			Path: path,
+			Headers: map[string]string{
+				eppSelectionHeader: fmt.Sprintf("%s:%d", targetPod.Status.PodIP, backendPort),
+			},
+		},
+		Backend:   targetPod.Name,
+		Namespace: targetPod.Namespace,
+	}
+
+	gwhttp.MakeRequestAndExpectEventuallyConsistentResponse(t, r, timeoutConfig, gwAddr, expectedResponse)
+}
+
+func makeExpectedResponse(t *testing.T, req Request) gwhttp.ExpectedResponse {
+	t.Helper()
+
+	method := http.MethodGet
+	if req.Method != "" {
+		method = req.Method
+	}
+
+	expectedResponse := gwhttp.ExpectedResponse{
+		Request: gwhttp.Request{
+			Host:    req.Host,
+			Path:    req.Path,
+			Method:  method,
+			Headers: req.Headers,
+		},
+		Response: gwhttp.Response{
+			StatusCode: req.ExpectedStatusCode,
+		},
+		Backend:   req.Backend,
+		Namespace: req.Namespace,
+	}
+
+	// For successful responses (200 OK), we also verify that the backend
+	// received the request with the correct details (Host, Path, etc.).
+	// For other statuses (e.g., 404), this check is skipped.
+	if req.ExpectedStatusCode == http.StatusOK {
+		expectedResponse.ExpectedRequest = &gwhttp.ExpectedRequest{
+			Request: gwhttp.Request{
+				Host:    req.Host,
+				Path:    req.Path,
+				Headers: req.Headers,
+				Method:  method,
+			},
+		}
+	}
+	return expectedResponse
+}
+
+// TODO: https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1031
+// replace the following method when sigs.k8s.io/gateway-api/conformance/utils/roundtripper is able to send request with body.
+func waitForConvergeToExpected(
+	t *testing.T,
+	r roundtripper.RoundTripper,
+	timeoutConfig gwconfig.TimeoutConfig,
+	gatewayAddress string,
+	requestBody string,
+	expectedResponse gwhttp.ExpectedResponse,
+) {
+	gwhttp.AwaitConvergence(t, timeoutConfig.RequiredConsecutiveSuccesses, timeoutConfig.MaxTimeToConsistency, func(elapsed time.Duration) bool {
+		req := gwhttp.MakeRequest(t, &expectedResponse, gatewayAddress, "HTTP", "http")
+		request := &RequestWithBody{Request: req}
+		if requestBody != "" {
+			request = &RequestWithBody{Request: req, Body: strings.NewReader(requestBody)}
+		}
+		cReq, cRes, err := MakeCallRoundTripper(t, r, request)
+		if err != nil {
+			tlog.Logf(t, "Request failed, not ready yet: %v (after %v)", err.Error(), elapsed)
+			return false
+		}
+
+		if err := gwhttp.CompareRequest(t, &request.Request, cReq, cRes, expectedResponse); err != nil {
+			tlog.Logf(t, "Response expectation failed for request: %+v  not ready yet: %v (after %v)", request.Request, err, elapsed)
+			return false
+		}
+
+		return true
+	})
+	tlog.Logf(t, "Request passed")
+}
+
+// TODO: https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1031
+// remove this when sigs.k8s.io/gateway-api/conformance/utils/roundtripper is able to send request with body.
+// RequestWithBody extends roundtripper.Request to include a request body.
+type RequestWithBody struct {
+	roundtripper.Request
+	Body io.Reader
+}
+
+// TODO: https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1031
+// remove this when sigs.k8s.io/gateway-api/conformance/utils/roundtripper is able to send request with body.
+// MakeCallRoundTripper executes an HTTP request using the provided RoundTripper and captures the request and response.
+func MakeCallRoundTripper(t *testing.T, r roundtripper.RoundTripper, request *RequestWithBody) (*roundtripper.CapturedRequest, *roundtripper.CapturedResponse, error) {
+	client := &http.Client{}
+
+	defaultRoundTripper, ok := r.(*roundtripper.DefaultRoundTripper)
+	if !ok {
+		t.Fatalf("Unsupported RoundTripper type: %T", r)
+	}
+	rt := defaultRoundTripper
+	if request.UnfollowRedirect {
+		client.CheckRedirect = func(_ *http.Request, _ []*http.Request) error {
+			return http.ErrUseLastResponse
+		}
+	}
+
+	client.Transport = &http.Transport{
+		DialContext: rt.CustomDialContext,
+		// We disable keep-alives so that we don't leak established TCP connections.
+		// Leaking TCP connections is bad because we could eventually hit the
+		// threshold of maximum number of open TCP connections to a specific
+		// destination. Keep-alives are not presently utilized so disabling this has
+		// no adverse affect.
+		//
+		// Ref. https://github.com/kubernetes-sigs/gateway-api/issues/2357
+		DisableKeepAlives: true,
+	}
+
+	method := "GET"
+	if request.Method != "" {
+		method = request.Method
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), rt.TimeoutConfig.RequestTimeout)
+	defer cancel()
+	req, err := http.NewRequestWithContext(ctx, method, request.URL.String(), request.Body)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if request.Host != "" {
+		req.Host = request.Host
+	}
+
+	if request.Headers != nil {
+		for name, value := range request.Headers {
+			req.Header.Set(name, value[0])
+		}
+	}
+
+	if rt.Debug {
+		var dump []byte
+		dump, err = httputil.DumpRequestOut(req, true)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		tlog.Logf(request.T, "Sending Request:\n%s\n\n", formatDump(dump, "< "))
+	}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, nil, err
+	}
+	defer resp.Body.Close()
+
+	if rt.Debug {
+		var dump []byte
+		dump, err = httputil.DumpResponse(resp, true)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		tlog.Logf(request.T, "Received Response:\n%s\n\n", formatDump(dump, "< "))
+	}
+
+	cReq := &roundtripper.CapturedRequest{}
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	// we cannot assume the response is JSON
+	if resp.Header.Get("Content-type") == "application/json" {
+		err = json.Unmarshal(body, cReq)
+		if err != nil {
+			return nil, nil, fmt.Errorf("unexpected error reading response: %w", err)
+		}
+	} else {
+		cReq.Method = method // assume it made the right request if the service being called isn't echoing
+	}
+
+	cRes := &roundtripper.CapturedResponse{
+		StatusCode:    resp.StatusCode,
+		ContentLength: resp.ContentLength,
+		Protocol:      resp.Proto,
+		Headers:       resp.Header,
+	}
+
+	if resp.TLS != nil {
+		cRes.PeerCertificates = resp.TLS.PeerCertificates
+	}
+
+	if roundtripper.IsRedirect(resp.StatusCode) {
+		redirectURL, err := resp.Location()
+		if err != nil {
+			return nil, nil, err
+		}
+		cRes.RedirectRequest = &roundtripper.RedirectRequest{
+			Scheme: redirectURL.Scheme,
+			Host:   redirectURL.Hostname(),
+			Port:   redirectURL.Port(),
+			Path:   redirectURL.Path,
+		}
+	}
+
+	return cReq, cRes, nil
+}
+
+var startLineRegex = regexp.MustCompile(`(?m)^`)
+
+func formatDump(data []byte, prefix string) string {
+	data = startLineRegex.ReplaceAllLiteral(data, []byte(prefix))
+	return string(data)
+}
diff --git a/docs/proposals/002-api-proposal/README.md b/docs/proposals/002-api-proposal/README.md
index 2eb1111..747ecfa 100644
--- a/docs/proposals/002-api-proposal/README.md
+++ b/docs/proposals/002-api-proposal/README.md
@@ -236,7 +236,7 @@ type InferenceModelSpec struct {
         // ModelNames are expected to be unique for a specific InferencePool 
         // (names can be reused for a different pool in the same cluster). 
         // The modelName with the oldest creation timestamp is retained, and the incoming
-        // InferenceModel is sets the Ready status to false with a corresponding reason. 
+        // InferenceModel's Ready status is set to false with a corresponding reason.
         // In the rare case of a race condition, one Model will be selected randomly to be considered valid, and the other rejected.
         // Names can be reserved without an underlying model configured in the pool.
         // This can be done by specifying a target model and setting the weight to zero,
diff --git a/docs/proposals/0845-scheduler-architecture-proposal/README.md b/docs/proposals/0845-scheduler-architecture-proposal/README.md
index 58f6389..4141ce6 100644
--- a/docs/proposals/0845-scheduler-architecture-proposal/README.md
+++ b/docs/proposals/0845-scheduler-architecture-proposal/README.md
@@ -9,18 +9,27 @@ The Scheduling Subsystem is a framework used to implement scheduling algorithms.
 
 ## Design Principles
 - The scheduler framework should act as an independent library, there should be no dependency on EPP packages defined outside of the scheduler
-- The *framework* should be agnostic to web protocols(such as HTTP), endpoint types (such as model servers), and K8s concepts. 
+- The *framework* should be agnostic to endpoint types (such as model servers), and K8s concepts. 
   - Opinions should be held by the plugins, not the framework
 - The entry & exit points should be defined by the framework, acting as the API surface of the system
 - Multiple scheduling 'profiles' should be able to be ran for a single request.
     - They can be conditionally dependent on previous runs, or in parallel
-- Plugin state is managed by the plugin itself
+- State management
+  - State per request: This is managed by what we are calling CycleState and its lifecycle is tied to the request.
+    Cycle state is created internally by the Scheduler per request and its pointer is passed as argument.
+  - State managed by the plugin struct itself: The lifecycle of this state is tied to the plugin, and since plugins will be instantiated once, 
+    it is a state that plugins can use across requests (like prefix-cache index).
+  - State managed by the data layer: each endpoint will be associated with state (currently metrics) that a data layer plugin can add to it. 
+    A data layer plugin could be one that scrapes v1/models from the endpoint for example.
 
 ## Definitions
 - **Scheduling Framework** - The system created to allow for a pluggable scheduling algorithm.
-- **Scheduling Profile** - A named, specific set of Filter(s), Scorer(s), & Picker used to select endpoints.
-- **Scheduler** - An extensible implementation of a scheduling algorithm. Including logic to select Scheduling Profiles, the Scheduling Profiles themselves, & logic to interpret the result.
-- **Scheduling Cycle** - A single run of a Scheduler through the Scheduling Framework.
+- **Scheduler Profile** - A named, specific set of Filter(s), Scorer(s), & Picker used to select endpoints.
+- **Scheduler Profile Run** - a one time run of the Scheduler Profile filters, scorers and picker given a request.
+- **Scheduler** - An extensible implementation of a scheduling algorithm. Including logic to select Scheduler Profiles iteratively, 
+  the Scheduler Profiles themselves, & logic to interpret the result.
+- **Scheduling Cycle** - A single run of a Scheduler through the Scheduling Framework. a scheduling cycle includes one or 
+  more Scheduler Profile runs (at least one).
 - **Plugin** - Implementation of framework-defined interface(s) to add or extend logic across the framework.
 
 ## Proposal
@@ -33,23 +42,32 @@ The Scheduling System can loosely be defined into 3 sections:
 - A *configuration API* to define the Scheduler, Profile(s), & the plugins used within those profiles
 
 A sketch of the System, with extension points is here:
-<img src="./images/scheduler_subsystem.svg" alt="Scheduling Algorithm" width="1000" />
+<img src="./images/scheduler_cycle.png" alt="Scheduling Algorithm" width="1000" />
 
 Describing the interface extension points & flow is the simplest way to convey the intent of what the framework should enable:
 
-### PreSchedule
+### ProfileHandler
 
-PreSchedule is the entry point into the scheduling cycle (called by the framework). PreSchedule, selects profiles conditionally based on: 
+ProfileHandler is a schedler plugin with two extension points - ProfilePick, and ProcessProfilesResults.
+Below is a detailed explanation about these extension points.
+Only a single ProfileHandler plugin may be defined per scheduler.
+
+### ProfilePick
+
+ProfilePick is the entry point into the scheduling cycle (called by the framework). 
+it selects profiles conditionally based on: 
 
 - Request data
-- Results
+- Results of previously executed SchedulerProfiles
 - Cycle State
 
-PreSchedule will be continuously called so long as profiles are returned; multiple profiles may be returned in a single call. Only a single PreSchedule function may be defined per scheduler.
+ProfilePick will be continuously called so long as profiles are returned; multiple profiles may be returned in a single call.
+ProfilePick extension point will be configured as part of a ProfileHandler plugin. 
+Since there is only a single ProfileHandler plugin, that means there is only a single ProfilePick function.
 
-### Profile Cycle
+### Scheduler Profile Run
 
-The profile cycle consists of 3 defined functions `Filter`, `Score`, & `Pick`
+The SchedulerProfile run consists of 3 defined phases `Filter`, `Score`, & `Pick`
 
 *Profile Constraints*
 - A profile can have any number of `Filter` plugins registered (including zero)
@@ -61,17 +79,15 @@ The profile cycle consists of 3 defined functions `Filter`, `Score`, & `Pick`
 Filter runs before any scoring, and remove endpoints that are not fit for selection. The framework will return an error to the client if the endpoints are filtered to zero.
 
 #### Score
-Score applies a score to each remaining endpoint provided. Scorers SHOULD keep their score values in a normalized range: [0-1]. Any weighting should be added at the SchedulingProfile configuration level.
+Score applies a score to each remaining endpoint provided. Scorers SHOULD keep their score values in a normalized range: [0-1]. Any weighting should be added at the SchedulerProfile configuration level.
 
 #### Pick
 Picker selects the endpoint(s) from the provided list of scored endpoints. Picker MUST return, one endpoint at minimum.
 
 
-### PostSchedule
-PostSchedule receives the output of the result(s) of the scheduling cycle(s) and makes sense of the data to be consumed by the calling system.
-
-### PostResponse
-PostResponse is a special case extension that can optionally be implemented by a plugin that needs to augment its state based on response or request data. This should only be implemented for plugins that need to update state outside of the scheduling cycle. PostResponse is ran at the time of processing a response.
+### ProcessProfilesResults
+ProcessProfilesResults receives the output of the result(s) of the scheduler profile(s) and makes sense of the data to be consumed by the calling system.
+Since there is only a single ProfileHandler plugin, that means there is only a single ProcessProfilesResults function.
 
 ## ConfigurationAPI
 TODO
\ No newline at end of file
diff --git a/docs/proposals/0845-scheduler-architecture-proposal/images/scheduler_cycle.png b/docs/proposals/0845-scheduler-architecture-proposal/images/scheduler_cycle.png
new file mode 100644
index 0000000..f819e50
Binary files /dev/null and b/docs/proposals/0845-scheduler-architecture-proposal/images/scheduler_cycle.png differ
diff --git a/docs/proposals/0845-scheduler-architecture-proposal/interfaces/interface.go b/docs/proposals/0845-scheduler-architecture-proposal/interfaces/interface.go
index 022ed36..35b787b 100644
--- a/docs/proposals/0845-scheduler-architecture-proposal/interfaces/interface.go
+++ b/docs/proposals/0845-scheduler-architecture-proposal/interfaces/interface.go
@@ -22,16 +22,8 @@ import (
 	scheduling "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// READER NOTE: Currently CycleState is assumed to have appropriate request data rather that making a new object.
-
-// Plugin is the parent type for all the scheduling framework plugins.
-type Plugin interface {
-	Name() string
-}
-
 type Endpoint struct {
 	State EndpointState
-	Score float64
 }
 
 type EndpointState struct {
@@ -39,65 +31,112 @@ type EndpointState struct {
 	storage map[string]any //nolint:unused
 }
 
-type SchedulingResult struct {
-	results map[string][]Endpoint //nolint:unused
+// Request is a structured representation of the fields we parse out of the Request body.
+type Request struct {
+	// RequestId is the Envoy generated Id for the request being processed
+	RequestId string
+	// TargetModel is the final target model after traffic split.
+	TargetModel string
+	// Prompt is the prompt that was sent in the request body.
+	Prompt string
+	// Headers is a map of the request headers.
+	Headers map[string]string
 }
 
-// Scheduler is the implementation of a... scheduler.
-// The scheduler object is created at startup using the provided configuration.
-type Scheduler interface {
-	// PreSchedule selects scheduling profiles through the implemented
-	// logic, and returns:
-	// - profiles - A subset of the registered scheduling profiles to be ran
-	PreSchedule(request map[string]any, data scheduling.CycleState, results map[string][]Endpoint) map[string]SchedulingProfile
-
-	// PostSchedule receives the output of the result(s) of the scheduling cycle(s)
-	// and makes sense of the data to be consumed by the calling system.
-	// For example: suppose you have 2 profiles ShadowBoxing Profile & Production Profile.
-	// PostSchedule would know to simply log the result of ShadowBoxing
-	// profile, and do nothing else with it.
-	PostSchedule(profileResults map[string][]Endpoint) SchedulingResult
+// ScoredEndpoint encapsulates Endpoint with its Score.
+// The lifecycle of an endpoint is typically different than a lifecycle of a request.
+// This is intended to be used only internally by Scheduler logic and/or scheduler plugins within the lifecycle of the request.
+// When returning the selected Endpoint(s) out of the Scheduler, an Endpoint is returned without the score.
+type ScoredEndpoint struct {
+	Endpoint
+	Score float64
+}
+
+type Scheduler struct {
+	SchedulerConfig
 }
 
-// SchedulingProfile is used to describe a profile that will
+// SchedulerConfig is the struct that maps to the configuration file that should be further discussed.
+// the configuration file should include the ProfileHandler plugin as well as the profiles with their plugins.
+type SchedulerConfig struct {
+	// exactly one ProfileHandler instance is required.
+	profileHandler ProfileHandler //nolint:unused
+	// map from profile name to its set of plugins.
+	profiles map[string]*SchedulerProfile //nolint:unused
+}
+
+// SchedulerProfile is used to describe a profile that will
 // run for a given scheduling cycle.
-type SchedulingProfile struct {
-	// Name of the profile.
-	Name string
-	// Filters lists all Filter plugins associated with this Profile. Filters
-	// are optional.
-	Filters []Filter
-	// Scorers lists all Score plugins associated with this Profile. Scorers
-	// are optional.
-	Scorers map[Scorer]int
+type SchedulerProfile struct {
+	// Filters lists all Filter plugins associated with this Profile.
+	// Filters are optional.
+	filters []Filter //nolint:unused
+	// Scorers lists all Score plugins associated with this Profile.
+	// Scorers are optional.
+	scorers []*WeightedScorer //nolint:unused
 	// Picker returns the function that picks the endpoint(s). Picker is required.
-	Picker Picker
+	picker Picker //nolint:unused
 }
 
-// Filter runs before any scoring, and remove endpoints that are not fit for
-// selection. The framework will return an error to the client if the endpoints
-// are filtered to zero.
+type SchedulingResult struct {
+	ProfileResults     map[string][]*Endpoint // a map from profile name to its scheduling result
+	PrimaryProfileName string                 // key of the primary profile, its selected endpoints will be used by default as the destination
+}
+
+// Plugin is the parent type for all the scheduling framework plugins.
+type Plugin interface {
+	Type() string
+}
+
+// ProfileHandler defines the interface for handling multi SchedulerProfile instances.
+// More specifically, this interfaction defines two extension points, 'PickProfiles'
+// which runs iteratively, and 'ProcessProfilesResults' which runs after all profiles runs complete
+// and process the results of all profiles.
+type ProfileHandler interface {
+	Plugin
+	// Pick picks the SchedulingProfile objects to run from a list of candidate profiles,
+	// while taking into consideration the request properties
+	// and the previously executed SchedluderProfile runs along with their results.
+	// returns:
+	// - profiles - A subset of the registered scheduling profiles to be ran in next iteration
+	Pick(request *Request, profiles map[string]*SchedulerProfile, executionResults map[string][]*ScoredEndpoint) map[string]*SchedulerProfile
+
+	// ProcessResults handles the outcome of each profile run.
+	// It may aggregate results, log test profile outputs, or apply custom logic. It specifies in the SchedulingResult the
+	// key of the primary profile that should be used to get the request selected destination.
+	// Example: suppose you have 2 profiles ShadowBoxing Profile & Production Profile.
+	// ProcessProfileResults would know to simply log the result of ShadowBoxing
+	// profile, and do nothing else with it.
+	ProcessResults(request *Request, profileResults map[string][]*ScoredEndpoint) *SchedulingResult
+}
+
+// Filter runs before any scoring, and remove endpoints that are not fit for selection.
+// The framework will return an error to the client if the endpoints are filtered to zero.
 type Filter interface {
 	Plugin
-	Filter(ctx context.Context, state scheduling.CycleState, endpoints []Endpoint) []Endpoint
+	Filter(ctx context.Context, request *Request, state *scheduling.CycleState, endpoints []*Endpoint) []*Endpoint
 }
 
-// Scorer applies a score to each remaining endpoint provided. Scorers SHOULD
-// keep their score values in a normalized range: [0-1]. Any weighting should
-// be added at the SchedulingProfile configuration level.
+// Scorer applies a score to each remaining endpoint provided.
+// Scorers SHOULD keep their score values in a normalized range: [0-1].
+// Any weighting should be added at the SchedulerProfile configuration level.
 type Scorer interface {
 	Plugin
-	Score(ctx context.Context, state scheduling.CycleState, endpoints []Endpoint) []Endpoint
+	Score(ctx context.Context, request *Request, state *scheduling.CycleState, endpoints []*Endpoint) []*ScoredEndpoint
+}
+
+// WeightedScorer is a struct that encapsulates a scorer with its weight.
+// We need this struct in order to be able to keep scorers in profile as a slice instead of a map.
+// This is very useful for having a generic AddPlugin function that registers a plugin to all its extension points.
+// Using a map is much less convenient for this purpose.
+type WeightedScorer struct {
+	Scorer
+	weight int //nolint:unused
 }
 
 // Picker selects the endpoint(s) from the provided list of scored endpoints.
 // Picker MUST return, one endpoint at minimum.
 type Picker interface {
 	Plugin
-	Pick(ctx context.Context, state scheduling.CycleState, endpoints []Endpoint) []Endpoint
-}
-
-type PostResponse interface {
-	Plugin
-	PostResponse(ctx context.Context, request map[string]any, response map[string]any)
+	Pick(ctx context.Context, state *scheduling.CycleState, endpoints []*ScoredEndpoint) []*ScoredEndpoint
 }
diff --git a/go.mod b/go.mod
index b6784ed..5e04947 100644
--- a/go.mod
+++ b/go.mod
@@ -9,22 +9,24 @@ require (
 	github.com/go-logr/logr v1.4.3
 	github.com/google/go-cmp v0.7.0
 	github.com/google/uuid v1.6.0
+	github.com/hashicorp/golang-lru/v2 v2.0.7
 	github.com/onsi/ginkgo/v2 v2.23.4
 	github.com/onsi/gomega v1.37.0
 	github.com/prometheus/client_golang v1.22.0
 	github.com/prometheus/client_model v0.6.2
-	github.com/prometheus/common v0.64.0
+	github.com/prometheus/common v0.65.0
 	github.com/stretchr/testify v1.10.0
 	go.uber.org/multierr v1.11.0
 	go.uber.org/zap v1.27.0
-	google.golang.org/grpc v1.71.1
+	golang.org/x/sync v0.14.0
+	google.golang.org/grpc v1.73.0
 	google.golang.org/protobuf v1.36.6
-	k8s.io/api v0.33.1
-	k8s.io/apiextensions-apiserver v0.33.1
-	k8s.io/apimachinery v0.33.1
-	k8s.io/client-go v0.33.1
-	k8s.io/code-generator v0.33.1
-	k8s.io/component-base v0.33.1
+	k8s.io/api v0.33.2
+	k8s.io/apiextensions-apiserver v0.33.2
+	k8s.io/apimachinery v0.33.2
+	k8s.io/client-go v0.33.2
+	k8s.io/code-generator v0.33.2
+	k8s.io/component-base v0.33.2
 	k8s.io/utils v0.0.0-20241210054802-24370beab758
 	sigs.k8s.io/controller-runtime v0.21.0
 	sigs.k8s.io/gateway-api v1.3.0
@@ -33,7 +35,7 @@ require (
 )
 
 require (
-	cel.dev/expr v0.19.1 // indirect
+	cel.dev/expr v0.23.0 // indirect
 	github.com/Masterminds/goutils v1.1.1 // indirect
 	github.com/Masterminds/semver v1.5.0 // indirect
 	github.com/Masterminds/sprig v2.22.0+incompatible // indirect
@@ -41,7 +43,7 @@ require (
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
 	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
-	github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3 // indirect
+	github.com/cncf/xds/go v0.0.0-20250326154945-ae57f3c0d45f // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/emicklei/go-restful/v3 v3.12.0 // indirect
 	github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
@@ -94,12 +96,12 @@ require (
 	github.com/x448/float16 v0.8.4 // indirect
 	go.opentelemetry.io/auto/sdk v1.1.0 // indirect
 	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0 // indirect
-	go.opentelemetry.io/otel v1.34.0 // indirect
+	go.opentelemetry.io/otel v1.35.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0 // indirect
 	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0 // indirect
-	go.opentelemetry.io/otel/metric v1.34.0 // indirect
-	go.opentelemetry.io/otel/sdk v1.34.0 // indirect
-	go.opentelemetry.io/otel/trace v1.34.0 // indirect
+	go.opentelemetry.io/otel/metric v1.35.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.35.0 // indirect
+	go.opentelemetry.io/otel/trace v1.35.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.4.0 // indirect
 	go.uber.org/automaxprocs v1.6.0 // indirect
 	golang.org/x/crypto v0.38.0 // indirect
@@ -107,7 +109,6 @@ require (
 	golang.org/x/mod v0.24.0 // indirect
 	golang.org/x/net v0.40.0 // indirect
 	golang.org/x/oauth2 v0.30.0 // indirect
-	golang.org/x/sync v0.14.0 // indirect
 	golang.org/x/sys v0.33.0 // indirect
 	golang.org/x/term v0.32.0 // indirect
 	golang.org/x/text v0.25.0 // indirect
@@ -115,13 +116,13 @@ require (
 	golang.org/x/tools v0.31.0 // indirect
 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20250106144421-5f5ef82da422 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20250324211829-b45e905df463 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20250324211829-b45e905df463 // indirect
 	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
-	k8s.io/apiserver v0.33.1 // indirect
+	k8s.io/apiserver v0.33.2 // indirect
 	k8s.io/gengo/v2 v2.0.0-20250207200755-1244d31929d7 // indirect
 	k8s.io/klog/v2 v2.130.1 // indirect
 	k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect
diff --git a/go.sum b/go.sum
index 2e49874..d4b03f7 100644
--- a/go.sum
+++ b/go.sum
@@ -1,5 +1,5 @@
-cel.dev/expr v0.19.1 h1:NciYrtDRIR0lNCnH1LFJegdjspNx9fI59O7TWcua/W4=
-cel.dev/expr v0.19.1/go.mod h1:MrpN08Q+lEBs+bGYdLxxHkZoUSsCp0nSKTs0nTymJgw=
+cel.dev/expr v0.23.0 h1:wUb94w6OYQS4uXraxo9U+wUAs9jT47Xvl4iPgAwM2ss=
+cel.dev/expr v0.23.0/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
 github.com/Masterminds/goutils v1.1.1 h1:5nUrii3FMTL5diU80unEVvNevw1nH4+ZV4DSLVJLSYI=
 github.com/Masterminds/goutils v1.1.1/go.mod h1:8cTjp+g8YejhMuvIA5y2vz3BpJxksy863GQaJW2MFNU=
 github.com/Masterminds/semver v1.5.0 h1:H65muMkzWKEuNDnfl9d70GUjFniHKHRbFPGBuZ3QEww=
@@ -18,8 +18,8 @@ github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK3
 github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3 h1:boJj011Hh+874zpIySeApCX4GeOjPl9qhRF3QuIZq+Q=
-github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
+github.com/cncf/xds/go v0.0.0-20250326154945-ae57f3c0d45f h1:C5bqEmzEPLsHm9Mv73lSE9e9bKV23aB1vxOsmZrkl3k=
+github.com/cncf/xds/go v0.0.0-20250326154945-ae57f3c0d45f/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
@@ -95,6 +95,8 @@ github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 h1:JeSE6pjso5T
 github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674/go.mod h1:r4w70xmWCQKmi1ONH4KIaBptdivuRPyosB9RmPlGEwA=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 h1:TmHmbvxPmaegwhDubVz0lICL0J5Ka2vwTzhoePEXsGE=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0/go.mod h1:qztMSjm835F2bXf+5HKAPIS5qsmQDqZna/PgVt4rWtI=
+github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
+github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/huandu/xstrings v1.3.3 h1:/Gcsuc1x8JVbJ9/rlye4xZnVAbEkGauT8lbebqcQws4=
 github.com/huandu/xstrings v1.3.3/go.mod h1:y5/lhBue+AyNmUVz9RLU9xbLR0o4KIIExikq4ovT0aE=
 github.com/imdario/mergo v0.3.16 h1:wwQJbIsHYGMUyLSPrEq1CT16AhnhNJQ51+4fdHUnCl4=
@@ -160,8 +162,8 @@ github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/
 github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
 github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
 github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
-github.com/prometheus/common v0.64.0 h1:pdZeA+g617P7oGv1CzdTzyeShxAGrTBsolKNOLQPGO4=
-github.com/prometheus/common v0.64.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
+github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE=
+github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
 github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
 github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
 github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
@@ -193,20 +195,20 @@ go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJyS
 go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0 h1:yd02MEjBdJkG3uabWP9apV+OuWRIXGDuJEUJbOHmCFU=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0/go.mod h1:umTcuxiv1n/s/S6/c2AT/g2CQ7u5C59sHDNmfSwgz7Q=
-go.opentelemetry.io/otel v1.34.0 h1:zRLXxLCgL1WyKsPVrgbSdMN4c0FMkDAskSTQP+0hdUY=
-go.opentelemetry.io/otel v1.34.0/go.mod h1:OWFPOQ+h4G8xpyjgqo4SxJYdDQ/qmRH+wivy7zzx9oI=
+go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
+go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0 h1:Vh5HayB/0HHfOQA7Ctx69E/Y/DcQSMPpKANYVMQ7fBA=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0/go.mod h1:cpgtDBaqD/6ok/UG0jT15/uKjAY8mRA53diogHBg3UI=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0 h1:5pojmb1U1AogINhN3SurB+zm/nIcusopeBNp42f45QM=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0/go.mod h1:57gTHJSE5S1tqg+EKsLPlTWhpHMsWlVmer+LA926XiA=
-go.opentelemetry.io/otel/metric v1.34.0 h1:+eTR3U0MyfWjRDhmFMxe2SsW64QrZ84AOhvqS7Y+PoQ=
-go.opentelemetry.io/otel/metric v1.34.0/go.mod h1:CEDrp0fy2D0MvkXE+dPV7cMi8tWZwX3dmaIhwPOaqHE=
-go.opentelemetry.io/otel/sdk v1.34.0 h1:95zS4k/2GOy069d321O8jWgYsW3MzVV+KuSPKp7Wr1A=
-go.opentelemetry.io/otel/sdk v1.34.0/go.mod h1:0e/pNiaMAqaykJGKbi+tSjWfNNHMTxoC9qANsCzbyxU=
-go.opentelemetry.io/otel/sdk/metric v1.34.0 h1:5CeK9ujjbFVL5c1PhLuStg1wxA7vQv7ce1EK0Gyvahk=
-go.opentelemetry.io/otel/sdk/metric v1.34.0/go.mod h1:jQ/r8Ze28zRKoNRdkjCZxfs6YvBTG1+YIqyFVFYec5w=
-go.opentelemetry.io/otel/trace v1.34.0 h1:+ouXS2V8Rd4hp4580a8q23bg0azF2nI8cqLYnC8mh/k=
-go.opentelemetry.io/otel/trace v1.34.0/go.mod h1:Svm7lSjQD7kG7KJ/MUHPVXSDGz2OX4h0M2jHBhmSfRE=
+go.opentelemetry.io/otel/metric v1.35.0 h1:0znxYu2SNyuMSQT4Y9WDWej0VpcsxkuklLa4/siN90M=
+go.opentelemetry.io/otel/metric v1.35.0/go.mod h1:nKVFgxBZ2fReX6IlyW28MgZojkoAkJGaE8CpgeAU3oE=
+go.opentelemetry.io/otel/sdk v1.35.0 h1:iPctf8iprVySXSKJffSS79eOjl9pvxV9ZqOWT0QejKY=
+go.opentelemetry.io/otel/sdk v1.35.0/go.mod h1:+ga1bZliga3DxJ3CQGg3updiaAJoNECOgJREo9KHGQg=
+go.opentelemetry.io/otel/sdk/metric v1.35.0 h1:1RriWBmCKgkeHEhM7a2uMjMUfP7MsOF5JpUCaEqEI9o=
+go.opentelemetry.io/otel/sdk/metric v1.35.0/go.mod h1:is6XYCUMpcKi+ZsOvfluY5YstFnhW0BidkR+gL+qN+w=
+go.opentelemetry.io/otel/trace v1.35.0 h1:dPpEfJu1sDIqruz7BHFG3c7528f6ddfSWfFDVt/xgMs=
+go.opentelemetry.io/otel/trace v1.35.0/go.mod h1:WUk7DtFp1Aw2MkvqGdwiXYDZZNvA/1J8o6xRXLrIkyc=
 go.opentelemetry.io/proto/otlp v1.4.0 h1:TA9WRvW6zMwP+Ssb6fLoUIuirti1gGbP28GcKG1jgeg=
 go.opentelemetry.io/proto/otlp v1.4.0/go.mod h1:PPBWZIP98o2ElSqI35IHfu7hIhSwvc5N38Jw8pXuGFY=
 go.uber.org/automaxprocs v1.6.0 h1:O3y2/QNTOdbF+e/dpXNNW7Rx2hZ4sTIPyybbxyNqTUs=
@@ -271,12 +273,12 @@ golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 h1:+cNy6SZtPcJQH3LJVLOSm
 golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028/go.mod h1:NDW/Ps6MPRej6fsCIbMTohpP40sJ/P/vI1MoTEGwX90=
 gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw=
 gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY=
-google.golang.org/genproto/googleapis/api v0.0.0-20250106144421-5f5ef82da422 h1:GVIKPyP/kLIyVOgOnTwFOrvQaQUzOzGMCxgFUOEmm24=
-google.golang.org/genproto/googleapis/api v0.0.0-20250106144421-5f5ef82da422/go.mod h1:b6h1vNKhxaSoEI+5jc3PJUCustfli/mRab7295pY7rw=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f h1:OxYkA3wjPsZyBylwymxSHa7ViiW1Sml4ToBrncvFehI=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20250115164207-1a7da9e5054f/go.mod h1:+2Yz8+CLJbIfL9z73EW45avw8Lmge3xVElCP9zEKi50=
-google.golang.org/grpc v1.71.1 h1:ffsFWr7ygTUscGPI0KKK6TLrGz0476KUvvsbqWK0rPI=
-google.golang.org/grpc v1.71.1/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
+google.golang.org/genproto/googleapis/api v0.0.0-20250324211829-b45e905df463 h1:hE3bRWtU6uceqlh4fhrSnUyjKHMKB9KrTLLG+bc0ddM=
+google.golang.org/genproto/googleapis/api v0.0.0-20250324211829-b45e905df463/go.mod h1:U90ffi8eUL9MwPcrJylN5+Mk2v3vuPDptd5yyNUiRR8=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250324211829-b45e905df463 h1:e0AIkUUhxyBKh6ssZNrAMeqhA7RKUj42346d1y02i2g=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250324211829-b45e905df463/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
+google.golang.org/grpc v1.73.0 h1:VIWSmpI2MegBtTuFt5/JWy2oXxtjJ/e89Z70ImfD2ok=
+google.golang.org/grpc v1.73.0/go.mod h1:50sbHOUqWoCQGI8V2HQLJM0B+LMlIUjNSZmow7EVBQc=
 google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
 google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
@@ -293,20 +295,20 @@ gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-k8s.io/api v0.33.1 h1:tA6Cf3bHnLIrUK4IqEgb2v++/GYUtqiu9sRVk3iBXyw=
-k8s.io/api v0.33.1/go.mod h1:87esjTn9DRSRTD4fWMXamiXxJhpOIREjWOSjsW1kEHw=
-k8s.io/apiextensions-apiserver v0.33.1 h1:N7ccbSlRN6I2QBcXevB73PixX2dQNIW0ZRuguEE91zI=
-k8s.io/apiextensions-apiserver v0.33.1/go.mod h1:uNQ52z1A1Gu75QSa+pFK5bcXc4hq7lpOXbweZgi4dqA=
-k8s.io/apimachinery v0.33.1 h1:mzqXWV8tW9Rw4VeW9rEkqvnxj59k1ezDUl20tFK/oM4=
-k8s.io/apimachinery v0.33.1/go.mod h1:BHW0YOu7n22fFv/JkYOEfkUYNRN0fj0BlvMFWA7b+SM=
-k8s.io/apiserver v0.33.1 h1:yLgLUPDVC6tHbNcw5uE9mo1T6ELhJj7B0geifra3Qdo=
-k8s.io/apiserver v0.33.1/go.mod h1:VMbE4ArWYLO01omz+k8hFjAdYfc3GVAYPrhP2tTKccs=
-k8s.io/client-go v0.33.1 h1:ZZV/Ks2g92cyxWkRRnfUDsnhNn28eFpt26aGc8KbXF4=
-k8s.io/client-go v0.33.1/go.mod h1:JAsUrl1ArO7uRVFWfcj6kOomSlCv+JpvIsp6usAGefA=
-k8s.io/code-generator v0.33.1 h1:ZLzIRdMsh3Myfnx9BaooX6iQry29UJjVfVG+BuS+UMw=
-k8s.io/code-generator v0.33.1/go.mod h1:HUKT7Ubp6bOgIbbaPIs9lpd2Q02uqkMCMx9/GjDrWpY=
-k8s.io/component-base v0.33.1 h1:EoJ0xA+wr77T+G8p6T3l4efT2oNwbqBVKR71E0tBIaI=
-k8s.io/component-base v0.33.1/go.mod h1:guT/w/6piyPfTgq7gfvgetyXMIh10zuXA6cRRm3rDuY=
+k8s.io/api v0.33.2 h1:YgwIS5jKfA+BZg//OQhkJNIfie/kmRsO0BmNaVSimvY=
+k8s.io/api v0.33.2/go.mod h1:fhrbphQJSM2cXzCWgqU29xLDuks4mu7ti9vveEnpSXs=
+k8s.io/apiextensions-apiserver v0.33.2 h1:6gnkIbngnaUflR3XwE1mCefN3YS8yTD631JXQhsU6M8=
+k8s.io/apiextensions-apiserver v0.33.2/go.mod h1:IvVanieYsEHJImTKXGP6XCOjTwv2LUMos0YWc9O+QP8=
+k8s.io/apimachinery v0.33.2 h1:IHFVhqg59mb8PJWTLi8m1mAoepkUNYmptHsV+Z1m5jY=
+k8s.io/apimachinery v0.33.2/go.mod h1:BHW0YOu7n22fFv/JkYOEfkUYNRN0fj0BlvMFWA7b+SM=
+k8s.io/apiserver v0.33.2 h1:KGTRbxn2wJagJowo29kKBp4TchpO1DRO3g+dB/KOJN4=
+k8s.io/apiserver v0.33.2/go.mod h1:9qday04wEAMLPWWo9AwqCZSiIn3OYSZacDyu/AcoM/M=
+k8s.io/client-go v0.33.2 h1:z8CIcc0P581x/J1ZYf4CNzRKxRvQAwoAolYPbtQes+E=
+k8s.io/client-go v0.33.2/go.mod h1:9mCgT4wROvL948w6f6ArJNb7yQd7QsvqavDeZHvNmHo=
+k8s.io/code-generator v0.33.2 h1:PCJ0Y6viTCxxJHMOyGqYwWEteM4q6y1Hqo2rNpl6jF4=
+k8s.io/code-generator v0.33.2/go.mod h1:hBjCA9kPMpjLWwxcr75ReaQfFXY8u+9bEJJ7kRw3J8c=
+k8s.io/component-base v0.33.2 h1:sCCsn9s/dG3ZrQTX/Us0/Sx2R0G5kwa0wbZFYoVp/+0=
+k8s.io/component-base v0.33.2/go.mod h1:/41uw9wKzuelhN+u+/C59ixxf4tYQKW7p32ddkYNe2k=
 k8s.io/gengo/v2 v2.0.0-20250207200755-1244d31929d7 h1:2OX19X59HxDprNCVrWi6jb7LW1PoqTlYqEq5H2oetog=
 k8s.io/gengo/v2 v2.0.0-20250207200755-1244d31929d7/go.mod h1:EJykeLsmFC60UQbYJezXkEsG2FLrt0GPNkU5iK5GWxU=
 k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
diff --git a/hack/release-quickstart.sh b/hack/release-quickstart.sh
index b2f5071..068be7f 100755
--- a/hack/release-quickstart.sh
+++ b/hack/release-quickstart.sh
@@ -29,11 +29,18 @@ else
   RELEASE_TAG="v${MAJOR}.${MINOR}.0-rc.${RC}"
 fi
 
-# vLLM image version (default to 0.7.2 if not defined)
-VLLM="${VLLM:-0.7.2}"
+# The vLLM image versions
+# The GPU image is from https://hub.docker.com/layers/vllm/vllm-openai
+VLLM_GPU="${VLLM_GPU:-0.9.1}"
+# The CPU image is from https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
+VLLM_CPU="${VLLM_CPU:-0.9.1}"
+# The sim image is from https://github.com/llm-d/llm-d-inference-sim/pkgs/container/llm-d-inference-sim
+VLLM_SIM="${VLLM_SIM:-0.1.1}"
 
 echo "Using release tag: ${RELEASE_TAG}"
-echo "Using vLLM image version: ${VLLM}"
+echo "Using vLLM GPU image version: ${VLLM_GPU}"
+echo "Using vLLM CPU image version: ${VLLM_CPU}"
+echo "Using vLLM Simulator image version: ${VLLM_SIM}"
 
 # -----------------------------------------------------------------------------
 # Update pkg/README.md
@@ -64,7 +71,7 @@ sed -i.bak -E "s|(tag: )[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$EPP_HELM"
 sed -i.bak -E "s|(tag: )[^\"[:space:]]+|\1${RELEASE_TAG}|g" "$BBR_HELM"
 
 # Update the container image pull policy.
-sed -i.bak '/us-central1-docker.pkg.dev\/k8s-staging-images\/gateway-api-inference-extension\/epp/ { n; s/Always/IfNotPresent/ }' "$EPP"
+sed -i.bak '/us-central1-docker.pkg.dev\/k8s-staging-images\/gateway-api-inference-extension\/epp/{n;s/Always/IfNotPresent/;}' "$EPP"
 
 # Update the container registry.
 sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$EPP"
@@ -72,22 +79,40 @@ sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io
 sed -i.bak -E "s|us-central1-docker\.pkg\.dev/k8s-staging-images|registry.k8s.io|g" "$BBR_HELM"
 
 # -----------------------------------------------------------------------------
-# Update config/manifests/vllm/gpu-deployment.yaml
+# Update vLLM deployment manifests
 # -----------------------------------------------------------------------------
-VLLM_DEPLOY="config/manifests/vllm/gpu-deployment.yaml"
-echo "Updating ${VLLM_DEPLOY} ..."
+VLLM_GPU_DEPLOY="config/manifests/vllm/gpu-deployment.yaml"
+echo "Updating ${VLLM_GPU_DEPLOY} ..."
 
-# Update the vLLM image version
-sed -i.bak -E "s|(vllm/vllm-openai:)[^\"[:space:]]+|\1v${VLLM}|g" "$VLLM_DEPLOY"
+# Update the vLLM GPU image version
+sed -i.bak -E "s|(vllm/vllm-openai:)[^\"[:space:]]+|\1v${VLLM_GPU}|g" "$VLLM_GPU_DEPLOY"
 
 # Also change the imagePullPolicy from Always to IfNotPresent on lines containing the vLLM image.
-sed -i.bak '/vllm\/vllm-openai/ { n; s/Always/IfNotPresent/ }' "$VLLM_DEPLOY"
+sed -i.bak '/vllm\/vllm-openai/{n;s/Always/IfNotPresent/;}' "$VLLM_GPU_DEPLOY"
+
+VLLM_CPU_DEPLOY="config/manifests/vllm/cpu-deployment.yaml"
+echo "Updating ${VLLM_CPU_DEPLOY} ..."
+
+# Update the vLLM CPU image version
+sed -i.bak -E "s|(q9t5s3a7/vllm-cpu-release-repo:)[^\"[:space:]]+|\1v${VLLM_CPU}|g" "$VLLM_CPU_DEPLOY"
+
+# Also change the imagePullPolicy from Always to IfNotPresent on lines containing the vLLM CPU image.
+sed -i.bak '/q9t5s3a7\/vllm-cpu-release-repo/{n;s/Always/IfNotPresent/;}' "$VLLM_CPU_DEPLOY"
+
+VLLM_SIM_DEPLOY="config/manifests/vllm/sim-deployment.yaml"
+echo "Updating ${VLLM_SIM_DEPLOY} ..."
+
+# Update the vLLM Simulator image version
+sed -i.bak -E "s|(llm-d/llm-d-inference-sim:)[^\"[:space:]]+|\1v${VLLM_SIM}|g" "$VLLM_SIM_DEPLOY"
+
+# Also change the imagePullPolicy from Always to IfNotPresent on lines containing the vLLM image.
+sed -i.bak '/llm-d\/llm-d-inference-sim/{n;s/Always/IfNotPresent/;}' "$VLLM_SIM_DEPLOY"
 
 # -----------------------------------------------------------------------------
 # Stage the changes
 # -----------------------------------------------------------------------------
-echo "Staging $README $EPP $EPP_HELM $BBR_HELM $VLLM_DEPLOY files..."
-git add $README $EPP $EPP_HELM $BBR_HELM $VLLM_DEPLOY
+echo "Staging $README $EPP $EPP_HELM $BBR_HELM $VLLM_GPU_DEPLOY $VLLM_CPU_DEPLOY $VLLM_SIM_DEPLOY files..."
+git add $README $EPP $EPP_HELM $BBR_HELM $VLLM_GPU_DEPLOY $VLLM_CPU_DEPLOY $VLLM_SIM_DEPLOY
 
 # -----------------------------------------------------------------------------
 # Cleanup backup files and finish
diff --git a/hack/run-e2es.sh b/hack/run-e2es.sh
new file mode 100755
index 0000000..9d4deda
--- /dev/null
+++ b/hack/run-e2es.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+# Copyright 2025 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -euox pipefail
+
+install_kind() {
+  if ! command -v kind &>/dev/null; then
+    echo "kind not found, installing..."
+    [ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-amd64
+    # For ARM64
+    [ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-arm64
+    chmod +x ./kind
+    mv ./kind /usr/local/bin/kind
+  else
+    echo "kind is already installed."
+  fi
+}
+
+if kubectl config current-context >/dev/null 2>&1; then
+  echo "Active kubecontext found. Running Go e2e tests in ./epp..."
+else
+  install_kind
+  kind create cluster --name inference-e2e
+  KIND_CLUSTER=inference-e2e make image-kind
+  # Add the Gateway API CRDs
+  VERSION=v0.3.0
+  echo "Kind cluster created. Running Go e2e tests in ./epp..."
+fi
+
+go test ./test/e2e/epp/ -v -ginkgo.v
diff --git a/mkdocs.yml b/mkdocs.yml
index cc76400..c8629c1 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -52,6 +52,7 @@ nav:
     - Introduction: index.md
     - Concepts:
         API Overview: concepts/api-overview.md
+        Design Principles: concepts/design-principles.md
         Conformance: concepts/conformance.md
         Roles and Personas: concepts/roles-and-personas.md
     - Implementations: 
@@ -71,6 +72,9 @@ nav:
       - Configuration Guide:
           - Prefix Cache Aware Plugin: guides/epp-configuration/prefix-aware.md
     - Implementer's Guide: guides/implementers.md
+    - Implementer Guides:
+      - Getting started: guides/implementers.md
+      - Conformance Tests: guides/conformance-tests.md
   - Performance:
     - Benchmark: performance/benchmark/index.md
     - Regression Testing: performance/regression-testing/index.md
diff --git a/pkg/epp/common/config/loader/configloader.go b/pkg/epp/common/config/loader/configloader.go
new file mode 100644
index 0000000..ba308fe
--- /dev/null
+++ b/pkg/epp/common/config/loader/configloader.go
@@ -0,0 +1,189 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loader
+
+import (
+	"errors"
+	"fmt"
+	"os"
+
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+
+	"sigs.k8s.io/gateway-api-inference-extension/api/config/v1alpha1"
+	configapi "sigs.k8s.io/gateway-api-inference-extension/api/config/v1alpha1"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+)
+
+var scheme = runtime.NewScheme()
+
+func init() {
+	configapi.SchemeBuilder.Register(configapi.RegisterDefaults)
+	utilruntime.Must(configapi.Install(scheme))
+}
+
+// Load config either from supplied text or from a file
+func LoadConfig(configText []byte, fileName string) (*configapi.EndpointPickerConfig, error) {
+	var err error
+	if len(configText) == 0 {
+		configText, err = os.ReadFile(fileName)
+		if err != nil {
+			return nil, fmt.Errorf("failed to load config file. Error: %s", err)
+		}
+	}
+
+	theConfig := &configapi.EndpointPickerConfig{}
+
+	codecs := serializer.NewCodecFactory(scheme, serializer.EnableStrict)
+	err = runtime.DecodeInto(codecs.UniversalDecoder(), configText, theConfig)
+	if err != nil {
+		return nil, fmt.Errorf("the configuration is invalid. Error: %s", err)
+	}
+
+	// Validate loaded configuration
+	err = validateConfiguration(theConfig)
+	if err != nil {
+		return nil, fmt.Errorf("the configuration is invalid. error: %s", err)
+	}
+	return theConfig, nil
+}
+
+func LoadPluginReferences(thePlugins []configapi.PluginSpec, handle plugins.Handle) error {
+	for _, pluginConfig := range thePlugins {
+		thePlugin, err := instantiatePlugin(pluginConfig, handle)
+		if err != nil {
+			return err
+		}
+		handle.Plugins().AddPlugin(pluginConfig.Name, thePlugin)
+	}
+	return nil
+}
+
+func LoadSchedulerConfig(configProfiles []v1alpha1.SchedulingProfile, handle plugins.Handle) (*scheduling.SchedulerConfig, error) {
+
+	var profiles = map[string]*framework.SchedulerProfile{}
+
+	for _, configProfile := range configProfiles {
+		profile := framework.SchedulerProfile{}
+
+		for _, plugin := range configProfile.Plugins {
+			var err error
+			thePlugin := handle.Plugins().Plugin(plugin.PluginRef)
+			if theScorer, ok := thePlugin.(framework.Scorer); ok {
+				if plugin.Weight == nil {
+					return nil, fmt.Errorf("scorer '%s' is missing a weight", plugin.PluginRef)
+				}
+				thePlugin = framework.NewWeightedScorer(theScorer, *plugin.Weight)
+			}
+			err = profile.AddPlugins(thePlugin)
+			if err != nil {
+				return nil, err
+			}
+		}
+		profiles[configProfile.Name] = &profile
+	}
+
+	var profileHandler framework.ProfileHandler
+	var profileHandlerName string
+
+	for pluginName, thePlugin := range handle.Plugins().GetAllPluginsWithNames() {
+		if theProfileHandler, ok := thePlugin.(framework.ProfileHandler); ok {
+			if profileHandler != nil {
+				return nil, fmt.Errorf("only one profile handler is allowed. Both %s and %s are profile handlers", profileHandlerName, pluginName)
+			}
+			profileHandler = theProfileHandler
+			profileHandlerName = pluginName
+		}
+	}
+	if profileHandler == nil {
+		return nil, errors.New("no profile handler was specified")
+	}
+
+	return scheduling.NewSchedulerConfig(profileHandler, profiles), nil
+}
+
+func instantiatePlugin(pluginSpec configapi.PluginSpec, handle plugins.Handle) (plugins.Plugin, error) {
+	factory, ok := plugins.Registry[pluginSpec.PluginName]
+	if !ok {
+		return nil, fmt.Errorf("failed to instantiate the plugin. plugin %s not found", pluginSpec.PluginName)
+	}
+	thePlugin, err := factory(pluginSpec.Name, pluginSpec.Parameters, handle)
+	if err != nil {
+		return nil, fmt.Errorf("failed to instantiate the plugin %s. Error: %s", pluginSpec.PluginName, err)
+	}
+	return thePlugin, err
+}
+
+func validateConfiguration(theConfig *configapi.EndpointPickerConfig) error {
+	names := make(map[string]struct{})
+
+	for _, pluginConfig := range theConfig.Plugins {
+		if pluginConfig.PluginName == "" {
+			return errors.New("plugin reference definition missing a plugin name")
+		}
+
+		if _, ok := names[pluginConfig.Name]; ok {
+			return fmt.Errorf("the name %s has been specified for more than one plugin", pluginConfig.Name)
+		}
+		names[pluginConfig.Name] = struct{}{}
+
+		_, ok := plugins.Registry[pluginConfig.PluginName]
+		if !ok {
+			return fmt.Errorf("plugin %s is not found", pluginConfig.PluginName)
+		}
+	}
+
+	if len(theConfig.SchedulingProfiles) == 0 {
+		return errors.New("there must be at least one scheduling profile in the configuration")
+	}
+
+	names = map[string]struct{}{}
+	for _, profile := range theConfig.SchedulingProfiles {
+		if profile.Name == "" {
+			return errors.New("SchedulingProfiles need a name")
+		}
+
+		if _, ok := names[profile.Name]; ok {
+			return fmt.Errorf("the name %s has been specified for more than one SchedulingProfile", profile.Name)
+		}
+		names[profile.Name] = struct{}{}
+
+		if len(profile.Plugins) == 0 {
+			return errors.New("SchedulingProfiles need at least one plugin")
+		}
+		for _, plugin := range profile.Plugins {
+			if len(plugin.PluginRef) == 0 {
+				return errors.New("SchedulingProfile's plugins need a plugin reference")
+			}
+
+			notFound := true
+			for _, pluginConfig := range theConfig.Plugins {
+				if plugin.PluginRef == pluginConfig.Name {
+					notFound = false
+					break
+				}
+			}
+			if notFound {
+				return errors.New(plugin.PluginRef + " is a reference to an undefined Plugin")
+			}
+		}
+	}
+	return nil
+}
diff --git a/pkg/epp/plugins/plugins.go b/pkg/epp/plugins/plugins.go
index 5dd8d87..b0bee1c 100644
--- a/pkg/epp/plugins/plugins.go
+++ b/pkg/epp/plugins/plugins.go
@@ -19,6 +19,27 @@ package plugins
 // Plugin defines the interface for a plugin.
 // This interface should be embedded in all plugins across the code.
 type Plugin interface {
-	// Name returns the name of the plugin.
-	Name() string
+	// Type returns the type of the plugin.
+	Type() string
+}
+
+// Handle provides plugins  set of standard data and tools to work with
+type Handle interface {
+	// Plugins returns the sub-handle for working with instantiated plugins
+	Plugins() HandlePlugins
+}
+
+// HandlePlugins defines a set of APIs to work with instantiated plugins
+type HandlePlugins interface {
+	// Plugin returns the named plugin instance
+	Plugin(name string) Plugin
+
+	// AddPlugin adds a plugin to the set of known plugin instances
+	AddPlugin(name string, plugin Plugin)
+
+	// GetAllPlugins returns all of the known plugins
+	GetAllPlugins() []Plugin
+
+	// GetAllPluginsWithNames returns all of the known plugins with their names
+	GetAllPluginsWithNames() map[string]Plugin
 }
diff --git a/pkg/epp/plugins/registry.go b/pkg/epp/plugins/registry.go
new file mode 100644
index 0000000..d33ce99
--- /dev/null
+++ b/pkg/epp/plugins/registry.go
@@ -0,0 +1,33 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package plugins
+
+import (
+	"encoding/json"
+)
+
+// Factory is the definition of the factory functions that are used to instantiate plugins
+// specified in a configuration.
+type Factory func(name string, parameters json.RawMessage, handle Handle) (Plugin, error)
+
+// Register is a static function that can be called to register plugin factory functions.
+func Register(name string, factory Factory) {
+	Registry[name] = factory
+}
+
+// Registry is a mapping from plugin name to Factory function
+var Registry map[string]Factory = map[string]Factory{}
diff --git a/pkg/epp/requestcontrol/director.go b/pkg/epp/requestcontrol/director.go
index 122694b..5bc5ca4 100644
--- a/pkg/epp/requestcontrol/director.go
+++ b/pkg/epp/requestcontrol/director.go
@@ -22,6 +22,7 @@ import (
 	"context"
 	"fmt"
 	"math/rand"
+	"net"
 	"strconv"
 	"time"
 
@@ -40,7 +41,7 @@ import (
 
 // Scheduler defines the interface required by the Director for scheduling.
 type Scheduler interface {
-	Schedule(ctx context.Context, b *schedulingtypes.LLMRequest) (result map[string]*schedulingtypes.Result, err error)
+	Schedule(ctx context.Context, request *schedulingtypes.LLMRequest, candidatePods []schedulingtypes.Pod) (result *schedulingtypes.SchedulingResult, err error)
 }
 
 // SaturationDetector provides a signal indicating whether the backends are considered saturated.
@@ -48,10 +49,15 @@ type SaturationDetector interface {
 	IsSaturated(ctx context.Context) bool
 }
 
-// NewDirector creates a new Director instance with all dependencies.
-// postResponsePlugins remains nil as this is an optional field that can be set using the "WithPostResponsePlugins" function.
-func NewDirector(datastore datastore.Datastore, scheduler Scheduler, saturationDetector SaturationDetector) *Director {
-	return &Director{datastore: datastore, scheduler: scheduler, saturationDetector: saturationDetector}
+// NewDirectorWithConfig creates a new Director instance with all dependencies.
+func NewDirectorWithConfig(datastore datastore.Datastore, scheduler Scheduler, saturationDetector SaturationDetector, config *Config) *Director {
+	return &Director{
+		datastore:           datastore,
+		scheduler:           scheduler,
+		saturationDetector:  saturationDetector,
+		preRequestPlugins:   config.preRequestPlugins,
+		postResponsePlugins: config.postResponsePlugins,
+	}
 }
 
 // Director orchestrates the request handling flow, including scheduling.
@@ -59,21 +65,15 @@ type Director struct {
 	datastore           datastore.Datastore
 	scheduler           Scheduler
 	saturationDetector  SaturationDetector
+	preRequestPlugins   []PreRequest
 	postResponsePlugins []PostResponse
 }
 
-// WithPostResponsePlugins sets the given plugins as the PostResponse plugins.
-// If the Director has PostResponse plugins already, this call replaces the existing plugins with the given ones.
-func (d *Director) WithPostResponsePlugins(plugins ...PostResponse) *Director {
-	d.postResponsePlugins = plugins
-	return d
-}
-
 // HandleRequest orchestrates the request lifecycle:
 //  1. Parses request details.
-//  2. Calls PreDispatch for admission control.
-//  3. Calls Dispatch (which calls Scheduler) if request is approved.
-//  4. Calls PostDispatch to populate RequestContext with results.
+//  2. Calls admitRequest for admission control.
+//  3. Calls Scheduler.Schedule if request is approved.
+//  4. Calls prepareRequest to populate RequestContext with results and call PreRequest plugins.
 //
 // It always returns the requestContext even in the error case, as the request context is used in error handling.
 func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestContext) (*handlers.RequestContext, error) {
@@ -91,12 +91,16 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 		return reqCtx, err
 	}
 
-	// NOTE: The nil checking for the modelObject means that we DO allow passthrough currently.
-	// This might be a security risk in the future where adapters not registered in the InferenceModel
-	// are able to be requested by using their distinct name.
 	modelObj := d.datastore.ModelGet(reqCtx.Model)
 	if modelObj == nil {
-		return reqCtx, errutil.Error{Code: errutil.BadConfiguration, Msg: fmt.Sprintf("error finding a model object in InferenceModel for input %v", reqCtx.Model)}
+		logger.Info("No associated inferenceModel found, using default", "model", reqCtx.Model)
+		sheddable := v1alpha2.Sheddable
+		modelObj = &v1alpha2.InferenceModel{
+			Spec: v1alpha2.InferenceModelSpec{
+				ModelName:   reqCtx.Model,
+				Criticality: &sheddable,
+			},
+		}
 	}
 
 	reqCtx.ResolvedTargetModel = reqCtx.Model
@@ -120,42 +124,43 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 		Prompt:      prompt,
 		Headers:     reqCtx.Request.Headers,
 	}
-	logger = logger.WithValues(
-		"model", reqCtx.Model,
-		"resolvedTargetModel", reqCtx.ResolvedTargetModel,
-		"criticality", requestCriticality,
-	)
+
+	logger = logger.WithValues("model", reqCtx.Model, "resolvedTargetModel", reqCtx.ResolvedTargetModel, "criticality", requestCriticality)
 	ctx = log.IntoContext(ctx, logger)
 	logger.V(logutil.DEBUG).Info("LLM request assembled")
 
-	// --- 2. Saturation Check ---
-	preDispatchErr := d.PreDispatch(ctx, reqCtx, requestCriticality)
-	if preDispatchErr != nil {
-		return reqCtx, preDispatchErr
+	// --- 2. Admission Control check --
+	if err := d.admitRequest(ctx, requestCriticality); err != nil {
+		return reqCtx, err
 	}
 
-	// --- 3. Dispatch (Calls Scheduler) ---
-	results, dispatchErr := d.Dispatch(ctx, reqCtx.SchedulingRequest)
-	if dispatchErr != nil {
-		return reqCtx, dispatchErr
+	// --- 3. Call Scheduler ---
+	// Snapshot pod metrics from the datastore to:
+	// 1. Reduce concurrent access to the datastore.
+	// 2. Ensure consistent data during the scheduling operation of a request between all scheduling cycles.
+	candidatePods := schedulingtypes.ToSchedulerPodMetrics(d.datastore.PodGetAll())
+	results, err := d.scheduler.Schedule(ctx, reqCtx.SchedulingRequest, candidatePods)
+	if err != nil {
+		return reqCtx, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
 	}
 
-	// --- 4. PostDispatch (Populates RequestContext) ---
-	// Insert target endpoint to instruct Envoy to route requests to the specified target pod.
-	// Attach the port number.
-	reqCtx, postDispatchErr := d.PostDispatch(ctx, reqCtx, results)
-	if postDispatchErr != nil {
-		return reqCtx, postDispatchErr
+	// --- 4. Prepare Request (Populates RequestContext and call PreRequest plugins) ---
+	// Insert target endpoint to instruct Envoy to route requests to the specified target pod and attach the port number.
+	// Invoke PreRequest registered plugins.
+	reqCtx, err = d.prepareRequest(ctx, reqCtx, results)
+	if err != nil {
+		return reqCtx, err
 	}
 
 	return reqCtx, nil
 }
 
-// PreDispatch handles admission control before dispatch.
-func (d *Director) PreDispatch(ctx context.Context, reqCtx *handlers.RequestContext, reqCriticality v1alpha2.Criticality) error {
+// admitRequest handles admission control to decide whether or not to accept the request
+// based on the request criticality and system saturation state.
+func (d *Director) admitRequest(ctx context.Context, requestCriticality v1alpha2.Criticality) error {
 	logger := log.FromContext(ctx)
 
-	if reqCriticality == v1alpha2.Critical {
+	if requestCriticality == v1alpha2.Critical {
 		logger.V(logutil.DEBUG).Info("Critical request bypassing saturation check.")
 		return nil
 	}
@@ -167,44 +172,34 @@ func (d *Director) PreDispatch(ctx context.Context, reqCtx *handlers.RequestCont
 			Msg:  "system saturated, non-critical request dropped",
 		}
 	}
-	return nil
-}
 
-// Dispatch runs one or many scheduling cycles.
-func (d *Director) Dispatch(ctx context.Context, llmReq *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
-	var err error
-	res, err := d.scheduler.Schedule(ctx, llmReq)
-	if err != nil {
-		return nil, errutil.Error{Code: errutil.InferencePoolResourceExhausted, Msg: fmt.Errorf("failed to find target pod: %w", err).Error()}
-	}
-
-	return res, nil // TODO handle multi cycle result after defining the PostDispatch extension point
+	return nil
 }
 
-// PostDispatch populates the RequestContext based on scheduling results.
-func (d *Director) PostDispatch(ctx context.Context, reqCtx *handlers.RequestContext, results map[string]*schedulingtypes.Result) (*handlers.RequestContext, error) {
+// prepareRequest populates the RequestContext and calls the registered PreRequest plugins
+// for allowing plugging customized logic based on the scheduling results.
+func (d *Director) prepareRequest(ctx context.Context, reqCtx *handlers.RequestContext, result *schedulingtypes.SchedulingResult) (*handlers.RequestContext, error) {
 	logger := log.FromContext(ctx)
-	// currently only get a single result. Will refactor to pluggably implement the PostSchedule
-	if len(results) == 0 {
+	if result == nil || len(result.ProfileResults) == 0 {
 		return reqCtx, errutil.Error{Code: errutil.Internal, Msg: "results must be greater than zero"}
 	}
-	var targetPod *backend.Pod
-	// TODO should handle multi cycle results, this should be pluggable logic
-	for _, result := range results {
-		targetPod = result.TargetPod.GetPod()
-	}
+	// primary profile is used to set destination
+	targetPod := result.ProfileResults[result.PrimaryProfileName].TargetPod.GetPod()
 
 	pool, err := d.datastore.PoolGet()
 	if err != nil {
 		return reqCtx, err
 	}
+	targetPort := int(pool.Spec.TargetPortNumber)
 
-	endpoint := targetPod.Address + ":" + strconv.Itoa(int(pool.Spec.TargetPortNumber))
+	endpoint := net.JoinHostPort(targetPod.Address, strconv.Itoa(targetPort))
 	logger.V(logutil.DEFAULT).Info("Request handled", "model", reqCtx.Model, "targetModel", reqCtx.ResolvedTargetModel, "endpoint", targetPod)
 
 	reqCtx.TargetPod = targetPod
 	reqCtx.TargetEndpoint = endpoint
 
+	d.runPreRequestPlugins(ctx, reqCtx.SchedulingRequest, result, targetPort)
+
 	return reqCtx, nil
 }
 
@@ -260,11 +255,21 @@ func RandomWeightedDraw(logger logr.Logger, model *v1alpha2.InferenceModel, seed
 	return ""
 }
 
+func (d *Director) runPreRequestPlugins(ctx context.Context, request *schedulingtypes.LLMRequest, schedulingResult *schedulingtypes.SchedulingResult,
+	targetPort int) {
+	for _, plugin := range d.preRequestPlugins {
+		log.FromContext(ctx).V(logutil.DEBUG).Info("Running pre-request plugin", "plugin", plugin.Type())
+		before := time.Now()
+		plugin.PreRequest(ctx, request, schedulingResult, targetPort)
+		metrics.RecordRequestControlPluginProcessingLatency(PreRequestPluginType, plugin.Type(), time.Since(before))
+	}
+}
+
 func (d *Director) runPostResponsePlugins(ctx context.Context, request *schedulingtypes.LLMRequest, response *Response, targetPod *backend.Pod) {
 	for _, plugin := range d.postResponsePlugins {
-		log.FromContext(ctx).V(logutil.DEBUG).Info("Running post-response plugin", "plugin", plugin.Name())
+		log.FromContext(ctx).V(logutil.DEBUG).Info("Running post-response plugin", "plugin", plugin.Type())
 		before := time.Now()
 		plugin.PostResponse(ctx, request, response, targetPod)
-		metrics.RecordRequestControlPluginProcessingLatency(PostResponsePluginType, plugin.Name(), time.Since(before))
+		metrics.RecordRequestControlPluginProcessingLatency(PostResponsePluginType, plugin.Type(), time.Since(before))
 	}
 }
diff --git a/pkg/epp/requestcontrol/plugins.go b/pkg/epp/requestcontrol/plugins.go
index 89dd9ad..ba51c2a 100644
--- a/pkg/epp/requestcontrol/plugins.go
+++ b/pkg/epp/requestcontrol/plugins.go
@@ -25,9 +25,17 @@ import (
 )
 
 const (
+	PreRequestPluginType   = "PreRequest"
 	PostResponsePluginType = "PostResponse"
 )
 
+// PreRequest is called by the director after a getting result from scheduling layer and
+// before a request is sent to the selected model server.
+type PreRequest interface {
+	plugins.Plugin
+	PreRequest(ctx context.Context, request *types.LLMRequest, schedulingResult *types.SchedulingResult, targetPort int)
+}
+
 // PostResponse is called by the director after a successful response was sent.
 // The given pod argument is the pod that served the request.
 type PostResponse interface {
diff --git a/pkg/epp/requestcontrol/request_control_config.go b/pkg/epp/requestcontrol/request_control_config.go
new file mode 100644
index 0000000..d658cde
--- /dev/null
+++ b/pkg/epp/requestcontrol/request_control_config.go
@@ -0,0 +1,69 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package requestcontrol
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+)
+
+// NewConfig creates a new Config object and returns its pointer.
+func NewConfig() *Config {
+	return &Config{
+		preRequestPlugins:   []PreRequest{},
+		postResponsePlugins: []PostResponse{},
+	}
+}
+
+// Config provides a configuration for the requestcontrol plugins.
+type Config struct {
+	preRequestPlugins   []PreRequest
+	postResponsePlugins []PostResponse
+}
+
+// WithPreRequestPlugins sets the given plugins as the PreRequest plugins.
+// If the Config has PreRequest plugins already, this call replaces the existing plugins with the given ones.
+func (c *Config) WithPreRequestPlugins(plugins ...PreRequest) *Config {
+	c.preRequestPlugins = plugins
+	return c
+}
+
+// WithPostResponsePlugins sets the given plugins as the PostResponse plugins.
+// If the Config has PostResponse plugins already, this call replaces the existing plugins with the given ones.
+func (c *Config) WithPostResponsePlugins(plugins ...PostResponse) *Config {
+	c.postResponsePlugins = plugins
+	return c
+}
+
+func (c *Config) AddPlugins(pluginObjects ...plugins.Plugin) {
+	for _, plugin := range pluginObjects {
+		if preRequestPlugin, ok := plugin.(PreRequest); ok {
+			c.preRequestPlugins = append(c.preRequestPlugins, preRequestPlugin)
+		}
+		if postResponsePlugin, ok := plugin.(PostResponse); ok {
+			c.postResponsePlugins = append(c.postResponsePlugins, postResponsePlugin)
+		}
+	}
+}
+
+func LoadRequestControlConfig(instantiatedPlugins map[string]plugins.Plugin) *Config {
+	config := NewConfig()
+	for _, plugin := range instantiatedPlugins {
+		config.AddPlugins(plugin)
+	}
+
+	return config
+}
diff --git a/pkg/epp/scheduling/config/config.go b/pkg/epp/scheduling/config/config.go
index 80efaaa..e7fd0a3 100644
--- a/pkg/epp/scheduling/config/config.go
+++ b/pkg/epp/scheduling/config/config.go
@@ -33,8 +33,8 @@ type Config struct {
 
 const (
 	// Default values for LoRA specific thresholds
-	defaultQueueingThresholdLoRA = 128
-	defaultLoraAffinityThreshold = 0.999
+	DefaultQueueingThresholdLoRA = 128
+	DefaultLoraAffinityThreshold = 0.999
 )
 
 // LoadConfig loads configuration from environment variables
@@ -45,8 +45,8 @@ func LoadConfig() Config {
 	config := Config{
 		KVCacheThreshold:       envutil.GetEnvFloat("KV_CACHE_THRESHOLD", commonconfig.DefaultKVCacheThreshold, baseLogger),
 		QueueThresholdCritical: envutil.GetEnvInt("QUEUE_THRESHOLD_CRITICAL", commonconfig.DefaultQueueThresholdCritical, baseLogger),
-		QueueingThresholdLoRA:  envutil.GetEnvInt("QUEUING_THRESHOLD_LORA", defaultQueueingThresholdLoRA, baseLogger),
-		LoraAffinityThreshold:  envutil.GetEnvFloat("LORA_AFFINITY_THRESHOLD", defaultLoraAffinityThreshold, baseLogger),
+		QueueingThresholdLoRA:  envutil.GetEnvInt("QUEUING_THRESHOLD_LORA", DefaultQueueingThresholdLoRA, baseLogger),
+		LoraAffinityThreshold:  envutil.GetEnvFloat("LORA_AFFINITY_THRESHOLD", DefaultLoraAffinityThreshold, baseLogger),
 	}
 
 	baseLogger.V(logutil.DEFAULT).Info("Scheduler configuration loaded", "config", config)
diff --git a/pkg/epp/scheduling/framework/plugins.go b/pkg/epp/scheduling/framework/plugins.go
index 56160cd..7e22d86 100644
--- a/pkg/epp/scheduling/framework/plugins.go
+++ b/pkg/epp/scheduling/framework/plugins.go
@@ -24,41 +24,53 @@ import (
 )
 
 const (
-	ProfilePickerType   = "ProfilePicker"
-	FilterPluginType    = "Filter"
-	ScorerPluginType    = "Scorer"
-	PickerPluginType    = "Picker"
-	PostCyclePluginType = "PostCycle"
+	ProfilePickerType          = "ProfilePicker"
+	FilterPluginType           = "Filter"
+	ScorerPluginType           = "Scorer"
+	PickerPluginType           = "Picker"
+	PostCyclePluginType        = "PostCycle"
+	ProcessProfilesResultsType = "ProcessProfilesResults"
 )
 
-// ProfilePicker selects the SchedulingProfiles to run from a list of candidate profiles, while taking into consideration the request properties
-// and the previously executed SchedluderProfile cycles along with their results.
-type ProfilePicker interface {
+// ProfileHandler defines the extension points for handling multi SchedulerProfile instances.
+// More specifically, this interface defines the 'Pick' and 'ProcessResults' extension points.
+type ProfileHandler interface {
 	plugins.Plugin
-	Pick(ctx context.Context, request *types.LLMRequest, profiles map[string]*SchedulerProfile, executionResults map[string]*types.Result) map[string]*SchedulerProfile
+	// Pick selects the SchedulingProfiles to run from a list of candidate profiles, while taking into consideration the request properties
+	// and the previously executed SchedluderProfile cycles along with their results.
+	Pick(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest, profiles map[string]*SchedulerProfile,
+		profileResults map[string]*types.ProfileRunResult) map[string]*SchedulerProfile
+
+	// ProcessResults handles the outcome of the profile runs after all profiles ran.
+	// It may aggregate results, log test profile outputs, or apply custom logic. It specifies in the SchedulingResult the
+	// key of the primary profile that should be used to get the request selected destination.
+	// When a profile run fails, its result in the profileResults map is nil.
+	ProcessResults(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest,
+		profileResults map[string]*types.ProfileRunResult) (*types.SchedulingResult, error)
 }
 
 // Filter defines the interface for filtering a list of pods based on context.
 type Filter interface {
 	plugins.Plugin
-	Filter(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, pods []types.Pod) []types.Pod
+	Filter(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest, pods []types.Pod) []types.Pod
 }
 
 // Scorer defines the interface for scoring a list of pods based on context.
 // Scorers must score pods with a value within the range of [0,1] where 1 is the highest score.
 type Scorer interface {
 	plugins.Plugin
-	Score(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, pods []types.Pod) map[types.Pod]float64
+	Score(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest, pods []types.Pod) map[types.Pod]float64
 }
 
 // Picker picks the final pod(s) to send the request to.
 type Picker interface {
 	plugins.Plugin
-	Pick(ctx context.Context, cycleState *types.CycleState, scoredPods []*types.ScoredPod) *types.Result
+	Pick(ctx context.Context, cycleState *types.CycleState, scoredPods []*types.ScoredPod) *types.ProfileRunResult
 }
 
 // PostCycle is called by the scheduler after it selects a targetPod for the request in the SchedulerProfile cycle.
+// DEPRECATED - do not use PostCycle. this is in the process of deprecation.
 type PostCycle interface {
 	plugins.Plugin
-	PostCycle(ctx context.Context, cycleState *types.CycleState, res *types.Result)
+	PostCycle(ctx context.Context, cycleState *types.CycleState, res *types.ProfileRunResult)
 }
diff --git a/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go b/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
index 194c17b..3f7c88f 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/decision_tree_filter.go
@@ -47,18 +47,18 @@ type DecisionTreeFilter struct {
 	NextOnSuccessOrFailure framework.Filter
 }
 
-// Name returns the name of the filter.
-func (f *DecisionTreeFilter) Name() string {
+// Type returns the type of the filter.
+func (f *DecisionTreeFilter) Type() string {
 	if f == nil {
 		return "nil"
 	}
-	return f.Current.Name()
+	return f.Current.Type()
 }
 
 // Filter filters out pods that doesn't meet the filter criteria.
-func (f *DecisionTreeFilter) Filter(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *DecisionTreeFilter) Filter(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest, pods []types.Pod) []types.Pod {
 	loggerTrace := log.FromContext(ctx).V(logutil.TRACE)
-	filteredPod := f.Current.Filter(ctx, request, cycleState, pods)
+	filteredPod := f.Current.Filter(ctx, cycleState, request, pods)
 
 	next := f.NextOnSuccessOrFailure
 	if len(filteredPod) > 0 {
@@ -69,9 +69,9 @@ func (f *DecisionTreeFilter) Filter(ctx context.Context, request *types.LLMReque
 		if f.NextOnSuccess != nil {
 			next = f.NextOnSuccess
 		}
-		loggerTrace.Info("Filter succeeded", "filter", f.Name(), "next", next.Name(), "filteredPodCount", len(filteredPod))
+		loggerTrace.Info("Filter succeeded", "filter", f.Type(), "next", next.Type(), "filteredPodCount", len(filteredPod))
 		// On success, pass the filtered result to the next filter.
-		return next.Filter(ctx, request, cycleState, filteredPod)
+		return next.Filter(ctx, cycleState, request, filteredPod)
 	} else {
 		if f.NextOnFailure == nil && f.NextOnSuccessOrFailure == nil {
 			// No succeeding filters to run, return.
@@ -80,8 +80,8 @@ func (f *DecisionTreeFilter) Filter(ctx context.Context, request *types.LLMReque
 		if f.NextOnFailure != nil {
 			next = f.NextOnFailure
 		}
-		loggerTrace.Info("Filter failed", "filter", f.Name(), "next", next.Name())
+		loggerTrace.Info("Filter failed", "filter", f.Type(), "next", next.Type())
 		// On failure, pass the initial set of pods to the next filter.
-		return next.Filter(ctx, request, cycleState, pods)
+		return next.Filter(ctx, cycleState, request, pods)
 	}
 }
diff --git a/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go b/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
index fe18395..0b4c1eb 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/least_kvcache_filter.go
@@ -18,15 +18,26 @@ package filter
 
 import (
 	"context"
+	"encoding/json"
 	"math"
 
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type assertion
+const (
+	LeastKVCacheFilterType = "least-KV-cache"
+)
+
+// compile-time type validation
 var _ framework.Filter = &LeastKVCacheFilter{}
 
+// LeastKVCacheFilterFactory defines the factory function for LeastKVCacheFilter.
+func LeastKVCacheFilterFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewLeastKVCacheFilter(), nil
+}
+
 // NewLeastKVCacheFilter initializes a new LeastKVCacheFilter and returns its pointer.
 func NewLeastKVCacheFilter() *LeastKVCacheFilter {
 	return &LeastKVCacheFilter{}
@@ -39,13 +50,13 @@ func NewLeastKVCacheFilter() *LeastKVCacheFilter {
 // least one as it gives more choices for the next filter, which on aggregate gave better results.
 type LeastKVCacheFilter struct{}
 
-// Name returns the name of the filter.
-func (f *LeastKVCacheFilter) Name() string {
-	return "least-KV-cache"
+// Type returns the type of the filter.
+func (f *LeastKVCacheFilter) Type() string {
+	return LeastKVCacheFilterType
 }
 
 // Filter filters out pods that doesn't meet the filter criteria.
-func (f *LeastKVCacheFilter) Filter(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *LeastKVCacheFilter) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
 	filteredPods := []types.Pod{}
 
 	min := math.MaxFloat64
diff --git a/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go b/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
index 8a43ce7..c43d4b3 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/least_queue_filter.go
@@ -18,15 +18,26 @@ package filter
 
 import (
 	"context"
+	"encoding/json"
 	"math"
 
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type assertion
+const (
+	LeastQueueFilterType = "least-queue"
+)
+
+// compile-time type validation
 var _ framework.Filter = &LeastQueueFilter{}
 
+// LeastQueueFilterFactory defines the factory function for LeastQueueFilter.
+func LeastQueueFilterFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewLeastQueueFilter(), nil
+}
+
 // NewLeastQueueFilter initializes a new LeastQueueFilter and returns its pointer.
 func NewLeastQueueFilter() *LeastQueueFilter {
 	return &LeastQueueFilter{}
@@ -39,13 +50,13 @@ func NewLeastQueueFilter() *LeastQueueFilter {
 // the least one as it gives more choices for the next filter, which on aggregate gave better results.
 type LeastQueueFilter struct{}
 
-// Name returns the name of the filter.
-func (f *LeastQueueFilter) Name() string {
-	return "least-queue"
+// Type returns the type of the filter.
+func (f *LeastQueueFilter) Type() string {
+	return LeastQueueFilterType
 }
 
 // Filter filters out pods that doesn't meet the filter criteria.
-func (f *LeastQueueFilter) Filter(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *LeastQueueFilter) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
 	filteredPods := []types.Pod{}
 
 	min := math.MaxInt
diff --git a/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go b/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
index 04c0107..2150cdb 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/lora_affinity_filter.go
@@ -18,21 +18,41 @@ package filter
 
 import (
 	"context"
+	"encoding/json"
+	"fmt"
 	"math/rand"
 	"time"
 
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type assertion
+const (
+	LoraAffinityFilterType = "lora-affinity"
+)
+
+type loraAffinityFilterParameters struct {
+	Threshold float64 `json:"threshold"`
+}
+
+// compile-time type validation
 var _ framework.Filter = &LoraAffinityFilter{}
 
+// LoraAffinityFilterFactory defines the factory function for LoraAffinityFilter.
+func LoraAffinityFilterFactory(name string, rawParameters json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	parameters := loraAffinityFilterParameters{Threshold: config.DefaultLoraAffinityThreshold}
+	if err := json.Unmarshal(rawParameters, &parameters); err != nil {
+		return nil, fmt.Errorf("failed to parse the parameters of the '%s' filter - %w", LoraAffinityFilterType, err)
+	}
+	return NewLoraAffinityFilter(parameters.Threshold), nil
+}
+
 // NewLoraAffinityFilter initializes a new LoraAffinityFilter and returns its pointer.
-func NewLoraAffinityFilter() *LoraAffinityFilter {
+func NewLoraAffinityFilter(threshold float64) *LoraAffinityFilter {
 	return &LoraAffinityFilter{
-		loraAffinityThreshold: config.Conf.LoraAffinityThreshold,
+		loraAffinityThreshold: threshold,
 	}
 }
 
@@ -47,13 +67,13 @@ type LoraAffinityFilter struct {
 	loraAffinityThreshold float64
 }
 
-// Name returns the name of the filter.
-func (f *LoraAffinityFilter) Name() string {
-	return "lora-affinity"
+// Type returns the type of the filter.
+func (f *LoraAffinityFilter) Type() string {
+	return LoraAffinityFilterType
 }
 
 // Filter filters out pods that doesn't meet the filter criteria.
-func (f *LoraAffinityFilter) Filter(_ context.Context, request *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *LoraAffinityFilter) Filter(_ context.Context, _ *types.CycleState, request *types.LLMRequest, pods []types.Pod) []types.Pod {
 	// Pre-allocate slices with estimated capacity
 	filtered_affinity := make([]types.Pod, 0, len(pods))
 	filtered_available := make([]types.Pod, 0, len(pods))
diff --git a/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go b/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
index 0192c32..2d8aa3d 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/low_queue_filter.go
@@ -18,19 +18,41 @@ package filter
 
 import (
 	"context"
+	"fmt"
 
+	"encoding/json"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
-// compile-time type assertion
+const (
+	LowQueueFilterType = "low-queue"
+)
+
+type lowQueueFilterParameters struct {
+	Threshold int `json:"threshold"`
+}
+
+// compile-time type validation
 var _ framework.Filter = &LowQueueFilter{}
 
+// LowQueueFilterFactory defines the factory function for LowQueueFilter.
+func LowQueueFilterFactory(name string, rawParameters json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	parameters := lowQueueFilterParameters{Threshold: config.DefaultQueueingThresholdLoRA}
+	if err := json.Unmarshal(rawParameters, &parameters); err != nil {
+		return nil, fmt.Errorf("failed to parse the parameters of the '%s' filter - %w", LowQueueFilterType, err)
+	}
+
+	return NewLowQueueFilter(parameters.Threshold), nil
+}
+
 // NewLowQueueFilter initializes a new LowQueueFilter and returns its pointer.
-func NewLowQueueFilter() *LowQueueFilter {
+func NewLowQueueFilter(threshold int) *LowQueueFilter {
 	return &LowQueueFilter{
-		queueingThresholdLoRA: config.Conf.QueueingThresholdLoRA,
+		queueingThresholdLoRA: threshold,
 	}
 }
 
@@ -39,13 +61,13 @@ type LowQueueFilter struct {
 	queueingThresholdLoRA int
 }
 
-// Name returns the name of the filter.
-func (f *LowQueueFilter) Name() string {
-	return "low-queue"
+// Type returns the type of the filter.
+func (f *LowQueueFilter) Type() string {
+	return LowQueueFilterType
 }
 
 // Filter filters out pods that doesn't meet the filter criteria.
-func (f *LowQueueFilter) Filter(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *LowQueueFilter) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
 	filteredPods := []types.Pod{}
 
 	for _, pod := range pods {
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go
index 4859357..716c9f2 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer.go
@@ -20,154 +20,130 @@ import (
 	"context"
 	"sync"
 	"time"
-	"unsafe"
-
-	"container/list"
 
+	lru "github.com/hashicorp/golang-lru/v2"
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-func newIndexer(maxCacheSize int) *indexer {
-	t := &indexer{
-		maxCacheSize: maxCacheSize,
-		table:        make(map[BlockHash]map[ServerID]*list.Element),
-		ll:           list.New(),
-	}
-	go t.ReportCacheSize(time.Second)
-	return t
-}
-
 // An indexer maintains an LRU cache of prompt prefix hashes and the server(s) that might have that
-// prefix cached .
+// prefix cached.
 type indexer struct {
-	mu           sync.RWMutex
-	maxCacheSize int
-	table        map[BlockHash]map[ServerID]*list.Element // from any prefix cache to the cache entry to find the server
-	ll           *list.List                               // LinkedList to keep track of the order of entries
+	mu         sync.RWMutex
+	hashToPods map[BlockHash]podSet                         // the lookup data structure to find pods that have the BlockHash cached
+	podToLRU   map[ServerID]*lru.Cache[BlockHash, struct{}] // key is pod namespacedName, value is an LRU cache
+	maxLRUSize int
 }
 
-// value is the value stored in the linked list.
-type value struct {
-	server ServerID
-	hash   BlockHash
-}
-
-// Get returns the set of servers that have the given prefix hash cached.
-func (i *indexer) Get(hash BlockHash) map[ServerID]bool {
-	i.mu.RLock()
-	defer i.mu.RUnlock()
-	res := map[ServerID]bool{}
-	for server := range i.table[hash] {
-		res[server] = true
+// newIndexer initializes an indexer with size limits and starts cache size reporting.
+func newIndexer(maxLRUSize int) *indexer {
+	ix := &indexer{
+		hashToPods: make(map[BlockHash]podSet),
+		podToLRU:   make(map[ServerID]*lru.Cache[BlockHash, struct{}]),
+		maxLRUSize: maxLRUSize,
 	}
-	return res
+
+	go ix.ReportLRUSize(time.Second)
+	return ix
 }
 
-// Add adds a list of prefix hashes of a single request to the server the request was sent to.
-// The intuition is that this server is likely to have the prefix cached, so next time a request
-// sharing the longest prefix should be sent to the same server to take advantage of the cache hit.
-func (i *indexer) Add(hashes []BlockHash, server ServerID) {
+// Add adds a list of prefix hashes to the cache, tied to the server.
+func (i *indexer) Add(hashes []BlockHash, pod ServerID) {
 	i.mu.Lock()
-	defer i.mu.Unlock()
-	for _, hash := range hashes {
-		i.add(hash, server)
+	// Check if the LRU pod exist
+	lruForPod, exists := i.podToLRU[pod]
+	if !exists {
+		newLRU, _ := lru.NewWithEvict[BlockHash, struct{}](i.maxLRUSize, i.makeEvictionFn(pod))
+		i.podToLRU[pod] = newLRU
+		lruForPod = newLRU
 	}
-}
 
-func (i *indexer) check(hash BlockHash, server ServerID) (*list.Element, bool) {
-	servers, ok := i.table[hash]
-	if !ok {
-		return nil, false
+	i.mu.Unlock()
+
+	// Add to LRU (may evict)
+	for _, hash := range hashes {
+		lruForPod.Add(hash, struct{}{})
 	}
-	e, ok := servers[server]
-	return e, ok
-}
 
-func (i *indexer) add(hash BlockHash, server ServerID) {
-	e, exists := i.check(hash, server)
-	if exists {
-		i.ll.MoveToBack(e)
-	} else {
-		i.create(hash, server)
+	// Update hashToPods once under lock
+	i.mu.Lock()
+	for _, hash := range hashes {
+		pods := i.hashToPods[hash]
+		if pods == nil {
+			pods = make(podSet)
+		}
+		pods[pod] = struct{}{}
+		i.hashToPods[hash] = pods
 	}
+
+	i.mu.Unlock()
 }
 
-func (i *indexer) create(hash BlockHash, server ServerID) {
-	for i.ll.Len() >= i.maxCacheSize {
-		// Evict the least recently used entry if we've exceeded the max cache size
-		i.evict()
-	}
+// Get returns a set of servers that have the given prefix hash cached.
+func (i *indexer) Get(hash BlockHash) podSet {
+	i.mu.RLock()
+	defer i.mu.RUnlock()
 
-	if _, ok := i.table[hash]; !ok {
-		i.table[hash] = make(map[ServerID]*list.Element)
-	}
-	v := &value{
-		server: server,
-		hash:   hash,
+	res := podSet{}
+	pods, ok := i.hashToPods[hash]
+	if !ok {
+		return res
 	}
-	e := i.ll.PushBack(v)
-	i.table[hash][server] = e
+
+	return pods
 }
 
-// evict removes the least recently used entry from the cache
-func (i *indexer) evict() {
-	oldestNode := i.ll.Front()
-	if oldestNode == nil {
-		return
+// makeEvictionFn returns a per-pod LRU eviction callback that removes the pod from hashToPods on eviction.
+func (i *indexer) makeEvictionFn(pod ServerID) func(BlockHash, struct{}) {
+	return func(hash BlockHash, _ struct{}) {
+		i.mu.Lock()
+		defer i.mu.Unlock()
+		// Remove the pod from the hash→pods map
+		if podSet, ok := i.hashToPods[hash]; ok {
+			delete(podSet, pod)
+			if len(podSet) == 0 {
+				delete(i.hashToPods, hash)
+			}
+		}
 	}
-	i.ll.Remove(oldestNode)
-
-	v := oldestNode.Value.(*value)
-	hash := v.hash
-	server := v.server
-	// Remove from the hash map
-	serverMap := i.table[hash]
-	delete(serverMap, server)
-
-	// If this was the last server for this hash, remove the hash entry entirely
-	if len(serverMap) == 0 {
-		delete(i.table, hash)
-	}
-
-	log.FromContext(context.TODO()).V(logutil.TRACE).Info("Evicted LRU entry", "hash", hash, "server", server)
 }
 
-// ReportCacheSize starts a goroutine that periodically reports the cache size metric
-func (i *indexer) ReportCacheSize(interval time.Duration) {
+// ReportLRUSize starts a goroutine that periodically reports the LRU cache size metric.
+func (i *indexer) ReportLRUSize(interval time.Duration) {
 	ticker := time.NewTicker(interval)
 	defer ticker.Stop()
 	for range ticker.C {
 		i.mu.RLock()
-		metrics.RecordPrefixCacheSize(int64(i.ll.Len()))
-		log.FromContext(context.TODO()).V(logutil.TRACE).Info("LRU", "# entries", i.ll.Len(), "estimated size MB", i.ll.Len()*i.estimateEntrySize()/1000000)
+		totalEntries := 0
+		maxPodEntries := 0
+		maxPodName := ServerID{}
+
+		for pod, lruCache := range i.podToLRU {
+			size := lruCache.Len()
+			totalEntries += size
+			if size > maxPodEntries {
+				maxPodEntries = size
+				maxPodName = pod
+			}
+		}
+
+		numPods := len(i.podToLRU)
+		avg := 0.0
+		if numPods > 0 {
+			avg = float64(totalEntries) / float64(numPods)
+		}
+
+		metrics.RecordPrefixCacheSize(int64(totalEntries))
+		log.FromContext(context.TODO()).V(logutil.TRACE).Info("Prefix cache state",
+			"total entries", totalEntries,
+			"# pods", numPods,
+			"avg entries per pod", avg,
+			"pod with max cache", maxPodName,
+			"max pod size", maxPodEntries,
+			"global max LRU cache capacity per pod", i.maxLRUSize,
+		)
+
 		i.mu.RUnlock()
 	}
 }
-
-// estimateEntrySize estimates the memory size of a cache entry in bytes.
-func (i *indexer) estimateEntrySize() int {
-	size := 0
-
-	// Estimate the size of a node in the linked list.
-	// First get the size of the node struct via unsafe.Sizeof.
-	// The prev and next pointers are 8 bytes each on a 64-bit system.
-	// The BlockHash is a uint64, which is 8 bytes.
-	// The ServerID is a NamespacedName, which contains two strings (Name and Namespace).
-	// The headers for the strings are 16 bytes each (8 bytes for the pointer and 8 bytes for the length).
-	// So unsafe.Sizeof(node{}) should return 2*8 + 8 + 2*16 = 48 bytes.
-	size += int(unsafe.Sizeof(value{}))
-	// Size of the Name and Namespace strings in ServerID, assuming 63 bytes each (max length for Kubernetes NamespacedName).
-	size += 2 * 63
-
-	// Estimate the size of an entry in the hash map. Note the overhead of the map headers and buckets are ignored.
-	size += 8      // Size of the BlockHash (uint64).
-	size += 2 * 16 // Size of the ServerID string headers (NamespacedName).
-	size += 2 * 63 // Size of the Name and Namespace strings in ServerID.
-	size += 8      // Size of the pointer to the node in the hash map.
-
-	// Based on the above estimates, the estimated size of an entry is:
-	// (48 + 2*63) + (8 + 2*16 + 2*63 + 8) = 348 bytes.
-	return size
-}
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
index 4e95fce..f79122a 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
@@ -19,12 +19,14 @@ package prefix
 import (
 	"context"
 	"encoding/binary"
+	"encoding/json"
 	"fmt"
 
 	"github.com/cespare/xxhash/v2"
 	k8stypes "k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
@@ -32,11 +34,6 @@ import (
 
 const (
 	DefaultScorerWeight = 1
-	// Attempt to return DefaultNumServersToMatch servers with their longest prefix match length.
-	// Why not just return the server with longest prefix match?
-	// It may not be the optimal choice, e.g., it may have a high queue depth.
-	// We optimistically search more than one to give more candidates for the scheduler to choose.
-	DefaultNumServersToMatch = 2
 	// vLLM default token block size is 16, and a good guess of average characters per token is 4.
 	DefaultHashBlockSize = 64
 	// The maximum number of blocks to match. Two long requests with the same prefix up to this
@@ -44,31 +41,29 @@ const (
 	// This parameter provides a trade-off between cache size, prefix matching speed and matching
 	// accuracy. Use a small value if most requests are short to reduce cache size and speed up the
 	// matching process. Use a large value if most requests are long to increase the matching accuracy.
-	DefaultMaxPrefixBlocks = 128
-	// The indexer is an approximation to the actual prefix cache state on the model servers.
+	DefaultMaxPrefixBlocks = 256
+	// The indexer is an approximation to the actual prefix LRU cache state on the model servers per server (pod).
 	// A small capacity ensures a high accuracy of cache hit on the model server, but it will
 	// increase the chance of false negatives. A high capacity does the opposite.
 	// To properly size this, consider the sum of the total number of cache entries on all model
-	// servers. Consider the llama3 8B model on 3 H100 80GB GPUs. The size of the model weight is
-	// about 16GB. Assume 50% of the remaining HBM is used for caching prefixes, we have 32GB. Each
-	// token is about 128KB in size, so we can cache 250K tokens. Using the default block size of 16
-	// in vLLM, we will have 250K / 16 = 15.6K blocks. In total we have 15.6K * 3 = 46.8K blocks, or
-	// roughly 50K.
-	// How much memory space does it require to hold the 50K block hashes?
-	// According to the estimates in indexer.estimateEntrySize(), the size of each entry is
-	// approximately 348 bytes. So in total we have 50K * 348 = 17.4MB.
-	DefaultLRUIndexerCapacity = 50000
+	// servers. Consider the llama3 8B model on a H100 80GB GPUs. The size of the model weight is
+	// about 16GB. The remaining HBM used for caching prefixes is 64GB. Each
+	// token is about 128KB in size, so we can cache 500K tokens. Using the default block size of 16
+	// in vLLM, we will have 250K / 16 = 31.25K blocks.
+	DefaultLRUCapacityPerServer = 31250
+
+	PrefixCachePluginType = "prefix-cache"
 )
 
 type Config struct {
 	// The input prompt is broken into sizes of HashBlockSize to calculate block hashes . Requests
 	// with length shorter than the block size will be ignored.
-	HashBlockSize int
+	HashBlockSize int `json:"hashBlockSize"`
 	// MaxPrefixBlocksToMatch is the maximum number of prefix blocks to match. Input beyond this limit will
 	// be ignored.
-	MaxPrefixBlocksToMatch int
-	// Max (approximate) size of the LRU indexer in number of entries.
-	LRUIndexerCapacity int
+	MaxPrefixBlocksToMatch int `json:"maxPrefixBlocksToMatch"`
+	// Max capacity size of the LRU indexer in number of entries per server (pod).
+	LRUCapacityPerServer int `json:"lruCapacityPerServer"`
 }
 
 type Plugin struct {
@@ -76,8 +71,11 @@ type Plugin struct {
 	indexer Indexer
 }
 
+// podSet holds an pods servers that may have a specific prefix hash.
+type podSet map[ServerID]struct{}
+
 type Indexer interface {
-	Get(hash BlockHash) map[ServerID]bool
+	Get(hash BlockHash) podSet
 	Add(hashes []BlockHash, server ServerID)
 }
 
@@ -90,7 +88,7 @@ func (s ServerID) String() string {
 	return k8stypes.NamespacedName(s).String()
 }
 
-// compile-time type assertion
+// compile-time type validation
 var _ types.StateData = &schedulingContextState{}
 
 // This is the state of this plugin to be used during a scheduling cycle.
@@ -119,34 +117,57 @@ func (s *schedulingContextState) Clone() types.StateData {
 var _ framework.Scorer = &Plugin{}
 var _ framework.PostCycle = &Plugin{}
 
+// PrefixCachePluginFactory defines the factory function for Prefix plugin.
+func PrefixCachePluginFactory(name string, rawParameters json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	parameters := Config{
+		HashBlockSize:          DefaultHashBlockSize,
+		MaxPrefixBlocksToMatch: DefaultMaxPrefixBlocks,
+		LRUCapacityPerServer:   DefaultLRUCapacityPerServer,
+	}
+	if err := json.Unmarshal(rawParameters, &parameters); err != nil {
+		return nil, fmt.Errorf("failed to parse the parameters of the %s plugin. Error: %s", PrefixCachePluginType, err)
+	}
+
+	return &Plugin{
+		Config:  parameters,
+		indexer: newIndexer(parameters.LRUCapacityPerServer),
+	}, nil
+}
+
 // New initializes a new prefix Plugin and returns its pointer.
 func New(config Config) *Plugin {
+	capacity := config.LRUCapacityPerServer
+	if capacity <= 0 {
+		capacity = DefaultLRUCapacityPerServer
+		log.FromContext(context.TODO()).V(logutil.DEFAULT).Info(
+			"LRUCapacityPerServer is not positive, using default value",
+			"defaultCapacity", DefaultLRUCapacityPerServer,
+		)
+	}
+
 	m := &Plugin{
 		Config:  config,
-		indexer: newIndexer(config.LRUIndexerCapacity),
+		indexer: newIndexer(capacity),
 	}
 	return m
 }
 
-// Name returns the name of the plugin.
-func (m *Plugin) Name() string {
-	return "prefix-cache"
+// Type returns the type of the plugin.
+func (m *Plugin) Type() string {
+	return PrefixCachePluginType
 }
 
 // Score returns the scoring result for the given list of pods based on context.
-func (m *Plugin) Score(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, pods []types.Pod) map[types.Pod]float64 {
+func (m *Plugin) Score(ctx context.Context, cycleState *types.CycleState, request *types.LLMRequest, pods []types.Pod) map[types.Pod]float64 {
 	loggerTrace := log.FromContext(ctx).V(logutil.TRACE)
 	// pre score step, hashing prompt and find longest prefix match.
 	hashes := hashPrompt(ctx, request, m.HashBlockSize, m.MaxPrefixBlocksToMatch)
-	numServers := DefaultNumServersToMatch
-	if numServers > len(pods) {
-		numServers = len(pods)
-	}
 	state := &schedulingContextState{
 		PrefixHashes:       hashes,
-		PrefixCacheServers: m.matchLongestPrefix(ctx, hashes, numServers),
+		PrefixCacheServers: m.matchLongestPrefix(ctx, hashes),
 	}
-	cycleState.Write(types.StateKey(m.Name()), state)
+
+	cycleState.Write(types.StateKey(m.Type()), state)
 	loggerTrace.Info(fmt.Sprintf("cached servers: %+v", state.PrefixCacheServers), "hashes", state.PrefixHashes)
 	// calculate the scores of pods
 	scores := make(map[types.Pod]float64, len(pods))
@@ -167,36 +188,38 @@ func (m *Plugin) Score(ctx context.Context, request *types.LLMRequest, cycleStat
 }
 
 // PostCycle records in the plugin cache the result of the scheduling selection.
-func (m *Plugin) PostCycle(ctx context.Context, cycleState *types.CycleState, res *types.Result) {
+func (m *Plugin) PostCycle(ctx context.Context, cycleState *types.CycleState, res *types.ProfileRunResult) {
 	targetPod := res.TargetPod.GetPod()
 	state, err := m.getPrefixState(cycleState)
 	if err != nil {
 		log.FromContext(ctx).Error(err, "failed to read prefix plugin cycle state")
 		return
 	}
+
 	m.indexer.Add(state.PrefixHashes, ServerID(targetPod.NamespacedName))
+
 	total := len(state.PrefixHashes)
 	matchLen := state.PrefixCacheServers[ServerID(targetPod.NamespacedName)]
 	metrics.RecordPrefixCacheMatch(matchLen*m.HashBlockSize, total*m.HashBlockSize)
 }
 
 // matchLongestPrefix returns a map of servers and length of prefix that each server caches.
-func (m *Plugin) matchLongestPrefix(ctx context.Context, hashes []BlockHash, numServers int) map[ServerID]int {
+func (m *Plugin) matchLongestPrefix(ctx context.Context, hashes []BlockHash) map[ServerID]int {
 	loggerTrace := log.FromContext(ctx).V(logutil.TRACE)
 	res := make(map[ServerID]int)
 	// Use a greedy strategy to search from the longest prefix.
 	// NOTE: It's possible to further optimize this with a binary search.
-	for i := len(hashes) - 1; i >= 0 && len(res) < numServers; i-- {
+	for i := 0; i < len(hashes); i++ {
 		hash := hashes[i]
 		cachedServers := m.indexer.Get(hash)
-		if len(cachedServers) > 0 {
+		if len(cachedServers) == 0 {
+			break
+		} else {
 			loggerTrace.Info("Found cached servers", "cachedServers", cachedServers, "total # blocks", len(hashes), "longest prefix", i)
 			for server := range cachedServers {
 				// Update servers with their longest prefix match.
-				// If we already found this server with longer prefix match, don't update it.
-				if _, ok := res[server]; !ok {
-					res[server] = i + 1
-				}
+				res[server]++
+
 			}
 		}
 	}
@@ -205,7 +228,7 @@ func (m *Plugin) matchLongestPrefix(ctx context.Context, hashes []BlockHash, num
 
 // getPrefixState returns the cycle state as a schedulingContextState.
 func (m *Plugin) getPrefixState(cycleState *types.CycleState) (*schedulingContextState, error) {
-	prefixStateKey := types.StateKey(m.Name())
+	prefixStateKey := types.StateKey(m.Type())
 	state, err := cycleState.Read(prefixStateKey)
 	if err != nil {
 		return nil, fmt.Errorf("failed reading %q from CycleState: %w", prefixStateKey, err)
diff --git a/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go b/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
index 8f06219..43c438d 100644
--- a/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
+++ b/pkg/epp/scheduling/framework/plugins/picker/max_score_picker.go
@@ -18,17 +18,28 @@ package picker
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 
 	"sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-// compile-time type assertion
+const (
+	MaxScorePickerType = "max-score"
+)
+
+// compile-time type validation
 var _ framework.Picker = &MaxScorePicker{}
 
+// MaxScorePickerFactory defines the factory function for MaxScorePicker.
+func MaxScorePickerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewMaxScorePicker(), nil
+}
+
 // NewMaxScorePicker initializes a new MaxScorePicker and returns its pointer.
 func NewMaxScorePicker() *MaxScorePicker {
 	return &MaxScorePicker{
@@ -41,13 +52,13 @@ type MaxScorePicker struct {
 	random *RandomPicker
 }
 
-// Name returns the name of the picker.
-func (p *MaxScorePicker) Name() string {
-	return "max_score"
+// Type returns the type of the picker.
+func (p *MaxScorePicker) Type() string {
+	return MaxScorePickerType
 }
 
 // Pick selects the pod with the maximum score from the list of candidates.
-func (p *MaxScorePicker) Pick(ctx context.Context, cycleState *types.CycleState, scoredPods []*types.ScoredPod) *types.Result {
+func (p *MaxScorePicker) Pick(ctx context.Context, cycleState *types.CycleState, scoredPods []*types.ScoredPod) *types.ProfileRunResult {
 	log.FromContext(ctx).V(logutil.DEBUG).Info(fmt.Sprintf("Selecting a pod with the max score from %d candidates: %+v", len(scoredPods), scoredPods))
 
 	highestScorePods := []*types.ScoredPod{}
@@ -65,5 +76,5 @@ func (p *MaxScorePicker) Pick(ctx context.Context, cycleState *types.CycleState,
 		return p.random.Pick(ctx, cycleState, highestScorePods) // pick randomly from the highest score pods
 	}
 
-	return &types.Result{TargetPod: highestScorePods[0]}
+	return &types.ProfileRunResult{TargetPod: highestScorePods[0]}
 }
diff --git a/pkg/epp/scheduling/framework/plugins/picker/random_picker.go b/pkg/epp/scheduling/framework/plugins/picker/random_picker.go
index 7ef2432..f12a05b 100644
--- a/pkg/epp/scheduling/framework/plugins/picker/random_picker.go
+++ b/pkg/epp/scheduling/framework/plugins/picker/random_picker.go
@@ -18,18 +18,29 @@ package picker
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"math/rand"
 
 	"sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
-// compile-time type assertion
+const (
+	RandomPickerType = "random"
+)
+
+// compile-time type validation
 var _ framework.Picker = &RandomPicker{}
 
+// RandomPickerFactory defines the factory function for RandomPicker.
+func RandomPickerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewRandomPicker(), nil
+}
+
 // NewRandomPicker initializes a new RandomPicker and returns its pointer.
 func NewRandomPicker() *RandomPicker {
 	return &RandomPicker{}
@@ -38,14 +49,14 @@ func NewRandomPicker() *RandomPicker {
 // RandomPicker picks a random pod from the list of candidates.
 type RandomPicker struct{}
 
-// Name returns the name of the picker.
-func (p *RandomPicker) Name() string {
-	return "random"
+// Type returns the type of the picker.
+func (p *RandomPicker) Type() string {
+	return RandomPickerType
 }
 
 // Pick selects a random pod from the list of candidates.
-func (p *RandomPicker) Pick(ctx context.Context, _ *types.CycleState, scoredPods []*types.ScoredPod) *types.Result {
+func (p *RandomPicker) Pick(ctx context.Context, _ *types.CycleState, scoredPods []*types.ScoredPod) *types.ProfileRunResult {
 	log.FromContext(ctx).V(logutil.DEBUG).Info(fmt.Sprintf("Selecting a random pod from %d candidates: %+v", len(scoredPods), scoredPods))
 	i := rand.Intn(len(scoredPods))
-	return &types.Result{TargetPod: scoredPods[i]}
+	return &types.ProfileRunResult{TargetPod: scoredPods[i]}
 }
diff --git a/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go b/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go
deleted file mode 100644
index 266b1fc..0000000
--- a/pkg/epp/scheduling/framework/plugins/profile-picker/all_profiles_picker.go
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
-Copyright 2025 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package profilepicker
-
-import (
-	"context"
-
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
-)
-
-// compile-time type assertion
-var _ framework.ProfilePicker = &AllProfilesPicker{}
-
-// NewAllProfilesPicker initializes a new AllProfilesPicker and returns its pointer.
-func NewAllProfilesPicker() *AllProfilesPicker {
-	return &AllProfilesPicker{}
-}
-
-// AllProfilesPicker picks all profiles always.
-type AllProfilesPicker struct{}
-
-// Name returns the name of the Profiles Picker.
-func (p *AllProfilesPicker) Name() string {
-	return "all-profiles"
-}
-
-// Pick selects the SchedulingProfiles to run from the list of candidate profiles, while taking into consideration the request properties and the
-// previously executed cycles along with their results.
-func (p *AllProfilesPicker) Pick(_ context.Context, request *types.LLMRequest, profiles map[string]*framework.SchedulerProfile,
-	executionResults map[string]*types.Result) map[string]*framework.SchedulerProfile {
-	if len(profiles) == len(executionResults) { // all profiles have been executed already in previous call
-		return map[string]*framework.SchedulerProfile{}
-	}
-	// return all profiles
-	return profiles
-}
diff --git a/pkg/epp/scheduling/framework/plugins/profile/single_profile_handler.go b/pkg/epp/scheduling/framework/plugins/profile/single_profile_handler.go
new file mode 100644
index 0000000..ca87f5e
--- /dev/null
+++ b/pkg/epp/scheduling/framework/plugins/profile/single_profile_handler.go
@@ -0,0 +1,90 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package profile
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+const (
+	SingleProfileHandlerType = "single-profile"
+)
+
+// compile-time type assertion
+var _ framework.ProfileHandler = &SingleProfileHandler{}
+
+// SingleProfileHandlerFactory defines the factory function for SingleProfileHandler.
+func SingleProfileHandlerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewSingleProfileHandler(), nil
+}
+
+// NewSingleProfileHandler initializes a new SingleProfileHandler and returns its pointer.
+func NewSingleProfileHandler() *SingleProfileHandler {
+	return &SingleProfileHandler{}
+}
+
+// SingleProfileHandler handles a single profile which is always the primary profile.
+type SingleProfileHandler struct{}
+
+// Type returns the type of the Profile Handler.
+func (h *SingleProfileHandler) Type() string {
+	return SingleProfileHandlerType
+}
+
+// Pick selects the SchedulingProfiles to run from the list of candidate profiles, while taking into consideration the request properties and the
+// previously executed cycles along with their results.
+func (h *SingleProfileHandler) Pick(_ context.Context, _ *types.CycleState, request *types.LLMRequest, profiles map[string]*framework.SchedulerProfile,
+	profileResults map[string]*types.ProfileRunResult) map[string]*framework.SchedulerProfile {
+	if len(profiles) == len(profileResults) { // all profiles have been executed already in previous call
+		return map[string]*framework.SchedulerProfile{}
+	}
+	// return all profiles
+	return profiles
+}
+
+// ProcessResults handles the outcome of the profile runs after all profiles ran.
+// It may aggregate results, log test profile outputs, or apply custom logic. It specifies in the SchedulingResult the
+// key of the primary profile that should be used to get the request selected destination.
+// When a profile run fails, its result in the profileResults map is nil.
+func (h *SingleProfileHandler) ProcessResults(_ context.Context, _ *types.CycleState, _ *types.LLMRequest,
+	profileResults map[string]*types.ProfileRunResult) (*types.SchedulingResult, error) {
+	if len(profileResults) != 1 {
+		return nil, errors.New("single profile handler is intended to be used with a single profile, failed to process multiple profiles")
+	}
+
+	var singleProfileName string
+	for profileName := range profileResults {
+		singleProfileName = profileName
+		break
+	}
+
+	if profileResults[singleProfileName] == nil { // there was an error while running the profile
+		return nil, fmt.Errorf("failed to run scheduler profile '%s'", singleProfileName)
+	}
+
+	return &types.SchedulingResult{
+		ProfileResults:     profileResults,
+		PrimaryProfileName: singleProfileName,
+	}, nil
+}
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go b/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
index f572805..6bab369 100644
--- a/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/kvcache.go
@@ -18,28 +18,41 @@ package scorer
 
 import (
 	"context"
+	"encoding/json"
 
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
 const (
 	DefaultKVCacheScorerWeight = 1
+	KvCacheScorerType          = "kv-cache"
 )
 
 // compile-time type assertion
 var _ framework.Scorer = &KVCacheScorer{}
 
+// KvCacheScorerFactory defines the factory function for KVCacheScorer.
+func KvCacheScorerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewKVCacheScorer(), nil
+}
+
+// NewKVCacheScorer initializes a new KVCacheScorer and returns its pointer.
+func NewKVCacheScorer() *KVCacheScorer {
+	return &KVCacheScorer{}
+}
+
 // KVCacheScorer scores list of candidate pods based on KV cache utilization.
 type KVCacheScorer struct{}
 
-// Name returns the name of the scorer.
-func (s *KVCacheScorer) Name() string {
-	return "kv-cache"
+// Type returns the type of the scorer.
+func (s *KVCacheScorer) Type() string {
+	return KvCacheScorerType
 }
 
 // Score returns the scoring result for the given list of pods based on context.
-func (s *KVCacheScorer) Score(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) map[types.Pod]float64 {
+func (s *KVCacheScorer) Score(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) map[types.Pod]float64 {
 	scores := make(map[types.Pod]float64, len(pods))
 	for _, pod := range pods {
 		scores[pod] = 1 - pod.GetMetrics().KVCacheUsagePercent
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/queue.go b/pkg/epp/scheduling/framework/plugins/scorer/queue.go
index c484402..a3c960e 100644
--- a/pkg/epp/scheduling/framework/plugins/scorer/queue.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/queue.go
@@ -18,30 +18,43 @@ package scorer
 
 import (
 	"context"
+	"encoding/json"
 	"math"
 
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
 const (
 	DefaultQueueScorerWeight = 1
+	QueueScorerType          = "queue"
 )
 
 // compile-time type assertion
 var _ framework.Scorer = &QueueScorer{}
 
+// QueueScorerFactory defines the factory function for QueueScorer.
+func QueueScorerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewQueueScorer(), nil
+}
+
+// NewQueueScorer initializes a new QueueScorer and returns its pointer.
+func NewQueueScorer() *QueueScorer {
+	return &QueueScorer{}
+}
+
 // QueueScorer scores list of candidate pods based on the pod's waiting queue size.
 // the less waiting queue size the pod has, the higher score it will get (since it's more available to serve new request).
 type QueueScorer struct{}
 
-// Name returns the name of the scorer.
-func (s *QueueScorer) Name() string {
-	return "queue"
+// Type returns the type of the scorer.
+func (s *QueueScorer) Type() string {
+	return QueueScorerType
 }
 
 // Score returns the scoring result for the given list of pods based on context.
-func (s *QueueScorer) Score(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) map[types.Pod]float64 {
+func (s *QueueScorer) Score(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) map[types.Pod]float64 {
 	minQueueSize := math.MaxInt
 	maxQueueSize := math.MinInt
 
diff --git a/pkg/epp/scheduling/framework/scheduler_profile.go b/pkg/epp/scheduling/framework/scheduler_profile.go
index ccbb4fb..f41a915 100644
--- a/pkg/epp/scheduling/framework/scheduler_profile.go
+++ b/pkg/epp/scheduling/framework/scheduler_profile.go
@@ -86,14 +86,14 @@ func (p *SchedulerProfile) AddPlugins(pluginObjects ...plugins.Plugin) error {
 			p.scorers = append(p.scorers, weightedScorer)
 			plugin = weightedScorer.Scorer // if we got WeightedScorer, unwrap the plugin
 		} else if scorer, ok := plugin.(Scorer); ok { // if we got a Scorer instead of WeightedScorer that's an error.
-			return fmt.Errorf("failed to register scorer '%s' without a weight. follow function documentation to register a scorer", scorer.Name())
+			return fmt.Errorf("failed to register scorer '%s' without a weight. follow function documentation to register a scorer", scorer.Type())
 		}
 		if filter, ok := plugin.(Filter); ok {
 			p.filters = append(p.filters, filter)
 		}
 		if picker, ok := plugin.(Picker); ok {
 			if p.picker != nil {
-				return fmt.Errorf("failed to set '%s' as picker, already have a registered picker plugin '%s'", picker.Name(), p.picker.Name())
+				return fmt.Errorf("failed to set '%s' as picker, already have a registered picker plugin '%s'", picker.Type(), p.picker.Type())
 			}
 			p.picker = picker
 		}
@@ -106,8 +106,8 @@ func (p *SchedulerProfile) AddPlugins(pluginObjects ...plugins.Plugin) error {
 
 // RunCycle runs a SchedulerProfile cycle. In other words, it invokes all the SchedulerProfile plugins in this
 // order - Filters, Scorers, Picker, PostCyclePlugins. After completing all, it returns the result.
-func (p *SchedulerProfile) Run(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, podsSnapshot []types.Pod) (*types.Result, error) {
-	pods := p.runFilterPlugins(ctx, request, cycleState, podsSnapshot)
+func (p *SchedulerProfile) Run(ctx context.Context, request *types.LLMRequest, cycleState *types.CycleState, candidatePods []types.Pod) (*types.ProfileRunResult, error) {
+	pods := p.runFilterPlugins(ctx, request, cycleState, candidatePods)
 	if len(pods) == 0 {
 		return nil, errutil.Error{Code: errutil.Internal, Msg: "no pods available for the given request"}
 	}
@@ -127,11 +127,11 @@ func (p *SchedulerProfile) runFilterPlugins(ctx context.Context, request *types.
 	loggerDebug.Info("Before running filter plugins", "pods", filteredPods)
 
 	for _, filter := range p.filters {
-		loggerDebug.Info("Running filter plugin", "plugin", filter.Name())
+		loggerDebug.Info("Running filter plugin", "plugin", filter.Type())
 		before := time.Now()
-		filteredPods = filter.Filter(ctx, request, cycleState, filteredPods)
-		metrics.RecordSchedulerPluginProcessingLatency(FilterPluginType, filter.Name(), time.Since(before))
-		loggerDebug.Info("Filter plugin result", "plugin", filter.Name(), "pods", filteredPods)
+		filteredPods = filter.Filter(ctx, cycleState, request, filteredPods)
+		metrics.RecordSchedulerPluginProcessingLatency(FilterPluginType, filter.Type(), time.Since(before))
+		loggerDebug.Info("Filter plugin result", "plugin", filter.Type(), "pods", filteredPods)
 		if len(filteredPods) == 0 {
 			break
 		}
@@ -151,21 +151,21 @@ func (p *SchedulerProfile) runScorerPlugins(ctx context.Context, request *types.
 	}
 	// Iterate through each scorer in the chain and accumulate the weighted scores.
 	for _, scorer := range p.scorers {
-		loggerDebug.Info("Running scorer", "scorer", scorer.Name())
+		loggerDebug.Info("Running scorer", "scorer", scorer.Type())
 		before := time.Now()
-		scores := scorer.Score(ctx, request, cycleState, pods)
-		metrics.RecordSchedulerPluginProcessingLatency(ScorerPluginType, scorer.Name(), time.Since(before))
+		scores := scorer.Score(ctx, cycleState, request, pods)
+		metrics.RecordSchedulerPluginProcessingLatency(ScorerPluginType, scorer.Type(), time.Since(before))
 		for pod, score := range scores { // weight is relative to the sum of weights
 			weightedScorePerPod[pod] += score * float64(scorer.Weight())
 		}
-		loggerDebug.Info("After running scorer", "scorer", scorer.Name())
+		loggerDebug.Info("After running scorer", "scorer", scorer.Type())
 	}
 	loggerDebug.Info("After running scorer plugins")
 
 	return weightedScorePerPod
 }
 
-func (p *SchedulerProfile) runPickerPlugin(ctx context.Context, cycleState *types.CycleState, weightedScorePerPod map[types.Pod]float64) *types.Result {
+func (p *SchedulerProfile) runPickerPlugin(ctx context.Context, cycleState *types.CycleState, weightedScorePerPod map[types.Pod]float64) *types.ProfileRunResult {
 	loggerDebug := log.FromContext(ctx).V(logutil.DEBUG)
 	scoredPods := make([]*types.ScoredPod, len(weightedScorePerPod))
 	i := 0
@@ -177,17 +177,17 @@ func (p *SchedulerProfile) runPickerPlugin(ctx context.Context, cycleState *type
 	loggerDebug.Info("Before running picker plugin", "pods weighted score", fmt.Sprint(weightedScorePerPod))
 	before := time.Now()
 	result := p.picker.Pick(ctx, cycleState, scoredPods)
-	metrics.RecordSchedulerPluginProcessingLatency(PickerPluginType, p.picker.Name(), time.Since(before))
+	metrics.RecordSchedulerPluginProcessingLatency(PickerPluginType, p.picker.Type(), time.Since(before))
 	loggerDebug.Info("After running picker plugin", "result", result)
 
 	return result
 }
 
-func (p *SchedulerProfile) runPostCyclePlugins(ctx context.Context, cycleState *types.CycleState, result *types.Result) {
+func (p *SchedulerProfile) runPostCyclePlugins(ctx context.Context, cycleState *types.CycleState, result *types.ProfileRunResult) {
 	for _, plugin := range p.postCyclePlugins {
-		log.FromContext(ctx).V(logutil.DEBUG).Info("Running post-cycle plugin", "plugin", plugin.Name())
+		log.FromContext(ctx).V(logutil.DEBUG).Info("Running post-cycle plugin", "plugin", plugin.Type())
 		before := time.Now()
 		plugin.PostCycle(ctx, cycleState, result)
-		metrics.RecordSchedulerPluginProcessingLatency(PostCyclePluginType, plugin.Name(), time.Since(before))
+		metrics.RecordSchedulerPluginProcessingLatency(PostCyclePluginType, plugin.Type(), time.Since(before))
 	}
 }
diff --git a/pkg/epp/scheduling/scheduler.go b/pkg/epp/scheduling/scheduler.go
index bc7bf13..b848b26 100644
--- a/pkg/epp/scheduling/scheduler.go
+++ b/pkg/epp/scheduling/scheduler.go
@@ -25,25 +25,30 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/config"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
-	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
+type Datastore interface {
+	PodGetAll() []backendmetrics.PodMetrics
+}
+
 // NewScheduler returns a new scheduler with default scheduler plugins configuration.
-func NewScheduler(datastore Datastore) *Scheduler {
+func NewScheduler() *Scheduler {
 	// When the scheduler is initialized with NewScheduler function, thw below config will be used as default.
 	// it's possible to call NewSchedulerWithConfig to pass a different scheduler config.
 	// For build time plugins changes, it's recommended to call in main.go to NewSchedulerWithConfig.
-	loraAffinityFilter := filter.NewLoraAffinityFilter()
+	loraAffinityFilter := filter.NewLoraAffinityFilter(config.Conf.LoraAffinityThreshold)
 	leastQueueFilter := filter.NewLeastQueueFilter()
 	leastKvCacheFilter := filter.NewLeastKVCacheFilter()
 
 	lowLatencyFilter := &filter.DecisionTreeFilter{
-		Current: filter.NewLowQueueFilter(),
+		Current: filter.NewLowQueueFilter(config.Conf.QueueingThresholdLoRA),
 		NextOnSuccess: &filter.DecisionTreeFilter{
 			Current: loraAffinityFilter,
 			NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
@@ -68,32 +73,26 @@ func NewScheduler(datastore Datastore) *Scheduler {
 		WithFilters(lowLatencyFilter).
 		WithPicker(&picker.RandomPicker{})
 
-	profilePicker := profilepicker.NewAllProfilesPicker()
+	profileHandler := profile.NewSingleProfileHandler()
 
-	return NewSchedulerWithConfig(datastore, NewSchedulerConfig(profilePicker, map[string]*framework.SchedulerProfile{"default": defaultProfile}))
+	return NewSchedulerWithConfig(NewSchedulerConfig(profileHandler, map[string]*framework.SchedulerProfile{"default": defaultProfile}))
 }
 
 // NewSchedulerWithConfig returns a new scheduler with the given scheduler plugins configuration.
-func NewSchedulerWithConfig(datastore Datastore, config *SchedulerConfig) *Scheduler {
+func NewSchedulerWithConfig(config *SchedulerConfig) *Scheduler {
 	return &Scheduler{
-		datastore:     datastore,
-		profilePicker: config.profilePicker,
-		profiles:      config.profiles,
+		profileHandler: config.profileHandler,
+		profiles:       config.profiles,
 	}
 }
 
 type Scheduler struct {
-	datastore     Datastore
-	profilePicker framework.ProfilePicker
-	profiles      map[string]*framework.SchedulerProfile
-}
-
-type Datastore interface {
-	PodGetAll() []backendmetrics.PodMetrics
+	profileHandler framework.ProfileHandler
+	profiles       map[string]*framework.SchedulerProfile
 }
 
 // Schedule finds the target pod based on metrics and the requested lora adapter.
-func (s *Scheduler) Schedule(ctx context.Context, request *types.LLMRequest) (map[string]*types.Result, error) {
+func (s *Scheduler) Schedule(ctx context.Context, request *types.LLMRequest, candidatePods []types.Pod) (*types.SchedulingResult, error) {
 	logger := log.FromContext(ctx).WithValues("request", request)
 	loggerDebug := logger.V(logutil.DEBUG)
 
@@ -102,36 +101,35 @@ func (s *Scheduler) Schedule(ctx context.Context, request *types.LLMRequest) (ma
 		metrics.RecordSchedulerE2ELatency(time.Since(scheduleStart))
 	}()
 
-	// Snapshot pod metrics from the datastore to:
-	// 1. Reduce concurrent access to the datastore.
-	// 2. Ensure consistent data during the scheduling operation of a request between all scheduling cycles.
-	podsSnapshot := types.ToSchedulerPodMetrics(s.datastore.PodGetAll())
-	loggerDebug.Info(fmt.Sprintf("Scheduling a request, Metrics: %+v", podsSnapshot))
-
-	profileExecutionResults := map[string]*types.Result{}
+	profileRunResults := map[string]*types.ProfileRunResult{}
+	cycleState := types.NewCycleState()
 
 	for { // get the next set of profiles to run iteratively based on the request and the previous execution results
 		before := time.Now()
-		profiles := s.profilePicker.Pick(ctx, request, s.profiles, profileExecutionResults)
-		metrics.RecordSchedulerPluginProcessingLatency(framework.ProfilePickerType, s.profilePicker.Name(), time.Since(before))
+		profiles := s.profileHandler.Pick(ctx, cycleState, request, s.profiles, profileRunResults)
+		metrics.RecordSchedulerPluginProcessingLatency(framework.ProfilePickerType, s.profileHandler.Type(), time.Since(before))
 		if len(profiles) == 0 { // profile picker didn't pick any profile to run
 			break
 		}
 
 		for name, profile := range profiles {
 			// run the selected profiles and collect results (current code runs all profiles)
-			profileExecutionResult, err := profile.Run(ctx, request, types.NewCycleState(), podsSnapshot)
+			profileRunResult, err := profile.Run(ctx, request, cycleState, candidatePods)
 			if err != nil {
-				return nil, fmt.Errorf("failed to run all required scheduling profiles - %w", err)
+				loggerDebug.Info("failed to run scheduler profile", "profile", name, "error", err.Error())
 			}
 
-			profileExecutionResults[name] = profileExecutionResult
+			profileRunResults[name] = profileRunResult // if profile failed to run, the run result is nil
 		}
 	}
 
-	if len(profileExecutionResults) == 0 {
+	if len(profileRunResults) == 0 {
 		return nil, fmt.Errorf("failed to run any SchedulingProfile for the request - %s", request)
 	}
 
-	return profileExecutionResults, nil
+	before := time.Now()
+	result, err := s.profileHandler.ProcessResults(ctx, cycleState, request, profileRunResults)
+	metrics.RecordSchedulerPluginProcessingLatency(framework.ProcessProfilesResultsType, s.profileHandler.Type(), time.Since(before))
+
+	return result, err
 }
diff --git a/pkg/epp/scheduling/scheduler_config.go b/pkg/epp/scheduling/scheduler_config.go
index 4008b70..41524f6 100644
--- a/pkg/epp/scheduling/scheduler_config.go
+++ b/pkg/epp/scheduling/scheduler_config.go
@@ -16,18 +16,20 @@ limitations under the License.
 
 package scheduling
 
-import "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+)
 
 // NewSchedulerConfig creates a new SchedulerConfig object and returns its pointer.
-func NewSchedulerConfig(profilePicker framework.ProfilePicker, profiles map[string]*framework.SchedulerProfile) *SchedulerConfig {
+func NewSchedulerConfig(profileHandler framework.ProfileHandler, profiles map[string]*framework.SchedulerProfile) *SchedulerConfig {
 	return &SchedulerConfig{
-		profilePicker: profilePicker,
-		profiles:      profiles,
+		profileHandler: profileHandler,
+		profiles:       profiles,
 	}
 }
 
 // SchedulerConfig provides a configuration for the scheduler which influence routing decisions.
 type SchedulerConfig struct {
-	profilePicker framework.ProfilePicker
-	profiles      map[string]*framework.SchedulerProfile
+	profileHandler framework.ProfileHandler
+	profiles       map[string]*framework.SchedulerProfile
 }
diff --git a/pkg/epp/scheduling/types/types.go b/pkg/epp/scheduling/types/types.go
index 58bae0d..4513847 100644
--- a/pkg/epp/scheduling/types/types.go
+++ b/pkg/epp/scheduling/types/types.go
@@ -78,7 +78,13 @@ func ToSchedulerPodMetrics(pods []backendmetrics.PodMetrics) []Pod {
 	return pm
 }
 
-// Result captures the scheduler result.
-type Result struct {
+// ProfileRunResult captures the profile run result.
+type ProfileRunResult struct {
 	TargetPod Pod
 }
+
+// SchedulingResult captures the result of the scheduling cycle.
+type SchedulingResult struct {
+	ProfileResults     map[string]*ProfileRunResult
+	PrimaryProfileName string
+}
diff --git a/pkg/epp/server/runserver.go b/pkg/epp/server/runserver.go
index cc29ee4..daee8e1 100644
--- a/pkg/epp/server/runserver.go
+++ b/pkg/epp/server/runserver.go
@@ -20,12 +20,15 @@ import (
 	"context"
 	"crypto/tls"
 	"fmt"
+
 	"time"
 
 	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
 	"github.com/go-logr/logr"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials"
+	"google.golang.org/grpc/health"
+	healthgrpc "google.golang.org/grpc/health/grpc_health_v1"
 	"k8s.io/apimachinery/pkg/types"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
@@ -46,6 +49,7 @@ type ExtProcServerRunner struct {
 	PoolNamespacedName                       types.NamespacedName
 	Datastore                                datastore.Datastore
 	SecureServing                            bool
+	HealthChecking                           bool
 	CertPath                                 string
 	RefreshPrometheusMetricsInterval         time.Duration
 	Director                                 *requestcontrol.Director
@@ -66,6 +70,7 @@ const (
 	DefaultRefreshMetricsInterval                   = 50 * time.Millisecond            // default for --refreshMetricsInterval
 	DefaultRefreshPrometheusMetricsInterval         = 5 * time.Second                  // default for --refreshPrometheusMetricsInterval
 	DefaultSecureServing                            = true                             // default for --secureServing
+	DefaultHealthChecking                           = false                            // default for --healthChecking
 )
 
 // NewDefaultExtProcServerRunner creates a runner with default values.
@@ -77,6 +82,7 @@ func NewDefaultExtProcServerRunner() *ExtProcServerRunner {
 		DestinationEndpointHintMetadataNamespace: DefaultDestinationEndpointHintMetadataNamespace,
 		PoolNamespacedName:                       types.NamespacedName{Name: DefaultPoolName, Namespace: DefaultPoolNamespace},
 		SecureServing:                            DefaultSecureServing,
+		HealthChecking:                           DefaultHealthChecking,
 		RefreshPrometheusMetricsInterval:         DefaultRefreshPrometheusMetricsInterval,
 		// Dependencies can be assigned later.
 	}
@@ -152,6 +158,16 @@ func (r *ExtProcServerRunner) AsRunnable(logger logr.Logger) manager.Runnable {
 			extProcServer,
 		)
 
+		if r.HealthChecking {
+			healthcheck := health.NewServer()
+			healthgrpc.RegisterHealthServer(srv,
+				healthcheck,
+			)
+			svcName := extProcPb.ExternalProcessor_ServiceDesc.ServiceName
+			logger.Info("Setting ExternalProcessor service status to SERVING", "serviceName", svcName)
+			healthcheck.SetServingStatus(svcName, healthgrpc.HealthCheckResponse_SERVING)
+		}
+
 		// Forward to the gRPC runnable.
 		return runnable.GRPCServer("ext-proc", srv, r.GrpcPort).Start(ctx)
 	}))
diff --git a/site-src/concepts/api-overview.md b/site-src/concepts/api-overview.md
index 921b6fa..ab07a1d 100644
--- a/site-src/concepts/api-overview.md
+++ b/site-src/concepts/api-overview.md
@@ -1,12 +1,24 @@
 # API Overview
 
 ## Background
-The Gateway API Inference Extension project is an extension of the Kubernetes Gateway API for serving Generative AI models on Kubernetes. Gateway API Inference Extension facilitates standardization of APIs for Kubernetes cluster operators and developers running generative AI inference, while allowing flexibility for underlying gateway implementations (such as Envoy Proxy) to iterate on mechanisms for optimized serving of models.
+Gateway API Inference Extension optimizes self-hosting Generative AI Models on Kubernetes. 
+It provides optimized load-balancing for self-hosted Generative AI Models on Kubernetes.
+The project’s goal is to improve and standardize routing to inference workloads across the ecosystem.
 
-<img src="/images/inference-overview.svg" alt="Overview of API integration" class="center" width="1000" />
+This is achieved by leveraging Envoy's [External Processing](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter) to extend any gateway that supports both ext-proc and [Gateway API](https://github.com/kubernetes-sigs/gateway-api) into an [inference gateway](../index.md#concepts-and-definitions).
+This extension extends popular gateways like Envoy Gateway, kgateway, and GKE Gateway - to become [Inference Gateway](../index.md#concepts-and-definitions) -
+supporting inference platform teams self-hosting Generative Models (with a current focus on large language models) on Kubernetes.
+This integration makes it easy to expose and control access to your local [OpenAI-compatible chat completion endpoints](https://platform.openai.com/docs/api-reference/chat)
+to other workloads on or off cluster, or to integrate your self-hosted models alongside model-as-a-service providers 
+in a higher level **AI Gateways** like [LiteLLM](https://www.litellm.ai/), [Gloo AI Gateway](https://www.solo.io/products/gloo-ai-gateway), or [Apigee](https://cloud.google.com/apigee).
 
 ## API Resources
 
+Gateway API Inference Extension introduces two inference-focused API resources with distinct responsibilities, 
+each aligning with a specific user persona in the Generative AI serving workflow.
+
+<img src="/images/inference-overview.svg" alt="Overview of API integration" class="center" width="1000" />
+
 ### InferencePool
 
 InferencePool represents a set of Inference-focused Pods and an extension that will be used to route to them. Within the broader Gateway API resource model, this resource is considered a "backend". In practice, that means that you'd replace a Kubernetes Service with an InferencePool. This resource has some similarities to Service (a way to select Pods and specify a port), but has some unique capabilities. With InferencePool, you can configure a routing extension as well as inference-specific routing optimizations. For more information on this resource, refer to our [InferencePool documentation](/api-types/inferencepool) or go directly to the [InferencePool spec](/reference/spec/#inferencepool).
diff --git a/site-src/concepts/design-principles.md b/site-src/concepts/design-principles.md
new file mode 100644
index 0000000..76c3b77
--- /dev/null
+++ b/site-src/concepts/design-principles.md
@@ -0,0 +1,47 @@
+# Design Principles
+
+These principles guide our efforts to build flexible [Gateway API] extensions
+that empower the development of high-performance [AI Inference] routing
+technologies—balancing rapid delivery with long-term growth.
+
+!!! note "Inference Gateways"
+
+    For simplicity, we'll refer to Gateway API Gateways which are
+    composed together with AI Inference extensions as "Inference Gateways"
+    throughout this document.
+
+[Gateway API]:https://github.com/kubernetes-sigs/gateway-api
+[AI Inference]:https://www.arm.com/glossary/ai-inference
+
+
+## Prioritize stability of the core interfaces
+
+The most critical part of this project is the interfaces between components. To encourage both controller and extension developers to integrate with this project, we need to prioritize the stability of these interfaces.
+Although we can extend these interfaces in the future, it’s critical the core is stable as soon as possible.
+
+When describing "core interfaces", we are referring to both of the following:
+
+### 1. Gateway -> Endpoint Picker
+At a high level, this defines how a Gateway provides information to an Endpoint Picker, and how the Endpoint Picker selects endpoint(s) that the Gateway should route to.
+
+### 2. Endpoint Picker -> Model Server Framework
+This defines what an Endpoint Picker should expect from a compatible Model Server Framework with a focus on health checks and metrics.
+
+## Our presets are finely tuned
+
+We provide APIs and reference implementations for the most common inference requirements. Our defaults for those APIs and implementations—shaped by extensive experience with leading model serving platforms and APIs—are designed to provide the majority of Inference Gateway users with a great default experience without the need for extensive configuration or customization. If you take all of our default extensions and attach them to a compatible `Gateway`, it just "works out of the box".
+
+## Encourage innovation via extensibility
+
+This project is largely based on the idea that extensibility will enable innovation. With that in mind, we should make it as easy as possible for AI researchers to experiment with custom scheduling and routing logic. They should not need to know how to build a Kubernetes controller, or replicate a full networking stack. Instead, all the information needed to make a routing decision should be provided in an accessible format, with clear guidelines and examples of how to customize routing logic.
+
+## Objectives over instructions
+
+The pace of innovation in this ecosystem has been rapid. Focusing too heavily on the specifics of current techniques could result in the API becoming outdated quickly. Instead of making the API too descriptive about _how_ an objective should be achieved, this API should focus on the objectives that a Gateway and/or Endpoint Picker should strive to attain. Overly specific instructions or configuration can start as implementation specific APIs and grow into standards as the concepts become more stable and widespread.
+
+## Composable components and reducing reinvention
+While it may be tempting to develop an entirely new AI-focused Gateway, many essential routing capabilities are already well established by Kubernetes. Our focus is on creating a layer of composable components that can be assembled together with other Kubernetes components. This approach empowers engineers to use our solution as a building block—combining established technologies like Gateway API with our extensible model to build higher level solutions. Should you encounter a limitation, consider how existing tooling may be extended or improved first.
+
+## Additions to the API should be carefully prioritized
+
+Every addition to the API should take the principles described above into account. Given that the goal of the API is to encourage a highly extensible ecosystem, each additional feature in the API is raising the barrier for entry to any new controller or extension. Our top priority should be to focus on concepts that we expect to be broadly implementable and useful. The extensible nature of this API will allow each individual implementation to experiment with new features via custom flags or APIs before they become part of the core API surface.
diff --git a/site-src/guides/conformance-tests.md b/site-src/guides/conformance-tests.md
new file mode 100644
index 0000000..aa88438
--- /dev/null
+++ b/site-src/guides/conformance-tests.md
@@ -0,0 +1,46 @@
+
+# Test Setup and Execution
+
+This document provides steps to run the Gateway API Inference Extension conformance tests.
+
+## Prerequisites
+
+1.  You need a Kubernetes cluster with [LoadBalancer](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) support.
+
+2.  Choose an Implementation -
+Install an [existing implementation](https://gateway-api-inference-extension.sigs.k8s.io/implementations/gateways/). For setup instructions, refer to the [The Quickstart Guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/).  Alternatively run tests against your implementation after completing the [implementer's guide](https://gateway-api-inference-extension.sigs.k8s.io/guides/implementers/#implementers-guide).
+
+Note: Since the EPP (EndPoint Picker) takes the `InferencePool` name as an environment variable, each conformance test creates a corresponding EPP deployment for each `InferencePool` it defines. For conformance testing, the EPP is configured with the `HeaderBasedTestingFilter`. This is enabled by setting the `ENABLE_REQ_HEADER_BASED_SCHEDULER_FOR_TESTING=true` environment variable in the EPP deployment manifest. 
+
+## Running Conformance Tests
+
+1.  **Clone the Repository**:
+    Create a local copy of the Gateway API Inference Extension repository:
+    ```bash
+    git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension.git
+    cd gateway-api-inference-extension
+    ```
+
+2.  **Execute Tests**:
+    Run the following command to execute all available tests. Replace `<your_gatewayclass_name>` with the GatewayClass used by the implementation under test.
+
+    ```bash
+    go test ./conformance -args -gateway-class <your_gatewayclass_name>
+    ```
+
+### Test Execution Options
+
+* **Speeding up Reruns**: For repeated runs, you can add the flag `-cleanup-base-resources=false`. This will preserve resources such as namespaces and gateways between test runs, speeding up the process.
+    ```bash
+    go test ./conformance -args -gateway-class <your_gatewayclass_name> -cleanup-base-resources=false
+    ```
+
+* **Running Specific Tests**: To run a specific test, you can reference the test name by using the `-run-test` flag. For example:
+    ```bash
+    go test ./conformance -args -gateway-class <your_gatewayclass_name> -run-test HTTPRouteMultipleGatewaysDifferentPools
+    ```
+
+* **Detailed Logging**: To view detailed logs, you can enable logging mode by adding the `-v` as well as `-debug` flags.
+    ```bash
+    go test -v ./conformance -args -debug -gateway-class <your_gatewayclass_name> -cleanup-base-resources=false -run-test HTTPRouteMultipleGatewaysDifferentPools
+    ```
diff --git a/site-src/guides/epp-configuration/prefix-aware.md b/site-src/guides/epp-configuration/prefix-aware.md
index a0ad8b5..43e2ef0 100644
--- a/site-src/guides/epp-configuration/prefix-aware.md
+++ b/site-src/guides/epp-configuration/prefix-aware.md
@@ -4,7 +4,7 @@ The [prefix cache plugin](https://github.com/kubernetes-sigs/gateway-api-inferen
 takes advantage of the prefix caching (e.g., [vllm APC](https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html))
 feature of model servers, and optimizes request scheduling by placing requests sharing the longest
 prefixes to the same server as much as possible, while balancing the server load by considering kv-cache
-and queue depth. 
+and queue depth.
 
 ## Enable the prefix cache plugin
 
@@ -32,16 +32,18 @@ extremely long inputs.
 128 (or 128*64=8192 characters, or roughly 2048 tokens). This is useful to tradeoff prefix match accuracy
 for performance.
 
-* `PREFIX_CACHE_LRU_CAPACITY`: Maximum capacity the prefix LRU indexer in number of block hashes. Below
+* `PREFIX_CACHE_LRU_CAPACITY_PER_SERVER`: Maximum capacity the prefix LRU cache in number of block hashes per server (pod). Below
 shows a detailed analysis on how to estimate this.
 
+
+
     The prefix cache plugin estimates the prefix cache indexes in model server HBMs.  In the perfect
     scenario, EPP has the exact same prefix cache entries per model server as their HBM cache entries. If
     the EPP cache is smaller than HBM cache, a positive EPP cache match is more accurate, but there are more
     false cache misses. If the EPP cache is larger than the HBM cache, then there are more false cache hits.
     Therefore **the EPP prefix cache indexer size should be as close as possible to the HBM cache size.**
 
-    NOTE: EPP builds prefix cache based on characters, while model server maintains prefix cache entries 
+    NOTE: EPP builds prefix cache based on characters, while model server maintains prefix cache entries
     in tokens, a conversion between character <-> token is needed.
 
     Below are the formulas to estimate the EPP prefix indexer size:
@@ -63,8 +65,7 @@ shows a detailed analysis on how to estimate this.
     max_kv_tokens_per_server = (80GB - 16GB) / 128KB = 500,000
     # assume avg_chars_per_token = 4, prefix_indexer_hash_block_size = 64 (default)
     # each entry is about 358KB, so the memory footrpint is abut 11 MB per server
-    lru_indexer_capacity_per_server = 500,000*4/64 = 31250 
-    lru_indexer_capacity_total = 3 * 31250 = 93750
+    lru_indexer_capacity_per_server = 500,000*4/64 = 31250
     ```
 
 See the [Use Helm section](#helm) to install an inferencepool with the environment variables.
@@ -83,7 +84,7 @@ $ helm install triton-llama3-8b-instruct \
   --set provider.name=[none|gke] \
   --set inferenceExtension.env.EXPERIMENTAL_USE_SCHEDULER_V2=true \
   --set inferenceExtension.env.ENABLE_PREFIX_CACHE_SCHEDULING=true \
-  --set inferenceExtension.env.PREFIX_CACHE_LRU_CAPACITY=93750 \
+  --set inferenceExtension.env.PREFIX_CACHE_LRU_CAPACITY_PER_SERVER=31250 \
   --set inferenceExtension.env.PREFIX_CACHE_MAX_PREFIX_BLOCKS=1024 \
   oci://us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/charts/inferencepool --version v0
 ```
diff --git a/site-src/guides/implementers.md b/site-src/guides/implementers.md
index 7bfd536..204c417 100644
--- a/site-src/guides/implementers.md
+++ b/site-src/guides/implementers.md
@@ -110,4 +110,4 @@ Here are some tips for testing your controller end-to-end:
 
 ### Conformance Tests
 
-A set of conformance tests will be developed soon to help verify that a controller is working as expected. This guide will be updated once we have more information. Stay tuned!
+See [Conformance Test Setup and Execution](https://gateway-api-inference-extension.sigs.k8s.io/guides/conformance-tests).
diff --git a/site-src/guides/index.md b/site-src/guides/index.md
index e05a7a3..04b781b 100644
--- a/site-src/guides/index.md
+++ b/site-src/guides/index.md
@@ -1,10 +1,10 @@
-# Getting started with Gateway API Inference Extension
+# Getting started with an Inference Gateway
 
 ??? example "Experimental"
 
     This project is still in an alpha state and breaking changes may occur in the future.
 
-This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running! 
+This quickstart guide is intended for engineers familiar with k8s and model servers (vLLM in this instance). The goal of this guide is to get an Inference Gateway up and running!
 
 ## **Prerequisites**
 
@@ -35,7 +35,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
       For this setup, you will need 3 GPUs to run the sample model server. Adjust the number of replicas in `./config/manifests/vllm/gpu-deployment.yaml` as needed.
       Create a Hugging Face secret to download the model [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). Ensure that the token grants access to this model.
-      
+
       Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
 
       ```bash
@@ -46,11 +46,11 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 === "CPU-Based Model Server"
 
       This setup is using the formal `vllm-cpu` image, which according to the documentation can run vLLM on x86 CPU platform.
-      For this setup, we use approximately 9.5GB of memory and 12 CPUs for each replica.  
-      
+      For this setup, we use approximately 9.5GB of memory and 12 CPUs for each replica.
+
       While it is possible to deploy the model server with less resources, this is not recommended. For example, in our tests, loading the model using 8GB of memory and 1 CPU was possible but took almost 3.5 minutes and inference requests took unreasonable time. In general, there is a tradeoff between the memory and CPU we allocate to our pods and the performance. The more memory and CPU we allocate the better performance we can get.
-      
-      After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times. For modifying the allocated resources, adjust the numbers in [cpu-deployment.yaml](https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml) as needed.  
+
+      After running multiple configurations of these values we decided in this sample to use 9.5GB of memory and 12 CPUs for each replica, which gives reasonable response times. You can increase those numbers and potentially may even get better response times. For modifying the allocated resources, adjust the numbers in [cpu-deployment.yaml](https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml) as needed.
 
       Deploy a sample vLLM deployment with the proper protocol to work with the LLM Instance Gateway.
 
@@ -74,8 +74,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 === "Latest Release"
 
       ```bash
-      VERSION=v0.3.0
-      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/$VERSION/manifests.yaml
+      kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/latest/download/manifests.yaml
       ```
 
 === "Dev Version"
@@ -98,13 +97,13 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
    kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool-resources.yaml
    ```
 
-### Deploy Inference Gateway
+### Deploy an Inference Gateway
 
    Choose one of the following options to deploy an Inference Gateway.
 
 === "GKE"
 
-      1. Enable the Gateway API and configure proxy-only subnets when necessary. See [Deploy Gateways](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways) 
+      1. Enable the Gateway API and configure proxy-only subnets when necessary. See [Deploy Gateways](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-gateways)
       for detailed instructions.
 
       1. Deploy Gateway and HealthCheckPolicy resources
@@ -141,7 +140,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
 === "Istio"
 
-      Please note that this feature is currently in an experimental phase and is not intended for production use. 
+      Please note that this feature is currently in an experimental phase and is not intended for production use.
       The implementation and user experience are subject to changes as we continue to iterate on this project.
 
       1.  Requirements
@@ -149,9 +148,9 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
          - Gateway API [CRDs](https://gateway-api.sigs.k8s.io/guides/#installing-gateway-api) installed.
 
       2. Install Istio
-      
+
          ```
-         TAG=1.26-alpha.9befed2f1439d883120f8de70fd70d84ca0ebc3d
+         TAG=1.26-alpha.665da00e1e5392c31cf44cd4dedecd354dd660d5
          # on Linux
          wget https://storage.googleapis.com/istio-build/dev/$TAG/istioctl-$TAG-linux-amd64.tar.gz
          tar -xvf istioctl-$TAG-linux-amd64.tar.gz
diff --git a/site-src/guides/metrics.md b/site-src/guides/metrics.md
index 856c247..b3ebca8 100644
--- a/site-src/guides/metrics.md
+++ b/site-src/guides/metrics.md
@@ -4,23 +4,32 @@ This guide describes the current state of exposed metrics and how to scrape them
 
 ## Requirements
 
-To have response metrics, ensure the body mode is set to `Buffered` or `Streamed` (this should be the default behavior for all implementations).
+=== "EPP"
 
-If you want to include usage metrics for vLLM model server streaming request, send the request with `include_usage`:
+      To have response metrics, ensure the body mode is set to `Buffered` or `Streamed` (this should be the default behavior for all implementations).
+
+      If you want to include usage metrics for vLLM model server streaming request, send the request with `include_usage`:
+
+      ```
+      curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
+      "model": "food-review",
+      "prompt": "whats your fav movie?",
+      "max_tokens": 10,
+      "temperature": 0,
+      "stream": true,
+      "stream_options": {"include_usage": "true"}
+      }'
+      ```
+
+=== "Dynamic LoRA Adapter Sidecar"
+
+      To have response metrics, ensure the vLLM model server is configured with the dynamic LoRA adapter as a sidecar container and a ConfigMap to configure which models to load/unload. See [this doc](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/tools/dynamic-lora-sidecar#example-configuration) for an example.
 
-```
-curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
-"model": "food-review",
-"prompt": "whats your fav movie?",
-"max_tokens": 10,
-"temperature": 0,
-"stream": true,
-"stream_options": {"include_usage": "true"}
-}'
-```
 
 ## Exposed metrics
 
+### EPP
+
 | **Metric name**                              | **Metric Type**  | <div style="width:200px">**Description**</div>  | <div style="width:250px">**Labels**</div>                                          | **Status**  |
 |:---------------------------------------------|:-----------------|:------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:------------|
 | inference_model_request_total                | Counter          | The counter of requests broken out for each model.                | `model_name`=&lt;model-name&gt; <br> `target_model_name`=&lt;target-model-name&gt; | ALPHA       |
@@ -38,10 +47,20 @@ curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
 | inference_pool_ready_pods                    | Gauge            | The number of ready pods for an inference server pool.            | `name`=&lt;inference-pool-name&gt;                                                 | ALPHA       |
 | inference_extension_info                     | Gauge            | The general information of the current build.                     | `commit`=&lt;hash-of-the-build&gt; <br> `build_ref`=&lt;ref-to-the-build&gt;        | ALPHA       |
 
+### Dynamic LoRA Adapter Sidecar
+
+| **Metric name**            | **Metric Type**  | <div style="width:200px">**Description**</div>   | <div style="width:250px">**Labels**</div> | **Status**  |
+|:---------------------------|:-----------------|:-------------------------------------------------|:------------------------------------------|:------------|
+| lora_syncer_adapter_status | Gauge            | Status of LoRA adapters (1=loaded, 0=not_loaded) | `adapter_name`=&lt;adapter-id&gt;         | ALPHA       |
 
 ## Scrape Metrics
 
-Metrics endpoint is exposed at port 9090 by default. To scrape metrics, the client needs a ClusterRole with the following rule:
+The metrics endpoints are exposed on different ports by default:
+
+- EPP exposes the metrics endpoint at port 9090
+- Dynamic LoRA adapter sidecar exposes the metrics endpoint at port 8080
+
+To scrape metrics, the client needs a ClusterRole with the following rule:
 `nonResourceURLs: "/metrics", verbs: get`.
 
 Here is one example if the client needs to mound the secret to act as the service account
@@ -73,9 +92,9 @@ subjects:
   name: inference-gateway-sa-metrics-reader
   namespace: default
 roleRef:
+  apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: inference-gateway-metrics-reader
-  apiGroup: rbac.authorization.k8s.io
 ---
 apiVersion: v1
 kind: Secret
@@ -86,11 +105,78 @@ metadata:
     kubernetes.io/service-account.name: inference-gateway-sa-metrics-reader
 type: kubernetes.io/service-account-token
 ```
-Then, you can curl the 9090 port like following
+
+Then, you can curl the appropriate port as follows. For EPP (port 9090)
+
 ```
 TOKEN=$(kubectl -n default get secret inference-gateway-sa-metrics-reader-secret  -o jsonpath='{.secrets[0].name}' -o jsonpath='{.data.token}' | base64 --decode)
 
 kubectl -n default port-forward inference-gateway-ext-proc-pod-name  9090
 
 curl -H "Authorization: Bearer $TOKEN" localhost:9090/metrics
-```
\ No newline at end of file
+```
+
+## Prometheus Alerts
+
+The section instructs how to configure prometheus alerts using collected metrics.
+
+### Configure alerts
+
+You can follow this [blog post](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/) for instruction of setting up alerts in your monitoring stacks with Prometheus.
+
+A template alert rule is available at [alert.yaml](../../tools/alerts/alert.yaml). You can modify and append these rules to your existing Prometheus deployment.
+
+#### High Inference Request Latency P99
+
+```yaml
+alert: HighInferenceRequestLatencyP99
+expr: histogram_quantile(0.99, rate(inference_model_request_duration_seconds_bucket[5m])) > 10.0 # Adjust threshold as needed (e.g., 10.0 seconds)
+for: 5m
+annotations:
+  title: 'High latency (P99) for model {% raw %}{{ $labels.model_name }}{% endraw %}'
+  description: 'The 99th percentile request duration for model {% raw %}{{ $labels.model_name }}{% endraw %} and target model {% raw %}{{ $labels.target_model_name }}{% endraw %} has been consistently above 10.0 seconds for 5 minutes.'
+labels:
+  severity: 'warning'
+```
+
+#### High Inference Error Rate
+
+```yaml
+alert: HighInferenceErrorRate
+expr: sum by (model_name) (rate(inference_model_request_error_total[5m])) / sum by (model_name) (rate(inference_model_request_total[5m])) > 0.05 # Adjust threshold as needed (e.g., 5% error rate)
+for: 5m
+annotations:
+  title: 'High error rate for model {% raw %}{{ $labels.model_name }}{% endraw %}'
+  description: 'The error rate for model {% raw %}{{ $labels.model_name }}{% endraw %} and target model {% raw %}{{ $labels.target_model_name }}{% endraw %} has been consistently above 5% for 5 minutes.'
+labels:
+  severity: 'critical'
+  impact: 'availability'
+```
+
+#### High Inference Pool Queue Average Size
+
+```yaml
+alert: HighInferencePoolAvgQueueSize
+expr: inference_pool_average_queue_size > 50 # Adjust threshold based on expected queue size
+for: 5m
+annotations:
+  title: 'High average queue size for inference pool {% raw %}{{ $labels.name }}{% endraw %}'
+  description: 'The average number of requests pending in the queue for inference pool {% raw %}{{ $labels.name }}{% endraw %} has been consistently above 50 for 5 minutes.'
+labels:
+  severity: 'critical'
+  impact: 'performance'
+```
+
+#### High Inference Pool Average KV Cache
+
+```yaml
+alert: HighInferencePoolAvgKVCacheUtilization
+expr: inference_pool_average_kv_cache_utilization > 0.9 # 90% utilization
+for: 5m
+annotations:
+  title: 'High KV cache utilization for inference pool {% raw %}{{ $labels.name }}{% endraw %}'
+  description: 'The average KV cache utilization for inference pool {% raw %}{{ $labels.name }}{% endraw %} has been consistently above 90% for 5 minutes, indicating potential resource exhaustion.'
+labels:
+  severity: 'critical'
+  impact: 'resource_exhaustion'
+```
diff --git a/site-src/guides/serve-multiple-genai-models.md b/site-src/guides/serve-multiple-genai-models.md
index 2c6d8a5..4eb120a 100644
--- a/site-src/guides/serve-multiple-genai-models.md
+++ b/site-src/guides/serve-multiple-genai-models.md
@@ -2,11 +2,14 @@
 A company wants to deploy multiple large language models (LLMs) to serve different workloads. 
 For example, they might want to deploy a Gemma3 model for a chatbot interface and a Deepseek model for a recommendation application. 
 The company needs to ensure optimal serving performance for these LLMs.
-Using Gateway API Inference Extension, you can deploy these LLMs on your cluster with your chosen accelerator configuration in an `InferencePool`. 
+By using an Inference Gateway, you can deploy these LLMs on your cluster with your chosen accelerator configuration in an `InferencePool`. 
 You can then route requests based on the model name (such as "chatbot" and "recommender") and the `Criticality` property.
 
 ## How
-The following diagram illustrates how Gateway API Inference Extension routes requests to different models based on the model name.
+The following diagram illustrates how an Inference Gateway routes requests to different models based on the model name.
+The model name is extracted by [Body-Based routing](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/bbr/README.md)
+ from the request body to the header. The header is then matched to dispatch
+ requests to different `InferencePool` (and their EPPs) instances.
 ![Serving multiple generative AI models](../images/serve-mul-gen-AI-models.png)
 
 This example illustrates a conceptual example regarding how to use the `HTTPRoute` object to route based on model name like “chatbot” or “recommender” to `InferencePool`.
@@ -63,9 +66,9 @@ curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
 3. Send a few requests to model "recommender" as follows:
 ```bash
 curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
-"model": "chatbot",
+"model": "recommender",
 "prompt": "Give me restaurant recommendations in Paris",
 "max_tokens": 100,
 "temperature": 0
 }'
-```
\ No newline at end of file
+```
diff --git a/site-src/guides/serve-multiple-lora-adapters.md b/site-src/guides/serve-multiple-lora-adapters.md
index b471faef..d1090b3 100644
--- a/site-src/guides/serve-multiple-lora-adapters.md
+++ b/site-src/guides/serve-multiple-lora-adapters.md
@@ -1,11 +1,11 @@
 # Serve LoRA adapters on a shared pool
 A company wants to serve LLMs for document analysis and focuses on audiences in multiple languages, such as English and Spanish.
 They have a fine-tuned LoRA adapter for each language, but need to efficiently use their GPU and TPU capacity.
-You can use Gateway API Inference Extension to deploy dynamic LoRA fine-tuned adapters for each language (for example, `english-bot` and `spanish-bot`) on a common base model and accelerator.
+You can use an Inference Gateway to deploy dynamic LoRA fine-tuned adapters for each language (for example, `english-bot` and `spanish-bot`) on a common base model and accelerator.
 This lets you reduce the number of required accelerators by densely packing multiple models in a shared pool.
 
 ## How
-The following diagram illustrates how Gateway API Inference Extension serves multiple LoRA adapters on a shared pool.
+The following diagram illustrates how Inference Gateway serves multiple LoRA adapters on a shared pool.
 ![Serving LoRA adapters on a shared pool](../images/serve-LoRA-adapters.png)
 This example illustrates how you can densely serve multiple LoRA adapters with distinct workload performance objectives on a common InferencePool.
 ```yaml
diff --git a/site-src/index.md b/site-src/index.md
index b304366..7a2a116 100644
--- a/site-src/index.md
+++ b/site-src/index.md
@@ -1,8 +1,6 @@
 # Introduction
 
-Gateway API Inference Extension is an official Kubernetes project focused on
-extending [Gateway API](https://gateway-api.sigs.k8s.io/) with inference
-specific routing extensions.
+Gateway API Inference Extension is an official Kubernetes project that optimizes self-hosting Generative Models on Kubernetes.
 
 The overall resource model focuses on 2 new inference-focused
 [personas](/concepts/roles-and-personas) and corresponding resources that
@@ -11,20 +9,49 @@ they are expected to manage:
 <!-- Source: https://docs.google.com/presentation/d/11HEYCgFi-aya7FS91JvAfllHiIlvfgcp7qpi_Azjk4E/edit#slide=id.g292839eca6d_1_0 -->
 <img src="/images/resource-model.png" alt="Gateway API Inference Extension Resource Model" class="center" width="550" />
 
+## Concepts and Definitions
+
+The following specific terms to this project:
+
+- **Inference Gateway**: A proxy/load-balancer that has been coupled with the
+  EndPointer Picker extension. It provides optimized routing and load balancing for
+  serving Kubernetes self-hosted generative Artificial Intelligence (AI)
+  workloads. It simplifies the deployment, management, and observability of AI
+  inference workloads.
+- **Inference Scheduler**: An extendable component that makes decisions about which endpoint is optimal (best cost /
+  best performance) for an inference request based on `Metrics and Capabilities`
+  from [Model Serving](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/003-model-server-protocol/README.md).
+- **Metrics and Capabilities**: Data provided by model serving platforms about
+  performance, availability and capabilities to optimize routing. Includes
+  things like [Prefix Cache] status or [LoRA Adapters] availability.
+- **Endpoint Picker(EPP)**: An implementation of an `Inference Scheduler` with additional Routing, Flow, and Request Control layers to allow for sophisticated routing strategies. Additional info on the architecture of the EPP [here](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/0683-epp-architecture-proposal).
+
+[Inference Gateway]:#concepts-and-definitions
+
 ## Key Features 
-Gateway API Inference Extension, along with a reference implementation in Envoy Proxy, provides the following key features: 
+Gateway API Inference Extension optimizes self-hosting Generative AI Models on Kubernetes.
+It provides optimized load-balancing for self-hosted Generative AI Models on Kubernetes.
+The project’s goal is to improve and standardize routing to inference workloads across the ecosystem.
+
+This is achieved by leveraging Envoy's [External Processing](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter) to extend any gateway that supports both ext-proc and [Gateway API](https://github.com/kubernetes-sigs/gateway-api) into an [inference gateway](#concepts-and-definitions).
+This extension extends popular gateways like Envoy Gateway, kgateway, and GKE Gateway - to become [Inference Gateway](#concepts-and-definitions) -
+supporting inference platform teams self-hosting Generative Models (with a current focus on large language models) on Kubernetes.
+This integration makes it easy to expose and control access to your local [OpenAI-compatible chat completion endpoints](https://platform.openai.com/docs/api-reference/chat)
+to other workloads on or off cluster, or to integrate your self-hosted models alongside model-as-a-service providers
+in a higher level **AI Gateways** like [LiteLLM](https://www.litellm.ai/), [Gloo AI Gateway](https://www.solo.io/products/gloo-ai-gateway), or [Apigee](https://cloud.google.com/apigee).
 
-- **Model-aware routing**: Instead of simply routing based on the path of the request, Gateway API Inference Extension allows you to route to models based on the model names. This is enabled by support for GenAI Inference API specifications (such as OpenAI API) in the gateway implementations such as in Envoy Proxy. This model-aware routing also extends to Low-Rank Adaptation (LoRA) fine-tuned models.
 
-- **Serving priority**: Gateway API Inference Extension allows you to specify the serving priority of your models. For example, you can specify that your models for online inference of chat tasks (which is more latency sensitive) have a higher [*Criticality*](/reference/spec/#criticality) than a model for latency tolerant tasks such as a summarization. 
+- **Model-aware routing**: Instead of simply routing based on the path of the request, an **[inference gateway]** allows you to route to models based on the model names. This is enabled by support for GenAI Inference API specifications (such as OpenAI API) in the gateway implementations such as in Envoy Proxy. This model-aware routing also extends to Low-Rank Adaptation (LoRA) fine-tuned models.
 
-- **Model rollouts**:  Gateway API Inference Extension allows you to incrementally roll out new model versions by traffic splitting definitions based on the model names. 
+- **Serving priority**: an **[inference gateway]** allows you to specify the serving priority of your models. For example, you can specify that your models for online inference of chat tasks (which is more latency sensitive) have a higher [*Criticality*](/reference/spec/#criticality) than a model for latency tolerant tasks such as a summarization. 
 
-- **Extensibility for Inference Services**: Gateway API Inference Extension defines extensibility pattern for additional Inference services to create bespoke routing capabilities should out of the box solutions not fit your needs.
+- **Model rollouts**:  an **[inference gateway]** allows you to incrementally roll out new model versions by traffic splitting definitions based on the model names. 
 
+- **Extensibility for Inference Services**: an **[inference gateway]** defines extensibility pattern for additional Inference services to create bespoke routing capabilities should out of the box solutions not fit your needs.
 
-- **Customizable Load Balancing for Inference**: Gateway API Inference Extension defines a pattern for customizable load balancing and request routing that is optimized for Inference. Gateway API Inference Extension provides a reference implementation of model endpoint picking leveraging metrics emitted from the model servers. This endpoint picking mechanism can be used in lieu of traditional load balancing mechanisms. Model Server-aware load balancing ("smart" load balancing as its sometimes referred to in this repo) has been proven to reduce the serving latency and improve utilization of accelerators in your clusters.
+- **Customizable Load Balancing for Inference**: an **[inference gateway]** defines a pattern for customizable load balancing and request routing that is optimized for Inference. An **[inference gateway]** provides a reference implementation of model endpoint picking leveraging metrics emitted from the model servers. This endpoint picking mechanism can be used in lieu of traditional load balancing mechanisms. Model Server-aware load balancing ("smart" load balancing as its sometimes referred to in this repo) has been proven to reduce the serving latency and improve utilization of accelerators in your clusters.
 
+By achieving these, the project aims to reduce latency and improve accelerator (GPU) utilization for AI workloads.
 
 ## API Resources
 
@@ -42,7 +69,7 @@ that are relevant to this project:
 Gateway API has [more than 25
 implementations](https://gateway-api.sigs.k8s.io/implementations/). As this
 pattern stabilizes, we expect a wide set of these implementations to support
-this project.
+this project to become an **[inference gateway]**
 
 ### Endpoint Picker
 
@@ -71,16 +98,16 @@ to any Gateway API users or implementers.
 2. If the request should be routed to an InferencePool, the Gateway will forward
 the request information to the endpoint selection extension for that pool.
 
-3. The extension will fetch metrics from whichever portion of the InferencePool
+3. The inference gateway will fetch metrics from whichever portion of the InferencePool
 endpoints can best achieve the configured objectives. Note that this kind of
-metrics probing may happen asynchronously, depending on the extension.
+metrics probing may happen asynchronously, depending on the inference gateway.
 
-4. The extension will instruct the Gateway which endpoint the request should be
+4. The inference gateway will instruct the Gateway which endpoint the request should be
 routed to.
 
 5. The Gateway will route the request to the desired endpoint.
 
-<img src="/images/request-flow.png" alt="Gateway API Inference Extension Request Flow" class="center" />
+<img src="/images/request-flow.png" alt="Inference Gateway Request Flow" class="center" />
 
 
 ## Who is working on Gateway API Inference Extension?
diff --git a/site-src/reference/spec.md b/site-src/reference/spec.md
index d8e0c95..263f54d 100644
--- a/site-src/reference/spec.md
+++ b/site-src/reference/spec.md
@@ -211,7 +211,7 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `modelName` _string_ | ModelName is the name of the model as it will be set in the "model" parameter for an incoming request.<br />ModelNames must be unique for a referencing InferencePool<br />(names can be reused for a different pool in the same cluster).<br />The modelName with the oldest creation timestamp is retained, and the incoming<br />InferenceModel is sets the Ready status to false with a corresponding reason.<br />In the rare case of a race condition, one Model will be selected randomly to be considered valid, and the other rejected.<br />Names can be reserved without an underlying model configured in the pool.<br />This can be done by specifying a target model and setting the weight to zero,<br />an error will be returned specifying that no valid target model is found. |  | MaxLength: 256 <br />Required: \{\} <br /> |
+| `modelName` _string_ | ModelName is the name of the model as it will be set in the "model" parameter for an incoming request.<br />ModelNames must be unique for a referencing InferencePool<br />(names can be reused for a different pool in the same cluster).<br />The modelName with the oldest creation timestamp is retained, and the incoming<br />InferenceModel's Ready status is set to false with a corresponding reason.<br />In the rare case of a race condition, one Model will be selected randomly to be considered valid, and the other rejected.<br />Names can be reserved without an underlying model configured in the pool.<br />This can be done by specifying a target model and setting the weight to zero,<br />an error will be returned specifying that no valid target model is found. |  | MaxLength: 256 <br />Required: \{\} <br /> |
 | `criticality` _[Criticality](#criticality)_ | Criticality defines how important it is to serve the model compared to other models referencing the same pool.<br />Criticality impacts how traffic is handled in resource constrained situations. It handles this by<br />queuing or rejecting requests of lower criticality. InferenceModels of an equivalent Criticality will<br />fairly share resources over throughput of tokens. In the future, the metric used to calculate fairness,<br />and the proportionality of fairness will be configurable.<br />Default values for this field will not be set, to allow for future additions of new field that may 'one of' with this field.<br />Any implementations that may consume this field may treat an unset value as the 'Standard' range. |  | Enum: [Critical Standard Sheddable] <br /> |
 | `targetModels` _[TargetModel](#targetmodel) array_ | TargetModels allow multiple versions of a model for traffic splitting.<br />If not specified, the target model name is defaulted to the modelName parameter.<br />modelName is often in reference to a LoRA adapter. |  | MaxItems: 10 <br /> |
 | `poolRef` _[PoolObjectReference](#poolobjectreference)_ | PoolRef is a reference to the inference pool, the pool must exist in the same namespace. |  | Required: \{\} <br /> |
diff --git a/tools/alerts/alert.yaml b/tools/alerts/alert.yaml
new file mode 100644
index 0000000..c712207
--- /dev/null
+++ b/tools/alerts/alert.yaml
@@ -0,0 +1,38 @@
+groups:
+- name: gateway-api-inference-extension
+  rules:
+  - alert: HighInferenceRequestLatencyP99
+    annotations:
+      title: 'High latency (P99) for model {{ $labels.model_name }}'
+      description: 'The 99th percentile request duration for model {{ $labels.model_name }} and target model {{ $labels.target_model_name }} has been consistently above 10.0 seconds for 5 minutes.'
+    expr: histogram_quantile(0.99, rate(inference_model_request_duration_seconds_bucket[5m])) > 10.0
+    for: 5m
+    labels:
+      severity: 'warning'
+  - alert: HighInferenceErrorRate
+    annotations:
+      title: 'High error rate for model {{ $labels.model_name }}'
+      description: 'The error rate for model {{ $labels.model_name }} and target model {{ $labels.target_model_name }} has been consistently above 5% for 5 minutes.'
+    expr: sum by (model_name) (rate(inference_model_request_error_total[5m])) / sum by (model_name) (rate(inference_model_request_total[5m])) > 0.05
+    for: 5m
+    labels:
+      severity: 'critical'
+      impact: 'availability'
+  - alert: HighInferencePoolAvgQueueSize
+    annotations:
+      title: 'High average queue size for inference pool {{ $labels.name }}'
+      description: 'The average number of requests pending in the queue for inference pool {{ $labels.name }} has been consistently above 50 for 5 minutes.'
+    expr: inference_pool_average_queue_size > 50
+    for: 5m
+    labels:
+      severity: 'critical'
+      impact: 'performance'
+  - alert: HighInferencePoolAvgKVCacheUtilization
+    annotations:
+      title: 'High KV cache utilization for inference pool {{ $labels.name }}'
+      description: 'The average KV cache utilization for inference pool {{ $labels.name }} has been consistently above 90% for 5 minutes, indicating potential resource exhaustion.'
+    expr: inference_pool_average_kv_cache_utilization > 0.9
+    for: 5m
+    labels:
+      severity: 'critical'
+      impact: 'resource_exhaustion'
diff --git a/tools/dashboards/README.md b/tools/dashboards/README.md
index 816bd2e..21282bf 100644
--- a/tools/dashboards/README.md
+++ b/tools/dashboards/README.md
@@ -4,7 +4,7 @@ This documentation provides instructions for setting up grafana dashboards to se
 
 ## Requirements
 
-Please follow [metrics](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp/metrics) page to configure the proxy to enable all metrics.
+Please follow [metrics](https://gateway-api-inference-extension.sigs.k8s.io/guides/metrics/?h=metrics) page to configure the proxy to enable all metrics.
 
 ## Load Inference Extension dashboard into Grafana
 
diff --git a/tools/dynamic-lora-sidecar/Dockerfile b/tools/dynamic-lora-sidecar/Dockerfile
index 4faf360..14703d1 100644
--- a/tools/dynamic-lora-sidecar/Dockerfile
+++ b/tools/dynamic-lora-sidecar/Dockerfile
@@ -1,4 +1,4 @@
-FROM python:3.9-slim-buster AS test
+FROM python:3.10-slim-buster AS test
 
 WORKDIR /dynamic-lora-reconciler-test
 COPY requirements.txt .
diff --git a/tools/dynamic-lora-sidecar/Makefile b/tools/dynamic-lora-sidecar/Makefile
new file mode 100644
index 0000000..d498750
--- /dev/null
+++ b/tools/dynamic-lora-sidecar/Makefile
@@ -0,0 +1,33 @@
+# Makefile for dynamic-lora-sidecar
+
+PYTHON_VERSION := 3.10
+VENV_DIR := venv
+PYTHON := $(VENV_DIR)/bin/python
+PIP := $(VENV_DIR)/bin/pip
+
+.PHONY: help venv install test clean
+
+help: ## Show available targets
+	@echo "Available targets:"
+	@echo "  venv     - Create virtual environment"
+	@echo "  install  - Install dependencies"
+	@echo "  test     - Run unit tests"
+	@echo "  clean    - Clean up virtual environment"
+
+venv: $(VENV_DIR)/bin/activate ## Create virtual environment
+
+$(VENV_DIR)/bin/activate:
+	python$(PYTHON_VERSION) -m venv $(VENV_DIR)
+
+install: venv ## Install dependencies
+	$(PIP) install --upgrade pip
+	$(PIP) install -r requirements.txt
+
+test: install ## Run unit tests
+	$(PYTHON) -m unittest discover -v -s sidecar
+
+clean: ## Clean up virtual environment
+	rm -rf $(VENV_DIR)
+	rm -rf .pytest_cache
+	find . -name "*.pyc" -delete
+	find . -name "__pycache__" -type d -exec rm -rf {} +
diff --git a/tools/dynamic-lora-sidecar/README.md b/tools/dynamic-lora-sidecar/README.md
index 65dc0d7..f50c783 100644
--- a/tools/dynamic-lora-sidecar/README.md
+++ b/tools/dynamic-lora-sidecar/README.md
@@ -1,6 +1,6 @@
 # Dynamic LORA Adapter Sidecar for vLLM
 
-This is a sidecar-based tool to help rolling out new LoRA adapters to a set of running vLLM model servers. The user deploys the sidecar with a vLLM server, and using a ConfigMap, the user can express their intent as to which LoRA adapters they want to have the running vLLM servers to be configure with. The sidecar watches the ConfigMap and sends load/unload requests to the vLLM container to actuate on the user intent. 
+This is a sidecar-based tool to help rolling out new LoRA adapters to a set of running vLLM model servers. The user deploys the sidecar with a vLLM server, and using a ConfigMap, the user can express their intent as to which LoRA adapters they want to have the running vLLM servers to be configure with. The sidecar watches the ConfigMap and sends load/unload requests to the vLLM container to actuate on the user intent.
 
 ## Overview
 
@@ -48,6 +48,17 @@ The sidecar uses the vLLM server's API to load or unload adapters based on the c
    ```
    Do not use subPath, since configmap updates are not reflected in the file
 
+## Development
+
+For local development and testing, use the provided Makefile:
+
+```bash
+make venv      # Create Python 3.10 virtual environment
+make install   # Install dependencies
+make test      # Run unit tests
+make clean     # Clean up
+```
+
 ## Command Line Arguments
 
 The sidecar supports the following command-line arguments:
@@ -59,7 +70,7 @@ The sidecar supports the following command-line arguments:
 - `--config-validation`: Enable config validation (default: True)
 
 ## Configuration Fields
-- `vLLMLoRAConfig`[**required**]  base key 
+- `vLLMLoRAConfig`[**required**]  base key
 - `host` [*optional*] Model server's host. defaults to localhost
 - `port` [*optional*] Model server's port. defaults to 8000
 - `name` [*optional*] Name of this config
@@ -121,6 +132,9 @@ spec:
       - name: reconciler
         image: your-image:tag
         command: ["python", "sidecar.py", "--health-check-timeout", "600", "--health-check-interval", "5", "--reconcile-trigger", "10"] #optional if overriding default values
+        ports:
+        - containerPort: 8080
+          name: metrics
         volumeMounts:
         - name: config-volume
           mountPath: /config
diff --git a/tools/dynamic-lora-sidecar/deployment.yaml b/tools/dynamic-lora-sidecar/deployment.yaml
index 0c0c178..dd1b6bb 100644
--- a/tools/dynamic-lora-sidecar/deployment.yaml
+++ b/tools/dynamic-lora-sidecar/deployment.yaml
@@ -69,7 +69,10 @@ spec:
           image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/lora-syncer:main
           restartPolicy: Always
           imagePullPolicy: Always
-          env: 
+          ports:
+          - containerPort: 8080
+            name: metrics
+          env:
             - name: DYNAMIC_LORA_ROLLOUT_CONFIG
               value: "/config/configmap.yaml"
           volumeMounts: # DO NOT USE subPath
diff --git a/tools/dynamic-lora-sidecar/requirements.txt b/tools/dynamic-lora-sidecar/requirements.txt
index 1f9b927..66a76bc 100644
--- a/tools/dynamic-lora-sidecar/requirements.txt
+++ b/tools/dynamic-lora-sidecar/requirements.txt
@@ -1,6 +1,7 @@
-aiohttp
-jsonschema
-pyyaml
-requests 
-watchfiles
-watchdog
\ No newline at end of file
+aiohttp==3.12.12
+jsonschema==4.24.0
+prometheus_client==0.22.1
+PyYAML==6.0.2
+requests==2.32.4
+watchfiles==1.0.5
+watchdog==6.0.0
diff --git a/tools/dynamic-lora-sidecar/sidecar/sidecar.py b/tools/dynamic-lora-sidecar/sidecar/sidecar.py
index 4e71670..a78f29d 100644
--- a/tools/dynamic-lora-sidecar/sidecar/sidecar.py
+++ b/tools/dynamic-lora-sidecar/sidecar/sidecar.py
@@ -24,9 +24,17 @@ import logging
 import datetime
 import os
 import sys
+from prometheus_client import Gauge, start_http_server
 from watchdog.observers.polling import PollingObserver as Observer
 from watchdog.events import FileSystemEventHandler
 
+# Initialize Prometheus metrics
+ADAPTER_STATUS_METRICS = Gauge(
+    'lora_syncer_adapter_status',
+    'Status of LoRA adapters (1=loaded, 0=not_loaded)',
+    ['adapter_name']
+)
+
 CONFIG_MAP_FILE = os.environ.get(
     "DYNAMIC_LORA_ROLLOUT_CONFIG", "/config/configmap.yaml"
 )
@@ -58,6 +66,8 @@ def parse_arguments():
                         help=f'Path to config map file (default: {CONFIG_MAP_FILE})')
     parser.add_argument('--config-validation', action='store_true', default=True,
                         help='Enable config validation (default: True)')
+    parser.add_argument('--metrics-port', type=int, default=8080,
+                        help='Port to listen for Prometheus metrics (default: 8080)')
     return parser.parse_args()
 
 
@@ -226,7 +236,7 @@ class LoraReconciler:
             time.sleep(self.health_check_interval.seconds)
         return False
 
-    def load_adapter(self, adapter: LoraAdapter):
+    def load_adapter(self, adapter: LoraAdapter) -> None | str:
         """Sends a request to load the specified model."""
         if adapter in self.registered_adapters:
             logging.info(
@@ -243,10 +253,12 @@ class LoraReconciler:
             response = requests.post(url, json=payload)
             response.raise_for_status()
             logging.info(f"loaded model {adapter.id}")
+            return None
         except requests.exceptions.RequestException as e:
             logging.error(f"error loading model {adapter.id}: {e}")
+            return f"error loading model {adapter.id}: {e}"
 
-    def unload_adapter(self, adapter: LoraAdapter):
+    def unload_adapter(self, adapter: LoraAdapter) -> None | str:
         """Sends a request to unload the specified model."""
         if adapter not in self.registered_adapters:
             logging.info(
@@ -284,20 +296,30 @@ class LoraReconciler:
         adapters_to_load_id = ", ".join(str(a.id) for a in adapters_to_load)
         logging.info(f"adapter to load {adapters_to_load_id}")
         for adapter in adapters_to_load:
-            self.load_adapter(adapter)
+            err = self.load_adapter(adapter)
+            if err is None:
+                self.update_adapter_status_metrics(adapter.id, is_loaded=True)
         adapters_to_unload = self.ensure_not_exist_adapters - self.ensure_exist_adapters
         adapters_to_unload_id = ", ".join(str(a.id) for a in adapters_to_unload)
         logging.info(f"adapters to unload {adapters_to_unload_id}")
         for adapter in adapters_to_unload:
-            self.unload_adapter(adapter)
+            err = self.unload_adapter(adapter)
+            if err is None:
+                self.update_adapter_status_metrics(adapter.id, is_loaded=False)
+
+    def update_adapter_status_metrics(self, adapter_id: str, is_loaded: bool):
+        """Update adapter status metrics"""
+        status = 1 if is_loaded else 0
+        ADAPTER_STATUS_METRICS.labels(adapter_name=adapter_id).set(status)
+
 
 
 async def main():
     args = parse_arguments()
-    
+
     # Update CONFIG_MAP_FILE with argument value
     config_file = args.config
-    
+
     reconciler_instance = LoraReconciler(
         config_file=config_file,
         health_check_timeout=args.health_check_timeout,
@@ -305,7 +327,11 @@ async def main():
         reconcile_trigger_seconds=args.reconcile_trigger,
         config_validation=args.config_validation
     )
-    
+
+    # Start metrics server
+    logging.info(f"Starting metrics server on port {args.metrics_port}")
+    start_http_server(args.metrics_port)
+
     logging.info(f"Running initial reconcile for config map {config_file}")
     reconciler_instance.reconcile()
 
diff --git a/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py b/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
index 90a1101..bd6c7ed 100644
--- a/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
+++ b/tools/dynamic-lora-sidecar/sidecar/test_sidecar.py
@@ -17,7 +17,7 @@ from unittest.mock import patch, Mock, mock_open, call
 import yaml
 import os
 import datetime
-from sidecar import LoraReconciler, LoraAdapter, CONFIG_MAP_FILE, BASE_FIELD
+from sidecar import LoraReconciler, LoraAdapter, CONFIG_MAP_FILE, BASE_FIELD, ADAPTER_STATUS_METRICS
 
 # Update TEST_CONFIG_DATA to include the new configuration parameters
 TEST_CONFIG_DATA = {
@@ -227,12 +227,55 @@ class LoraReconcilerTest(unittest.TestCase):
             reconcile_trigger_seconds=45,
             config_validation=False
         )
-        
+
         # Check that values are properly set
         self.assertEqual(reconciler.health_check_timeout, datetime.timedelta(seconds=240))
         self.assertEqual(reconciler.health_check_interval, datetime.timedelta(seconds=15))
         self.assertEqual(reconciler.reconcile_trigger_seconds, 45)
 
+    def test_update_adapter_status_metrics(self):
+        """Test that update_adapter_status_metrics method works correctly"""
+        # Clear any existing metrics
+        ADAPTER_STATUS_METRICS.clear()
+
+        # Create reconciler
+        reconciler = LoraReconciler(
+            config_file=CONFIG_MAP_FILE,
+            health_check_timeout=180,
+            health_check_interval=10,
+            reconcile_trigger_seconds=30,
+            config_validation=False
+        )
+
+        # Test setting loaded status
+        reconciler.update_adapter_status_metrics("test-adapter-1", is_loaded=True)
+        reconciler.update_adapter_status_metrics("test-adapter-2", is_loaded=False)
+
+        # Get all metric samples
+        metric_samples = list(ADAPTER_STATUS_METRICS.collect())[0].samples
+
+        # Check that metrics were set correctly
+        adapter_metrics = {}
+        for sample in metric_samples:
+            adapter_name = sample.labels['adapter_name']
+            adapter_metrics[adapter_name] = sample.value
+
+        self.assertEqual(adapter_metrics.get('test-adapter-1'), 1.0, "test-adapter-1 should be marked as loaded")
+        self.assertEqual(adapter_metrics.get('test-adapter-2'), 0.0, "test-adapter-2 should be marked as not loaded")
+
+    def test_metrics_endpoint(self):
+        """Test that Prometheus metrics can be collected"""
+        from prometheus_client import generate_latest
+
+        # Clear metrics and set a test value
+        ADAPTER_STATUS_METRICS.clear()
+        ADAPTER_STATUS_METRICS.labels(adapter_name='test-adapter').set(1)
+
+        # Test that generate_latest produces valid output
+        metrics_bytes = generate_latest()
+        metrics = metrics_bytes.decode('utf-8')
+        self.assertIn('lora_syncer_adapter_status{adapter_name="test-adapter"} 1.0', metrics)
+
 
 if __name__ == "__main__":
     unittest.main()



Test patch:
diff --git a/conformance/testing-epp/plugins/filter/filter_test.go b/conformance/testing-epp/plugins/filter/filter_test.go
index 6e5f221..2c00821 100644
--- a/conformance/testing-epp/plugins/filter/filter_test.go
+++ b/conformance/testing-epp/plugins/filter/filter_test.go
@@ -79,11 +79,45 @@ func TestFilter(t *testing.T) {
 				},
 			},
 		},
+		{
+			name:   "TestHeaderBasedFilter, multiple header endpoints set and multiple matches",
+			req:    &types.LLMRequest{Headers: map[string]string{headerTestEppEndPointSelectionKey: "test-endpoint3,test-endpoint2"}},
+			filter: &HeaderBasedTestingFilter{},
+			input: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{
+						Address: "test-endpoint1",
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{
+						Address: "test-endpoint2",
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{
+						Address: "test-endpoint3",
+					},
+				},
+			},
+			output: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{
+						Address: "test-endpoint3",
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{
+						Address: "test-endpoint2",
+					},
+				},
+			},
+		},
 	}
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			got := test.filter.Filter(context.Background(), test.req, types.NewCycleState(), test.input)
+			got := test.filter.Filter(context.Background(), types.NewCycleState(), test.req, test.input)
 
 			if diff := cmp.Diff(test.output, got); diff != "" {
 				t.Errorf("Unexpected output (-want +got): %v", diff)
diff --git a/conformance/testing-epp/plugins/filter/request_header_based_filter.go b/conformance/testing-epp/plugins/filter/request_header_based_filter.go
index a3d63c2..41194b5 100644
--- a/conformance/testing-epp/plugins/filter/request_header_based_filter.go
+++ b/conformance/testing-epp/plugins/filter/request_header_based_filter.go
@@ -18,20 +18,25 @@ package filter
 
 import (
 	"context"
+	"strings"
 
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
 const (
+	// headerTestEppEndPointSelectionKey is the header used for testing purposes to make EPP behavior controllable.
+	// The header value should be a comma-separated list of endpoint IP addresses.
+	// E.g., "test-epp-endpoint-selection": "10.0.0.7,10.0.0.8"
+	// The returned order is the same as the order provided in the header.
 	headerTestEppEndPointSelectionKey = "test-epp-endpoint-selection"
 )
 
 // compile-time type assertion
 var _ framework.Filter = &HeaderBasedTestingFilter{}
 
-// NewHeaderBasedTestingFilter initializes a new HeaderBasedTestingFilter and returns its pointer.
-// This should be only used in testing purpose.
+// NewHeaderBasedTestingFilter initializes a new HeaderBasedTestingFilter.
+// This should only be used for testing purposes.
 func NewHeaderBasedTestingFilter() *HeaderBasedTestingFilter {
 	return &HeaderBasedTestingFilter{}
 }
@@ -39,22 +44,28 @@ func NewHeaderBasedTestingFilter() *HeaderBasedTestingFilter {
 // HeaderBasedTestingFilter filters Pods based on an address specified in the "test-epp-endpoint-selection" request header.
 type HeaderBasedTestingFilter struct{}
 
-// Name returns the name of the filter.
-func (f *HeaderBasedTestingFilter) Name() string {
-	return "test-header-based"
+// Type returns the type of the filter.
+func (f *HeaderBasedTestingFilter) Type() string {
+	return "header-based-testing"
 }
 
-// Filter filters out pods that doesn't meet the filter criteria.
-func (f *HeaderBasedTestingFilter) Filter(_ context.Context, request *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
-	filteredPods := []types.Pod{}
-
-	endPointInReqeust, found := request.Headers[headerTestEppEndPointSelectionKey]
-	if !found {
-		return filteredPods
+// Filter selects pods that match the IP addresses specified in the request header.
+func (f *HeaderBasedTestingFilter) Filter(_ context.Context, _ *types.CycleState, request *types.LLMRequest, pods []types.Pod) []types.Pod {
+	headerValue, ok := request.Headers[headerTestEppEndPointSelectionKey]
+	if !ok || headerValue == "" {
+		return []types.Pod{}
 	}
 
+	podAddressMap := make(map[string]types.Pod, len(pods))
 	for _, pod := range pods {
-		if pod.GetPod().Address == endPointInReqeust {
+		podAddressMap[pod.GetPod().Address] = pod
+	}
+
+	endpoints := strings.Split(headerValue, ",")
+	filteredPods := make([]types.Pod, 0, len(endpoints))
+	for _, endpoint := range endpoints {
+		trimmedEndpoint := strings.TrimSpace(endpoint)
+		if pod, found := podAddressMap[trimmedEndpoint]; found {
 			filteredPods = append(filteredPods, pod)
 		}
 	}
diff --git a/conformance/testing-epp/scheduler.go b/conformance/testing-epp/scheduler.go
index 4035e25..94f9ee5 100644
--- a/conformance/testing-epp/scheduler.go
+++ b/conformance/testing-epp/scheduler.go
@@ -21,14 +21,17 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
-	profilepicker "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile-picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
 )
 
 // NewReqHeaderBasedScheduler creates a scheduler for conformance tests that selects
 // an endpoint based on the "test-epp-endpoint-selection" request header. If the
 // header is missing or the specified endpoint doesn't exist, no endpoint is returned.
-func NewReqHeaderBasedScheduler(datastore scheduling.Datastore) *scheduling.Scheduler {
-	predicatableSchedulerProfile := framework.NewSchedulerProfile().WithFilters(filter.NewHeaderBasedTestingFilter()).WithPicker(picker.NewMaxScorePicker())
-	return scheduling.NewSchedulerWithConfig(datastore, scheduling.NewSchedulerConfig(
-		profilepicker.NewAllProfilesPicker(), map[string]*framework.SchedulerProfile{"req-header-based-profile": predicatableSchedulerProfile}))
+func NewReqHeaderBasedScheduler() *scheduling.Scheduler {
+	predicatableSchedulerProfile := framework.NewSchedulerProfile().
+		WithFilters(filter.NewHeaderBasedTestingFilter()).
+		WithPicker(picker.NewMaxScorePicker())
+
+	return scheduling.NewSchedulerWithConfig(scheduling.NewSchedulerConfig(
+		profile.NewSingleProfileHandler(), map[string]*framework.SchedulerProfile{"req-header-based-profile": predicatableSchedulerProfile}))
 }
diff --git a/conformance/testing-epp/sheduler_test.go b/conformance/testing-epp/sheduler_test.go
index 105422e..4901e03 100644
--- a/conformance/testing-epp/sheduler_test.go
+++ b/conformance/testing-epp/sheduler_test.go
@@ -31,13 +31,13 @@ import (
 func TestSchedule(t *testing.T) {
 	tests := []struct {
 		name    string
-		input   []*backendmetrics.FakePodMetrics
+		input   []backendmetrics.PodMetrics
 		req     *types.LLMRequest
-		wantRes map[string]*types.Result
+		wantRes *types.SchedulingResult
 		err     bool
 	}{
 		{
-			name: "no pods in datastore and req header is set",
+			name: "no candidate pods and req header is set",
 			req: &types.LLMRequest{
 				Headers:   map[string]string{"test-epp-endpoint-selection": "random-endpoint"},
 				RequestId: uuid.NewString(),
@@ -47,8 +47,8 @@ func TestSchedule(t *testing.T) {
 		},
 		{
 			name: "req header not set",
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{Address: "random-endpoint"}},
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{Address: "random-endpoint"}},
 			},
 			req: &types.LLMRequest{
 				Headers:   map[string]string{}, // Deliberately set an empty header.
@@ -58,9 +58,9 @@ func TestSchedule(t *testing.T) {
 			err:     true,
 		},
 		{
-			name: "no pods address in datastore matches req header address",
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{Address: "nonmatched-endpoint"}},
+			name: "no pods address from the candidate pods matches req header address",
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{Address: "nonmatched-endpoint"}},
 			},
 			req: &types.LLMRequest{
 				Headers:   map[string]string{"test-epp-endpoint-selection": "matched-endpoint"},
@@ -70,34 +70,37 @@ func TestSchedule(t *testing.T) {
 			err:     true,
 		},
 		{
-			name: "one pod address in datastore matches req header address",
-			input: []*backendmetrics.FakePodMetrics{
-				{Pod: &backend.Pod{Address: "nonmatched-endpoint"}},
-				{Pod: &backend.Pod{Address: "matched-endpoint"}},
+			name: "one pod address from the candidate pods matches req header address",
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{Address: "nonmatched-endpoint"}},
+				&backendmetrics.FakePodMetrics{Pod: &backend.Pod{Address: "matched-endpoint"}},
 			},
 			req: &types.LLMRequest{
 				Headers:   map[string]string{"test-epp-endpoint-selection": "matched-endpoint"},
 				RequestId: uuid.NewString(),
 			},
-			wantRes: map[string]*types.Result{
-				"req-header-based-profile": {
-					TargetPod: &types.ScoredPod{
-						Pod: &types.PodMetrics{
-							Pod: &backend.Pod{
-								Address: "matched-endpoint",
-								Labels:  map[string]string{},
+			wantRes: &types.SchedulingResult{
+				ProfileResults: map[string]*types.ProfileRunResult{
+					"req-header-based-profile": {
+						TargetPod: &types.ScoredPod{
+							Pod: &types.PodMetrics{
+								Pod: &backend.Pod{
+									Address: "matched-endpoint",
+									Labels:  map[string]string{},
+								},
 							},
 						},
 					},
 				},
+				PrimaryProfileName: "req-header-based-profile",
 			},
 		},
 	}
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			scheduler := NewReqHeaderBasedScheduler(&fakeDataStore{pods: test.input})
-			got, err := scheduler.Schedule(context.Background(), test.req)
+			scheduler := NewReqHeaderBasedScheduler()
+			got, err := scheduler.Schedule(context.Background(), test.req, types.ToSchedulerPodMetrics(test.input))
 			if test.err != (err != nil) {
 				t.Errorf("Unexpected error, got %v, want %v", err, test.err)
 			}
@@ -108,15 +111,3 @@ func TestSchedule(t *testing.T) {
 		})
 	}
 }
-
-type fakeDataStore struct {
-	pods []*backendmetrics.FakePodMetrics
-}
-
-func (fds *fakeDataStore) PodGetAll() []backendmetrics.PodMetrics {
-	pm := make([]backendmetrics.PodMetrics, 0, len(fds.pods))
-	for _, pod := range fds.pods {
-		pm = append(pm, pod)
-	}
-	return pm
-}
diff --git a/conformance/tests/basic/epp_unavailable_fail_open.go b/conformance/tests/basic/epp_unavailable_fail_open.go
new file mode 100644
index 0000000..2f68606
--- /dev/null
+++ b/conformance/tests/basic/epp_unavailable_fail_open.go
@@ -0,0 +1,117 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"net/http"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+	"sigs.k8s.io/gateway-api/pkg/features"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
+	trafficutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+)
+
+func init() {
+	tests.ConformanceTests = append(tests.ConformanceTests, EppUnAvailableFailOpen)
+}
+
+var EppUnAvailableFailOpen = suite.ConformanceTest{
+	ShortName:   "EppUnAvailableFailOpen",
+	Description: "Inference gateway should send traffic to backends even when the EPP is unavailable (fail-open)",
+	Manifests:   []string{"tests/basic/epp_unavailable_fail_open.yaml"},
+	Features: []features.FeatureName{
+		features.FeatureName("SupportInferencePool"),
+		features.SupportGateway,
+	},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			appBackendNamespace    = "gateway-conformance-app-backend"
+			infraNamespace         = "gateway-conformance-infra"
+			hostname               = "secondary.example.com"
+			path                   = "/failopen-pool-test"
+			expectedPodReplicas    = 3
+			eppSelectionHeaderName = "test-epp-endpoint-selection"
+			appPodBackendPrefix    = "secondary-inference-model-server"
+			requestBody            = `{
+                "model": "conformance-fake-model",
+                "prompt": "Write as if you were a critic: San Francisco"
+            }`
+		)
+
+		httpRouteNN := types.NamespacedName{Name: "httproute-for-failopen-pool-gw", Namespace: appBackendNamespace}
+		gatewayNN := types.NamespacedName{Name: "conformance-secondary-gateway", Namespace: infraNamespace}
+		poolNN := types.NamespacedName{Name: "secondary-inference-pool", Namespace: appBackendNamespace}
+		eppDeploymentNN := types.NamespacedName{Name: "secondary-app-endpoint-picker", Namespace: appBackendNamespace}
+		backendPodLabels := map[string]string{"app": "secondary-inference-model-server"}
+
+		k8sutils.HTTPRouteMustBeAcceptedAndResolved(t, s.Client, s.TimeoutConfig, httpRouteNN, gatewayNN)
+		k8sutils.InferencePoolMustBeAcceptedByParent(t, s.Client, poolNN)
+		gwAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, gatewayNN)
+
+		pods, err := k8sutils.GetPodsWithLabel(t, s.Client, appBackendNamespace, backendPodLabels)
+		require.NoError(t, err, "Failed to get backend pods")
+		require.Len(t, pods, expectedPodReplicas, "Expected to find %d backend pod, but found %d.", expectedPodReplicas, len(pods))
+
+		targetPodIP := pods[0].Status.PodIP
+		t.Run("Phase 1: Verify baseline connectivity with EPP available", func(t *testing.T) {
+			t.Log("Sending request to ensure the Gateway and EPP are working correctly...")
+			trafficutils.MakeRequestWithRequestParamAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gwAddr,
+				trafficutils.Request{
+					Host:      hostname,
+					Path:      path,
+					Headers:   map[string]string{eppSelectionHeaderName: targetPodIP},
+					Method:    http.MethodPost,
+					Body:      requestBody,
+					Backend:   pods[0].Name, // Make sure the request is from the targetPod when the EPP is alive.
+					Namespace: appBackendNamespace,
+				},
+			)
+		})
+
+		t.Run("Phase 2: Verify fail-open behavior after EPP becomes unavailable", func(t *testing.T) {
+			t.Log("Simulating an EPP failure by deleting its deployment...")
+			deleteErr := k8sutils.DeleteDeployment(t, s.Client, s.TimeoutConfig, eppDeploymentNN)
+			require.NoError(t, deleteErr, "Failed to delete the EPP deployment")
+
+			t.Log("Sending request again, expecting success to verify fail-open...")
+			trafficutils.MakeRequestWithRequestParamAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gwAddr,
+				trafficutils.Request{
+					Host:      hostname,
+					Path:      path,
+					Headers:   map[string]string{eppSelectionHeaderName: targetPodIP},
+					Method:    http.MethodPost,
+					Body:      requestBody,
+					Backend:   appPodBackendPrefix, // Only checks the prefix since the EPP is not alive and the response can return from any Pod.
+					Namespace: appBackendNamespace,
+				},
+			)
+		})
+	},
+}
diff --git a/conformance/tests/basic/epp_unavailable_fail_open.yaml b/conformance/tests/basic/epp_unavailable_fail_open.yaml
new file mode 100644
index 0000000..cd681cb
--- /dev/null
+++ b/conformance/tests/basic/epp_unavailable_fail_open.yaml
@@ -0,0 +1,36 @@
+# --- InferenceModel Definition ---
+# TODO: remove inferenceModel dependency https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/1002
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: conformance-fake-model-server
+  namespace: gateway-conformance-app-backend
+spec:
+  modelName: conformance-fake-model
+  criticality: Critical # Mark it as critical to bypass the saturation check since the model server is fake and don't have such metrics. 
+  poolRef:
+    name: secondary-inference-pool
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-for-failopen-pool-gw
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: conformance-secondary-gateway
+    namespace: gateway-conformance-infra
+    sectionName: http
+  hostnames:
+  - "secondary.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: secondary-inference-pool # Use secondary-inferencePool because it has failureMode set to failOpen
+    matches:
+    - path:
+        type: PathPrefix
+        value: /failopen-pool-test
diff --git a/conformance/tests/basic/gateway_following_epp_routing.go b/conformance/tests/basic/gateway_following_epp_routing.go
new file mode 100644
index 0000000..9877999
--- /dev/null
+++ b/conformance/tests/basic/gateway_following_epp_routing.go
@@ -0,0 +1,194 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"fmt"
+	"net/http"
+	"slices"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+	"golang.org/x/sync/errgroup"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+	"sigs.k8s.io/gateway-api/pkg/features"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+	trafficutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+	gwhttp "sigs.k8s.io/gateway-api/conformance/utils/http"
+)
+
+func init() {
+	// Register the GatewayFollowingEPPRouting test case with the conformance suite.
+	// This ensures it will be discovered and run by the test runner.
+	tests.ConformanceTests = append(tests.ConformanceTests, GatewayFollowingEPPRouting)
+}
+
+// GatewayFollowingEPPRouting defines the test case for verifying gateway should send traffic to an endpoint in the list returned by EPP.
+var GatewayFollowingEPPRouting = suite.ConformanceTest{
+	ShortName:   "GatewayFollowingEPPRouting",
+	Description: "Inference gateway should send traffic to an endpoint in the list returned by EPP",
+	Manifests:   []string{"tests/basic/gateway_following_epp_routing.yaml"},
+	Features: []features.FeatureName{
+		features.FeatureName("SupportInferencePool"),
+		features.SupportGateway,
+	},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			appBackendNamespace = "gateway-conformance-app-backend"
+			infraNamespace      = "gateway-conformance-infra"
+			hostname            = "primary.example.com"
+			path                = "/primary-gateway-test"
+			expectedPodReplicas = 3
+			// eppSelectionHeaderName is the custom header used by the testing-EPP service
+			// to determine which endpoint to select.
+			eppSelectionHeaderName = "test-epp-endpoint-selection"
+			appPodBackendPrefix    = "primary-inference-model-server"
+		)
+
+		httpRouteNN := types.NamespacedName{Name: "httproute-for-primary-gw", Namespace: appBackendNamespace}
+		gatewayNN := types.NamespacedName{Name: "conformance-primary-gateway", Namespace: infraNamespace}
+		poolNN := types.NamespacedName{Name: "primary-inference-pool", Namespace: appBackendNamespace}
+		backendPodLabels := map[string]string{"app": "primary-inference-model-server"}
+
+		t.Log("Verifying HTTPRoute and InferencePool are accepted and the Gateway has an address.")
+		k8sutils.HTTPRouteMustBeAcceptedAndResolved(t, s.Client, s.TimeoutConfig, httpRouteNN, gatewayNN)
+		k8sutils.InferencePoolMustBeAcceptedByParent(t, s.Client, poolNN)
+		gwAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, gatewayNN)
+
+		t.Logf("Fetching backend pods with labels: %v", backendPodLabels)
+		pods, err := k8sutils.GetPodsWithLabel(t, s.Client, appBackendNamespace, backendPodLabels)
+		require.NoError(t, err, "Failed to get backend pods")
+		require.Len(t, pods, expectedPodReplicas, "Expected to find %d backend pods, but found %d.", expectedPodReplicas, len(pods))
+
+		podIPs := make([]string, len(pods))
+		podNames := make([]string, len(pods))
+		for i, pod := range pods {
+			podIPs[i] = pod.Status.PodIP
+			podNames[i] = pod.Name
+		}
+
+		requestBody := `{
+            "model": "conformance-fake-model",
+            "prompt": "Write as if you were a critic: San Francisco"
+        }`
+
+		for i := 0; i < len(pods); i++ {
+			// Send an initial request targeting a single pod and wait for it to be successful to ensure the Gateway and EPP
+			// are functioning correctly before running the main test cases.
+			trafficutils.MakeRequestWithRequestParamAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gwAddr,
+				trafficutils.Request{
+					Host:      hostname,
+					Path:      path,
+					Headers:   map[string]string{eppSelectionHeaderName: podIPs[i]},
+					Method:    http.MethodPost,
+					Body:      requestBody,
+					Backend:   podNames[i],
+					Namespace: appBackendNamespace,
+				},
+			)
+		}
+
+		testCases := []struct {
+			name                                  string
+			podIPsToBeReturnedByEPP               []string
+			expectAllRequestsRoutedWithinPodNames []string
+		}{
+			{
+				name:                                  "should route traffic to a single designated pod",
+				podIPsToBeReturnedByEPP:               []string{podIPs[2]},
+				expectAllRequestsRoutedWithinPodNames: []string{podNames[2]},
+			},
+			{
+				name:                                  "should route traffic to two designated pods",
+				podIPsToBeReturnedByEPP:               []string{podIPs[0], podIPs[1]},
+				expectAllRequestsRoutedWithinPodNames: []string{podNames[0], podNames[1]},
+			},
+			{
+				name:                                  "should route traffic to all available pods",
+				podIPsToBeReturnedByEPP:               []string{podIPs[0], podIPs[1], podIPs[2]},
+				expectAllRequestsRoutedWithinPodNames: []string{podNames[0], podNames[1], podNames[2]},
+			},
+		}
+
+		for _, tc := range testCases {
+			t.Run(tc.name, func(t *testing.T) {
+				eppHeaderValue := strings.Join(tc.podIPsToBeReturnedByEPP, ",")
+				headers := map[string]string{eppSelectionHeaderName: eppHeaderValue}
+
+				t.Logf("Sending request to %s with EPP header '%s: %s'", gwAddr, eppSelectionHeaderName, eppHeaderValue)
+				t.Logf("Expecting traffic to be routed to pod: %v", tc.expectAllRequestsRoutedWithinPodNames)
+
+				assertTrafficOnlyReachesToExpectedPods(t, s, gwAddr, gwhttp.ExpectedResponse{
+					Request: gwhttp.Request{
+						Host:    hostname,
+						Path:    path,
+						Method:  http.MethodPost,
+						Headers: headers,
+					},
+					Response: gwhttp.Response{
+						StatusCode: http.StatusOK,
+					},
+					Backend:   appPodBackendPrefix,
+					Namespace: appBackendNamespace,
+				}, requestBody, tc.expectAllRequestsRoutedWithinPodNames)
+			})
+		}
+	},
+}
+
+func assertTrafficOnlyReachesToExpectedPods(t *testing.T, suite *suite.ConformanceTestSuite, gwAddr string, expected gwhttp.ExpectedResponse, requestBody string, expectedPodNames []string) {
+	t.Helper()
+	const (
+		concurrentRequests = 10
+		totalRequests      = 100
+	)
+	var (
+		roundTripper = suite.RoundTripper
+		g            errgroup.Group
+		req          = gwhttp.MakeRequest(t, &expected, gwAddr, "HTTP", "http")
+	)
+	g.SetLimit(concurrentRequests)
+	for i := 0; i < totalRequests; i++ {
+		g.Go(func() error {
+			cReq, cRes, err := traffic.MakeCallRoundTripper(t, roundTripper, &traffic.RequestWithBody{Request: req, Body: strings.NewReader(requestBody)})
+			if err != nil {
+				return fmt.Errorf("failed to roundtrip request: %w", err)
+			}
+			if err := gwhttp.CompareRequest(t, &req, cReq, cRes, expected); err != nil {
+				return fmt.Errorf("response expectation failed for request: %w", err)
+			}
+
+			if slices.Contains(expectedPodNames, cReq.Pod) {
+				return nil
+			}
+			return fmt.Errorf("request was handled by an unexpected pod %q", cReq.Pod)
+		})
+	}
+	if err := g.Wait(); err != nil {
+		t.Fatalf("Not all the requests are sent to the expectedPods successfully, err: %v", err)
+	}
+	t.Logf("Traffic successfully reached only to expected pods: %v", expectedPodNames)
+}
diff --git a/conformance/tests/basic/gateway_following_epp_routing.yaml b/conformance/tests/basic/gateway_following_epp_routing.yaml
new file mode 100644
index 0000000..d290b75
--- /dev/null
+++ b/conformance/tests/basic/gateway_following_epp_routing.yaml
@@ -0,0 +1,38 @@
+# --- InferenceModel Definition ---
+# Service for the infra-backend-deployment.
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferenceModel
+metadata:
+  name: conformance-fake-model-server
+  namespace: gateway-conformance-app-backend
+spec:
+  modelName: conformance-fake-model
+  criticality: Critical # Mark it as critical to bypass the saturation check since the model server is fake and don't have such metrics. 
+  poolRef:
+    name: primary-inference-pool
+---
+# --- HTTPRoute for Primary Gateway (conformance-gateway) ---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-for-primary-gw
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: conformance-primary-gateway
+    namespace: gateway-conformance-infra
+    sectionName: http
+  hostnames:
+  - "primary.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: primary-inference-pool
+    matches:
+    - path:
+        type: PathPrefix
+        value: /primary-gateway-test
+  
\ No newline at end of file
diff --git a/conformance/tests/basic/httproute_invalid_inferencepool_ref.go b/conformance/tests/basic/httproute_invalid_inferencepool_ref.go
index 967df35..5fc2a66 100644
--- a/conformance/tests/basic/httproute_invalid_inferencepool_ref.go
+++ b/conformance/tests/basic/httproute_invalid_inferencepool_ref.go
@@ -47,7 +47,7 @@ var HTTPRouteInvalidInferencePoolRef = suite.ConformanceTest{
 			appBackendNamespace = "gateway-conformance-app-backend"
 			infraNamespace      = "gateway-conformance-infra"
 			routeName           = "httproute-to-non-existent-pool"
-			gatewayName         = "conformance-gateway"
+			gatewayName         = "conformance-primary-gateway"
 		)
 		routeNN := types.NamespacedName{Name: routeName, Namespace: appBackendNamespace}
 		gatewayNN := types.NamespacedName{Name: gatewayName, Namespace: infraNamespace}
diff --git a/conformance/tests/basic/httproute_invalid_inferencepool_ref.yaml b/conformance/tests/basic/httproute_invalid_inferencepool_ref.yaml
index 61bdb54..227aeef 100644
--- a/conformance/tests/basic/httproute_invalid_inferencepool_ref.yaml
+++ b/conformance/tests/basic/httproute_invalid_inferencepool_ref.yaml
@@ -16,7 +16,7 @@ spec:
   parentRefs:
   - group: gateway.networking.k8s.io
     kind: Gateway
-    name: conformance-gateway # Name of the shared Gateway from base manifests
+    name: conformance-primary-gateway # Name of the shared Gateway from base manifests
     namespace: gateway-conformance-infra  # Namespace of the shared Gateway
     sectionName: http
   rules:
@@ -24,7 +24,6 @@ spec:
     - group: inference.networking.x-k8s.io
       kind: InferencePool
       name: non-existent-inference-pool # Intentionally Non-Existing
-      port: 8080
     matches:
     - path:
         type: PathPrefix
diff --git a/conformance/tests/basic/httproute_multiple_gateways_different_pools.go b/conformance/tests/basic/httproute_multiple_gateways_different_pools.go
new file mode 100644
index 0000000..251f55c
--- /dev/null
+++ b/conformance/tests/basic/httproute_multiple_gateways_different_pools.go
@@ -0,0 +1,118 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"net/http"
+	"testing"
+
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+)
+
+func init() {
+	tests.ConformanceTests = append(tests.ConformanceTests, HTTPRouteMultipleGatewaysDifferentPools)
+}
+
+var HTTPRouteMultipleGatewaysDifferentPools = suite.ConformanceTest{
+	ShortName:   "HTTPRouteMultipleGatewaysDifferentPools",
+	Description: "Validates two HTTPRoutes on different Gateways successfully referencing different InferencePools and routes traffic accordingly.",
+	Manifests:   []string{"tests/basic/httproute_multiple_gateways_different_pools.yaml"},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			appBackendNamespace = "gateway-conformance-app-backend"
+			infraNamespace      = "gateway-conformance-infra"
+
+			primaryGatewayName    = "conformance-primary-gateway"
+			routeForPrimaryGWName = "route-for-primary-gateway"
+			primaryPoolName       = "primary-inference-pool"
+			primaryBackendPodName = "primary-inference-model-server"
+			primaryRoutePath      = "/test-primary-gateway"
+			primaryRouteHostname  = "primary.example.com"
+
+			secondaryGatewayName    = "conformance-secondary-gateway"
+			routeForSecondaryGWName = "route-for-secondary-gateway"
+			secondaryPoolName       = "secondary-inference-pool"
+			secondaryBackendPodName = "secondary-inference-model-server"
+			secondaryRoutePath      = "/test-secondary-gateway"
+			secondaryRouteHostname  = "secondary.example.com"
+		)
+
+		routeForPrimaryGWNN := types.NamespacedName{Name: routeForPrimaryGWName, Namespace: appBackendNamespace}
+		routeForSecondaryGWNN := types.NamespacedName{Name: routeForSecondaryGWName, Namespace: appBackendNamespace}
+		primaryPoolNN := types.NamespacedName{Name: primaryPoolName, Namespace: appBackendNamespace}
+		secondaryPoolNN := types.NamespacedName{Name: secondaryPoolName, Namespace: appBackendNamespace}
+		primaryGatewayNN := types.NamespacedName{Name: primaryGatewayName, Namespace: infraNamespace}
+		secondaryGatewayNN := types.NamespacedName{Name: secondaryGatewayName, Namespace: infraNamespace}
+
+		t.Run("Primary HTTPRoute, InferencePool, and Gateway path: verify status and traffic", func(t *testing.T) {
+			k8sutils.HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted(
+				t,
+				s.Client,
+				routeForPrimaryGWNN,
+				primaryGatewayNN,
+				primaryPoolNN,
+			)
+
+			primaryGwAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, primaryGatewayNN)
+
+			traffic.MakeRequestAndExpectEventuallyConsistentResponse(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				primaryGwAddr,
+				traffic.Request{
+					Host:               primaryRouteHostname,
+					Path:               primaryRoutePath,
+					ExpectedStatusCode: http.StatusOK,
+					Backend:            primaryBackendPodName,
+					Namespace:          appBackendNamespace,
+				},
+			)
+		})
+
+		t.Run("Secondary HTTPRoute, InferencePool, and Gateway path: verify status and traffic", func(t *testing.T) {
+			k8sutils.HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted(
+				t,
+				s.Client,
+				routeForSecondaryGWNN,
+				secondaryGatewayNN,
+				secondaryPoolNN,
+			)
+
+			secondaryGwAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, secondaryGatewayNN)
+
+			traffic.MakeRequestAndExpectEventuallyConsistentResponse(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				secondaryGwAddr,
+				traffic.Request{
+					Host:               secondaryRouteHostname,
+					Path:               secondaryRoutePath,
+					ExpectedStatusCode: http.StatusOK,
+					Backend:            secondaryBackendPodName,
+					Namespace:          appBackendNamespace,
+				},
+			)
+		})
+	},
+}
diff --git a/conformance/tests/basic/httproute_multiple_gateways_different_pools.yaml b/conformance/tests/basic/httproute_multiple_gateways_different_pools.yaml
new file mode 100644
index 0000000..b8ec2d1
--- /dev/null
+++ b/conformance/tests/basic/httproute_multiple_gateways_different_pools.yaml
@@ -0,0 +1,44 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: route-for-primary-gateway
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - kind: Gateway
+    name: conformance-primary-gateway
+    namespace: gateway-conformance-infra
+  hostnames:
+  - "primary.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: primary-inference-pool
+    matches:
+    - path:
+        type: PathPrefix
+        value: /test-primary-gateway
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: route-for-secondary-gateway
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - kind: Gateway
+    name: conformance-secondary-gateway
+    namespace: gateway-conformance-infra
+  hostnames:
+  - "secondary.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: secondary-inference-pool
+    matches:
+    - path:
+        type: PathPrefix
+        value: /test-secondary-gateway
diff --git a/conformance/tests/basic/inferencepool_accepted.go b/conformance/tests/basic/inferencepool_accepted.go
index d74f738..f34c5fc 100644
--- a/conformance/tests/basic/inferencepool_accepted.go
+++ b/conformance/tests/basic/inferencepool_accepted.go
@@ -47,7 +47,7 @@ var InferencePoolAccepted = suite.ConformanceTest{
 	},
 	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
 		// created by the associated manifest file.
-		poolNN := types.NamespacedName{Name: "inferencepool-basic-accepted", Namespace: "gateway-conformance-app-backend"}
+		poolNN := types.NamespacedName{Name: "primary-inference-pool", Namespace: "gateway-conformance-app-backend"}
 
 		t.Run("InferencePool should have Accepted condition set to True", func(t *testing.T) {
 			// Define the expected status condition. We use the standard "Accepted"
diff --git a/conformance/tests/basic/inferencepool_accepted.yaml b/conformance/tests/basic/inferencepool_accepted.yaml
index ecee5b3..8381500 100644
--- a/conformance/tests/basic/inferencepool_accepted.yaml
+++ b/conformance/tests/basic/inferencepool_accepted.yaml
@@ -1,72 +1,3 @@
-# Basic InferencePool for acceptance testing.
-# This manifest defines the minimal required fields to create valid
-# InferencePool and HTTPRoute resources, which the InferencePoolAccepted
-# test will use to verify that the controller recognizes and accepts the resource.
-
-# --- Minimal Backend Deployment (using agnhost echo server) ---
-# This Deployment provides Pods for the InferencePool to select.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: infra-backend-v1-deployment
-  namespace: gateway-conformance-app-backend
-  labels:
-    app: infra-backend-v1
-spec:
-  replicas: 1
-  selector:
-    matchLabels:
-      app: infra-backend-v1
-  template:
-    metadata:
-      labels:
-        app: infra-backend-v1
-    spec:
-      containers:
-      - name: agnhost-echo
-        image: k8s.gcr.io/e2e-test-images/agnhost:2.39
-        args:
-        - serve-hostname
-        - --http-port=8080
-        ports:
-        - name: http
-          containerPort: 8080
-        readinessProbe:
-          httpGet:
-            path: /
-            port: 8080
-          initialDelaySeconds: 3
-          periodSeconds: 5
-          failureThreshold: 2
-
----
-# --- InferencePool Definition ---
-apiVersion: inference.networking.x-k8s.io/v1alpha2
-kind: InferencePool
-metadata:
-  # This name must match the 'poolNN' variable defined in the
-  # conformance/tests/basic/inferencepool_accepted.go test file.
-  name: inferencepool-basic-accepted
-  # This namespace should be one created by the base manifests.
-  namespace: gateway-conformance-app-backend
-spec:
-  # --- Selector (Required) ---
-  # Selects the Pods belonging to this pool.
-  selector:
-    app: "infra-backend-v1"
-
-  # --- Target Port (Required) ---
-  # The port the model server container (agnhost in this case) listens on.
-  targetPortNumber: 8080 # Matches agnhost's http-port
-
-  # --- Extension Reference ---
-  # GKE-specific configuration reference.
-  extensionRef:
-    # group: "" # Optional
-    # kind: Service # Optional
-    name: infra-backend-v1-epp
-
----
 # --- HTTPRoute Definition ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
@@ -77,16 +8,15 @@ spec:
   parentRefs:
   - group: gateway.networking.k8s.io
     kind: Gateway
-    name: conformance-gateway # Name of the shared Gateway from maniffests.yaml
+    name: conformance-primary-gateway # Name of the shared Gateway from maniffests.yaml
     namespace: gateway-conformance-infra  # Namespace of the shared Gateway
     sectionName: http
   rules:
   - backendRefs:
     - group: inference.networking.x-k8s.io # InferencePool API group
       kind: InferencePool
-      name: inferencepool-basic-accepted # Name of the InferencePool this route points to
+      name: primary-inference-pool # Name of the InferencePool this route points to
       # namespace: gateway-conformance-app-backend - is omitted since it is in the same namespace as HTTPRoute
-      port: 8080 # Matching the InferencePool's targetPortNumber
     matches:
     - path:
         type: PathPrefix
diff --git a/conformance/tests/basic/inferencepool_httproute_port_validation.go b/conformance/tests/basic/inferencepool_httproute_port_validation.go
new file mode 100644
index 0000000..888b659
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_httproute_port_validation.go
@@ -0,0 +1,119 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"testing"
+
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+	"sigs.k8s.io/gateway-api/pkg/features"
+
+	// Local project imports
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
+	trafficutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+)
+
+func init() {
+	tests.ConformanceTests = append(tests.ConformanceTests, InferencePoolHTTPRoutePortValidation)
+}
+
+var InferencePoolHTTPRoutePortValidation = suite.ConformanceTest{
+	ShortName:   "InferencePoolHTTPRoutePortValidation",
+	Description: "Validates HTTPRoute backendRef port configurations (unspecified, matching, non-matching) when referencing an InferencePool, and checks resulting status conditions.",
+	Manifests:   []string{"tests/basic/inferencepool_httproute_port_validation.yaml"},
+	Features: []features.FeatureName{
+		features.FeatureName("SupportInferencePool"),
+		features.SupportGateway,
+	},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			appBackendNamespace   = "gateway-conformance-app-backend"
+			infraNamespace        = "gateway-conformance-infra"
+			gatewayName           = "conformance-gateway"
+			poolName              = "target-pool-port-validation"
+			backendDeploymentName = "infra-backend-deployment-port-test"
+		)
+
+		gatewayNN := types.NamespacedName{Name: gatewayName, Namespace: infraNamespace}
+		poolNN := types.NamespacedName{Name: poolName, Namespace: appBackendNamespace}
+
+		gatewayAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, gatewayNN)
+
+		t.Run("Scenario 1: HTTPRoute backendRef to InferencePool with Port Unspecified", func(t *testing.T) {
+			routeNN := types.NamespacedName{Name: "httproute-pool-port-unspecified", Namespace: appBackendNamespace}
+			hostname := "port-unspecified.example.com"
+			path := "/test-port-unspecified"
+
+			k8sutils.HTTPRouteMustBeAcceptedAndResolved(t, s.Client, s.TimeoutConfig, routeNN, gatewayNN)
+			k8sutils.InferencePoolMustBeAcceptedByParent(t, s.Client, poolNN)
+
+			trafficutils.MakeRequestAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gatewayAddr,
+				hostname,
+				path,
+				backendDeploymentName,
+				appBackendNamespace,
+			)
+		})
+
+		t.Run("Scenario 2: HTTPRoute backendRef to InferencePool with Port Specified and Matching", func(t *testing.T) {
+			routeNN := types.NamespacedName{Name: "httproute-pool-port-matching", Namespace: appBackendNamespace}
+			hostname := "port-matching.example.com"
+			path := "/test-port-matching"
+
+			k8sutils.HTTPRouteMustBeAcceptedAndResolved(t, s.Client, s.TimeoutConfig, routeNN, gatewayNN)
+			k8sutils.InferencePoolMustBeAcceptedByParent(t, s.Client, poolNN)
+
+			trafficutils.MakeRequestAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gatewayAddr,
+				hostname,
+				path,
+				backendDeploymentName,
+				appBackendNamespace,
+			)
+		})
+
+		// TODO: Add a warning check after the required change is made per discussion in github.com/kubernetes-sigs/gateway-api-inference-extension/discussions/918
+		t.Run("Scenario 3: HTTPRoute backendRef to InferencePool with Port Specified and Non-Matching. Request still passing because HTTP Port is ignored when inferencePool is backendRef", func(t *testing.T) {
+			routeNN := types.NamespacedName{Name: "httproute-pool-port-non-matching", Namespace: appBackendNamespace}
+			hostname := "port-non-matching.example.com"
+			path := "/test-port-non-matching"
+
+			k8sutils.HTTPRouteMustBeAcceptedAndResolved(t, s.Client, s.TimeoutConfig, routeNN, gatewayNN)
+			k8sutils.InferencePoolMustBeAcceptedByParent(t, s.Client, poolNN)
+
+			trafficutils.MakeRequestAndExpectSuccess(
+				t,
+				s.RoundTripper,
+				s.TimeoutConfig,
+				gatewayAddr,
+				hostname,
+				path,
+				backendDeploymentName,
+				appBackendNamespace,
+			)
+		})
+	},
+}
diff --git a/conformance/tests/basic/inferencepool_httproute_port_validation.yaml b/conformance/tests/basic/inferencepool_httproute_port_validation.yaml
new file mode 100644
index 0000000..d15d317
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_httproute_port_validation.yaml
@@ -0,0 +1,244 @@
+# conformance/tests/basic/inferencepool_httproute_port_validation.yaml
+
+# --- Backend Deployment (reusing standard echoserver) ---
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: infra-backend-deployment-port-test
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: infra-backend-port-test
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: infra-backend-port-test
+  template:
+    metadata:
+      labels:
+        app: infra-backend-port-test
+    spec:
+      containers:
+      - name: echoserver
+        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
+        ports:
+        - containerPort: 3000
+        readinessProbe:
+          httpGet:
+            path: /
+            port: 3000
+          initialDelaySeconds: 3
+          periodSeconds: 5
+          failureThreshold: 2
+        env:
+        - name: POD_NAME
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name
+        - name: NAMESPACE
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.namespace
+---
+# --- Backend Service ---
+# Service for the infra-backend-deployment-port-test.
+apiVersion: v1
+kind: Service
+metadata:
+  name: infra-backend-svc-port-test
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: infra-backend-port-test
+  ports:
+  - name: http
+    port: 3000
+    targetPort: 3000
+---
+# --- InferencePool Definition ---
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: target-pool-port-validation
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: "infra-backend-port-test"
+  targetPortNumber: 3000
+  extensionRef:
+    name: target-pool-port-validation-epp
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: target-pool-port-validation-epp
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: target-pool-port-validation-epp
+  ports:
+    - protocol: TCP
+      port: 9002
+      targetPort: 9002
+      appProtocol: http2
+  type: ClusterIP
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: target-pool-port-validation-epp
+  namespace: gateway-conformance-app-backend
+  labels:
+    app: target-pool-port-validation-epp
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: target-pool-port-validation-epp
+  template:
+    metadata:
+      labels:
+        app: target-pool-port-validation-epp
+    spec:
+      terminationGracePeriodSeconds: 130
+      containers:
+        - name: epp
+          image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/epp:main
+          imagePullPolicy: Always
+          args:
+            - -poolName
+            - "target-pool-port-validation"
+            - "-poolNamespace"
+            - "gateway-conformance-app-backend"
+            - -v
+            - "4"
+            - --zap-encoder
+            - "json"
+            - -grpcPort
+            - "9002"
+            - -grpcHealthPort
+            - "9003"
+          ports:
+            - containerPort: 9002
+            - containerPort: 9003
+            - name: metrics
+              containerPort: 9090
+          livenessProbe:
+            grpc:
+              port: 9003
+              service: inference-extension
+            initialDelaySeconds: 5
+            periodSeconds: 10
+          readinessProbe:
+            grpc:
+              port: 9003
+              service: inference-extension
+            initialDelaySeconds: 5
+            periodSeconds: 10
+---
+# --- HTTPRoute Scenario 1: Port Unspecified ---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-pool-port-unspecified
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: conformance-gateway
+    namespace: gateway-conformance-infra
+    sectionName: http
+  hostnames:
+  - "port-unspecified.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: target-pool-port-validation
+      # Port is intentionally unspecified here
+    matches:
+    - path:
+        type: PathPrefix
+        value: /test-port-unspecified
+---
+# --- HTTPRoute Scenario 2: Port Matching ---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-pool-port-matching
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: conformance-gateway
+    namespace: gateway-conformance-infra
+    sectionName: http
+  hostnames:
+  - "port-matching.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: target-pool-port-validation
+      port: 3000 # Port matches InferencePool's targetPortNumber
+    matches:
+    - path:
+        type: PathPrefix
+        value: /test-port-matching
+---
+# --- HTTPRoute Scenario 3: Port Non-Matching ---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-pool-port-non-matching
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - group: gateway.networking.k8s.io
+    kind: Gateway
+    name: conformance-gateway
+    namespace: gateway-conformance-infra
+    sectionName: http
+  hostnames:
+  - "port-non-matching.example.com"
+  rules:
+  - backendRefs:
+    - group: inference.networking.x-k8s.io
+      kind: InferencePool
+      name: target-pool-port-validation
+      port: 8888 # Port does NOT match InferencePool's targetPortNumber
+    matches:
+    - path:
+        type: PathPrefix
+        value: /test-port-non-matching
+---
+# --- Conformance EPP Requried Role and RoleBindings ---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: inference-model-reader
+  namespace: gateway-conformance-app-backend
+rules:
+- apiGroups: ["inference.networking.x-k8s.io"]
+  resources: ["inferencemodels", "inferencepools"]
+  verbs: ["get", "list", "watch"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: epp-to-inference-model-reader
+  namespace: gateway-conformance-app-backend
+subjects:
+- kind: ServiceAccount
+  name: default
+  namespace: gateway-conformance-app-backend
+roleRef:
+  kind: Role
+  name: inference-model-reader
+  apiGroup: rbac.authorization.k8s.io
diff --git a/conformance/tests/basic/inferencepool_invalid_epp_service.go b/conformance/tests/basic/inferencepool_invalid_epp_service.go
new file mode 100644
index 0000000..73e6dc0
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_invalid_epp_service.go
@@ -0,0 +1,79 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"net/http"
+	"testing"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+	"sigs.k8s.io/gateway-api/pkg/features"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
+	conformancehttp "sigs.k8s.io/gateway-api/conformance/utils/http"
+)
+
+func init() {
+	tests.ConformanceTests = append(tests.ConformanceTests, InferencePoolInvalidEPPService)
+}
+
+var InferencePoolInvalidEPPService = suite.ConformanceTest{
+	ShortName:   "InferencePoolInvalidEPPService",
+	Description: "An HTTPRoute that references an InferencePool with a non-existent EPP service should have a ResolvedRefs condition with a status of False and a reason of BackendNotFound.",
+	Manifests:   []string{"tests/basic/inferencepool_invalid_epp_service.yaml"},
+	Features: []features.FeatureName{
+		features.SupportGateway,
+		features.SupportHTTPRoute,
+		features.FeatureName("SupportInferencePool"),
+	},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			routePath      = "/invalid-epp-test"
+			infraNamespace = "gateway-conformance-infra"
+			appNamespace   = "gateway-conformance-app-backend"
+		)
+
+		routeNN := types.NamespacedName{Name: "httproute-for-invalid-epp-pool", Namespace: appNamespace}
+		gwNN := types.NamespacedName{Name: "conformance-primary-gateway", Namespace: infraNamespace}
+
+		gwAddr := kubernetes.GatewayAndHTTPRoutesMustBeAccepted(t, s.Client, s.TimeoutConfig, s.ControllerName, kubernetes.NewGatewayRef(gwNN), routeNN)
+
+		t.Run("HTTPRoute has a ResolvedRefs Condition with status False and Reason BackendNotFound", func(t *testing.T) {
+			resolvedRefsCond := metav1.Condition{
+				Type:   string(gatewayv1.RouteConditionResolvedRefs),
+				Status: metav1.ConditionFalse,
+				Reason: string(gatewayv1.RouteReasonBackendNotFound),
+			}
+			kubernetes.HTTPRouteMustHaveCondition(t, s.Client, s.TimeoutConfig, routeNN, gwNN, resolvedRefsCond)
+		})
+
+		t.Run("Request to a route with an invalid backend reference receives a 500 response", func(t *testing.T) {
+			conformancehttp.MakeRequestAndExpectEventuallyConsistentResponse(t, s.RoundTripper, s.TimeoutConfig, gwAddr, conformancehttp.ExpectedResponse{
+				Request: conformancehttp.Request{
+					Path: routePath,
+				},
+				Response: conformancehttp.Response{
+					StatusCode: http.StatusInternalServerError,
+				},
+			})
+		})
+	},
+}
diff --git a/conformance/tests/basic/inferencepool_invalid_epp_service.yaml b/conformance/tests/basic/inferencepool_invalid_epp_service.yaml
new file mode 100644
index 0000000..3a19036
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_invalid_epp_service.yaml
@@ -0,0 +1,30 @@
+apiVersion: inference.networking.x-k8s.io/v1alpha2
+kind: InferencePool
+metadata:
+  name: pool-with-invalid-epp
+  namespace: gateway-conformance-app-backend
+spec:
+  selector:
+    app: "inference-model-1"
+  targetPortNumber: 3000
+  extensionRef:
+    name: non-existent-epp-svc
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-for-invalid-epp-pool
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+  - name: conformance-primary-gateway
+    namespace: gateway-conformance-infra
+  rules:
+  - backendRefs:
+    - name: pool-with-invalid-epp
+      kind: InferencePool
+      group: inference.networking.x-k8s.io
+    matches:
+    - path:
+        type: PathPrefix
+        value: /invalid-epp-test
diff --git a/conformance/tests/basic/inferencepool_multiple_rules_different_pools.go b/conformance/tests/basic/inferencepool_multiple_rules_different_pools.go
new file mode 100644
index 0000000..d91a2fa
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_multiple_rules_different_pools.go
@@ -0,0 +1,91 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package basic
+
+import (
+	"testing"
+
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/gateway-api/conformance/utils/suite"
+	"sigs.k8s.io/gateway-api/pkg/features"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
+	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
+	trafficutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+)
+
+func init() {
+	tests.ConformanceTests = append(tests.ConformanceTests, HTTPRouteMultipleRulesDifferentPools)
+}
+
+var HTTPRouteMultipleRulesDifferentPools = suite.ConformanceTest{
+	ShortName:   "HTTPRouteMultipleRulesDifferentPools",
+	Description: "An HTTPRoute with two rules routing to two different InferencePools",
+	Manifests:   []string{"tests/basic/inferencepool_multiple_rules_different_pools.yaml"},
+	Features: []features.FeatureName{
+		features.SupportGateway,
+		features.SupportHTTPRoute,
+		features.FeatureName("SupportInferencePool"),
+	},
+	Test: func(t *testing.T, s *suite.ConformanceTestSuite) {
+		const (
+			appBackendNamespace = "gateway-conformance-app-backend"
+			infraNamespace      = "gateway-conformance-infra"
+
+			poolPrimaryName   = "primary-inference-pool"
+			poolSecondaryName = "secondary-inference-pool"
+			routeName         = "httproute-multiple-rules-different-pools"
+			gatewayName       = "conformance-primary-gateway"
+
+			backendPrimaryLabelValue   = "primary-inference-model-server"
+			backendSecondaryLabelValue = "secondary-inference-model-server"
+			backendAppLabelKey         = "app"
+
+			primaryPath   = "/primary"
+			secondaryPath = "/secondary"
+		)
+
+		primaryPoolNN := types.NamespacedName{Name: poolPrimaryName, Namespace: appBackendNamespace}
+		secondaryPoolNN := types.NamespacedName{Name: poolSecondaryName, Namespace: appBackendNamespace}
+		routeNN := types.NamespacedName{Name: routeName, Namespace: appBackendNamespace}
+		gatewayNN := types.NamespacedName{Name: gatewayName, Namespace: infraNamespace}
+
+		t.Run("Wait for resources to be accepted", func(t *testing.T) {
+			k8sutils.HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted(t, s.Client, routeNN, gatewayNN, primaryPoolNN)
+			k8sutils.HTTPRouteAndInferencePoolMustBeAcceptedAndRouteAccepted(t, s.Client, routeNN, gatewayNN, secondaryPoolNN)
+		})
+
+		t.Run("Traffic should be routed to the correct pool based on path", func(t *testing.T) {
+			primarySelector := labels.SelectorFromSet(labels.Set{backendAppLabelKey: backendPrimaryLabelValue})
+			secondarySelector := labels.SelectorFromSet(labels.Set{backendAppLabelKey: backendSecondaryLabelValue})
+
+			primaryPod := k8sutils.GetPod(t, s.Client, appBackendNamespace, primarySelector, s.TimeoutConfig.RequestTimeout)
+			secondaryPod := k8sutils.GetPod(t, s.Client, appBackendNamespace, secondarySelector, s.TimeoutConfig.RequestTimeout)
+
+			gwAddr := k8sutils.GetGatewayEndpoint(t, s.Client, s.TimeoutConfig, gatewayNN)
+
+			t.Run("request to primary pool", func(t *testing.T) {
+				trafficutils.MakeRequestAndExpectResponseFromPod(t, s.RoundTripper, s.TimeoutConfig, gwAddr, primaryPath, primaryPod)
+			})
+
+			t.Run("request to secondary pool", func(t *testing.T) {
+				trafficutils.MakeRequestAndExpectResponseFromPod(t, s.RoundTripper, s.TimeoutConfig, gwAddr, secondaryPath, secondaryPod)
+			})
+		})
+	},
+}
diff --git a/conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml b/conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml
new file mode 100644
index 0000000..7ef7ced
--- /dev/null
+++ b/conformance/tests/basic/inferencepool_multiple_rules_different_pools.yaml
@@ -0,0 +1,27 @@
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+  name: httproute-multiple-rules-different-pools
+  namespace: gateway-conformance-app-backend
+spec:
+  parentRefs:
+    - name: conformance-primary-gateway
+      namespace: gateway-conformance-infra
+  rules:
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /primary
+      backendRefs:
+        - name: primary-inference-pool
+          kind: InferencePool
+          group: inference.networking.x-k8s.io
+    - matches:
+        - path:
+            type: PathPrefix
+            value: /secondary
+      backendRefs:
+        - name: secondary-inference-pool
+          kind: InferencePool
+          group: inference.networking.x-k8s.io
diff --git a/conformance/tests/basic/inferencepool_resolvedrefs_condition.go b/conformance/tests/basic/inferencepool_resolvedrefs_condition.go
index 1bb8c7f..bf9dd12 100644
--- a/conformance/tests/basic/inferencepool_resolvedrefs_condition.go
+++ b/conformance/tests/basic/inferencepool_resolvedrefs_condition.go
@@ -50,8 +50,8 @@ var InferencePoolParentStatus = suite.ConformanceTest{
 		const (
 			appBackendNamespace        = "gateway-conformance-app-backend"
 			infraNamespace             = "gateway-conformance-infra"
-			poolName                   = "multi-gateway-pool"
-			sharedPrimaryGatewayName   = "conformance-gateway"
+			poolName                   = "primary-inference-pool"
+			sharedPrimaryGatewayName   = "conformance-primary-gateway"
 			sharedSecondaryGatewayName = "conformance-secondary-gateway"
 			httpRoutePrimaryName       = "httproute-for-primary-gw"
 			httpRouteSecondaryName     = "httproute-for-secondary-gw"
@@ -59,7 +59,7 @@ var InferencePoolParentStatus = suite.ConformanceTest{
 			pathPrimaryGw              = "/primary-gateway-test"
 			hostnameSecondaryGw        = "secondary.example.com"
 			pathSecondaryGw            = "/secondary-gateway-test"
-			backendServicePodName      = "infra-backend-deployment"
+			backendServicePodName      = "primary-inference-model-server-deployment"
 		)
 
 		poolNN := types.NamespacedName{Name: poolName, Namespace: appBackendNamespace}
diff --git a/conformance/tests/basic/inferencepool_resolvedrefs_condition.yaml b/conformance/tests/basic/inferencepool_resolvedrefs_condition.yaml
index 4b58f67..416352e 100644
--- a/conformance/tests/basic/inferencepool_resolvedrefs_condition.yaml
+++ b/conformance/tests/basic/inferencepool_resolvedrefs_condition.yaml
@@ -3,82 +3,7 @@
 # This manifest defines the initial resources for the
 # inferencepool_resolvedrefs_condition.go conformance test.
 
-# --- Backend Deployment (using standard Gateway API echoserver) ---
-# This Deployment provides Pods for the InferencePool to select.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: infra-backend-deployment
-  namespace: gateway-conformance-app-backend
-  labels:
-    app: infra-backend
-spec:
-  selector:
-    matchLabels:
-      app: infra-backend
-  template:
-    metadata:
-      labels:
-        app: infra-backend
-    spec:
-      containers:
-      - name: echoserver
-        image: gcr.io/k8s-staging-gateway-api/echo-basic:v20240412-v1.0.0-394-g40c666fd
-        ports:
-        - containerPort: 3000
-        readinessProbe:
-          httpGet:
-            path: /
-            port: 3000
-          initialDelaySeconds: 3
-          periodSeconds: 5
-          failureThreshold: 2
-        env:
-        - name: POD_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.name
-        - name: NAMESPACE
-          valueFrom:
-            fieldRef:
-              fieldPath: metadata.namespace
-        - name: POD_IP
-          valueFrom:
-            fieldRef:
-              fieldPath: status.podIP
----
-# --- Backend Service ---
-# Service for the infra-backend-deployment.
-apiVersion: v1
-kind: Service
-metadata:
-  name: infra-backend-svc
-  namespace: gateway-conformance-app-backend
-spec:
-  selector:
-    app: infra-backend
-  ports:
-  - name: http
-    port: 3000
-    targetPort: 3000
-  - name: epp
-    port: 9002
-    targetPort: 9002
----
-# --- InferencePool Definition ---
-apiVersion: inference.networking.x-k8s.io/v1alpha2
-kind: InferencePool
-metadata:
-  name: multi-gateway-pool
-  namespace: gateway-conformance-app-backend
-spec:
-  selector:
-    app: "infra-backend"
-  targetPortNumber: 3000
-  extensionRef:
-    name: infra-backend-svc
----
-# --- HTTPRoute for Primary Gateway (conformance-gateway) ---
+# --- HTTPRoute for Primary Gateway (conformance-primary-gateway) ---
 apiVersion: gateway.networking.k8s.io/v1
 kind: HTTPRoute
 metadata:
@@ -88,7 +13,7 @@ spec:
   parentRefs:
   - group: gateway.networking.k8s.io
     kind: Gateway
-    name: conformance-gateway
+    name: conformance-primary-gateway
     namespace: gateway-conformance-infra
     sectionName: http
   hostnames:
@@ -97,8 +22,7 @@ spec:
   - backendRefs:
     - group: inference.networking.x-k8s.io
       kind: InferencePool
-      name: multi-gateway-pool
-      port: 3000
+      name: primary-inference-pool
     matches:
     - path:
         type: PathPrefix
@@ -123,8 +47,7 @@ spec:
   - backendRefs:
     - group: inference.networking.x-k8s.io
       kind: InferencePool
-      name: multi-gateway-pool
-      port: 3000
+      name: primary-inference-pool
     matches:
     - path:
         type: PathPrefix
diff --git a/pkg/epp/common/config/loader/configloader_test.go b/pkg/epp/common/config/loader/configloader_test.go
new file mode 100644
index 0000000..475a594
--- /dev/null
+++ b/pkg/epp/common/config/loader/configloader_test.go
@@ -0,0 +1,773 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package loader
+
+import (
+	"context"
+	"encoding/json"
+	"testing"
+
+	"github.com/google/go-cmp/cmp"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	configapi "sigs.k8s.io/gateway-api-inference-extension/api/config/v1alpha1"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+	"sigs.k8s.io/gateway-api-inference-extension/test/utils"
+)
+
+const (
+	testProfileHandlerType = "test-profile-handler"
+	test1Type              = "test-one"
+	test2Type              = "test-two"
+	testPickerType         = "test-picker"
+)
+
+func TestLoadConfiguration(t *testing.T) {
+	test2Weight := 50
+
+	registerTestPlugins()
+
+	goodConfig := &configapi.EndpointPickerConfig{
+		TypeMeta: metav1.TypeMeta{
+			Kind:       "EndpointPickerConfig",
+			APIVersion: "inference.networking.x-k8s.io/v1alpha1",
+		},
+		Plugins: []configapi.PluginSpec{
+			{
+				Name:       "test1",
+				PluginName: test1Type,
+				Parameters: json.RawMessage("{\"threshold\":10}"),
+			},
+			{
+				Name:       "profileHandler",
+				PluginName: "test-profile-handler",
+			},
+			{
+				Name:       test2Type,
+				PluginName: test2Type,
+				Parameters: json.RawMessage("{\"hashBlockSize\":32}"),
+			},
+			{
+				Name:       "testPicker",
+				PluginName: testPickerType,
+			},
+		},
+		SchedulingProfiles: []configapi.SchedulingProfile{
+			{
+				Name: "default",
+				Plugins: []configapi.SchedulingPlugin{
+					{
+						PluginRef: "test1",
+					},
+					{
+						PluginRef: "test-two",
+						Weight:    &test2Weight,
+					},
+					{
+						PluginRef: "testPicker",
+					},
+				},
+			},
+		},
+	}
+
+	tests := []struct {
+		name       string
+		configText string
+		configFile string
+		want       *configapi.EndpointPickerConfig
+		wantErr    bool
+	}{
+		{
+			name:       "success",
+			configText: successConfigText,
+			configFile: "",
+			want:       goodConfig,
+			wantErr:    false,
+		},
+		{
+			name:       "errorBadYaml",
+			configText: errorBadYamlText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorNoProfileHandler",
+			configText: errorNoProfileHandlerText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadPluginReferenceText",
+			configText: errorBadPluginReferenceText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadPluginReferencePluginText",
+			configText: errorBadPluginReferencePluginText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorNoProfiles",
+			configText: errorNoProfilesText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorNoProfileName",
+			configText: errorNoProfileNameText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorNoProfilePlugins",
+			configText: errorNoProfilePluginsText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadProfilePlugin",
+			configText: errorBadProfilePluginText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadProfilePluginRef",
+			configText: errorBadProfilePluginRefText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadProfilePluginName",
+			configText: errorBadProfilePluginNameText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorDuplicatePlugin",
+			configText: errorDuplicatePluginText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "errorDuplicateProfile",
+			configText: errorDuplicateProfileText,
+			configFile: "",
+			wantErr:    true,
+		},
+		{
+			name:       "successFromFile",
+			configText: "",
+			configFile: "../../../../../test/testdata/configloader_1_test.yaml",
+			want:       goodConfig,
+			wantErr:    false,
+		},
+		{
+			name:       "noSuchFile",
+			configText: "",
+			configFile: "../../../../../test/testdata/configloader_error_test.yaml",
+			wantErr:    true,
+		},
+	}
+
+	for _, test := range tests {
+		got, err := LoadConfig([]byte(test.configText), test.configFile)
+		if err != nil {
+			if !test.wantErr {
+				t.Fatalf("In test %s LoadConfig returned unexpected error: %v, want %v", test.name, err, test.wantErr)
+			}
+			t.Logf("error was %s", err)
+		} else {
+			if test.wantErr {
+				t.Fatalf("In test %s LoadConfig did not return an expected error", test.name)
+			}
+			if diff := cmp.Diff(test.want, got); diff != "" {
+				t.Errorf("In test %s LoadConfig returned unexpected response, diff(-want, +got): %v", test.name, diff)
+			}
+		}
+	}
+}
+
+func TestLoadPluginReferences(t *testing.T) {
+	theConfig, err := LoadConfig([]byte(successConfigText), "")
+	if err != nil {
+		t.Fatalf("LoadConfig returned unexpected error: %v", err)
+	}
+	handle := utils.NewTestHandle()
+	err = LoadPluginReferences(theConfig.Plugins, handle)
+	if err != nil {
+		t.Fatalf("LoadPluginReferences returned unexpected error: %v", err)
+	}
+	if len(handle.Plugins().GetAllPlugins()) == 0 {
+		t.Fatalf("LoadPluginReferences returned an empty set of references")
+	}
+	if t1 := handle.Plugins().Plugin("test1"); t1 == nil {
+		t.Fatalf("LoadPluginReferences returned references did not contain test1")
+	} else if _, ok := t1.(*test1); !ok {
+		t.Fatalf("LoadPluginReferences returned references value for test1 has the wrong type %#v", t1)
+	}
+
+	theConfig, err = LoadConfig([]byte(errorBadPluginReferenceParametersText), "")
+	if err != nil {
+		t.Fatalf("LoadConfig returned unexpected error: %v", err)
+	}
+	err = LoadPluginReferences(theConfig.Plugins, utils.NewTestHandle())
+	if err == nil {
+		t.Fatalf("LoadPluginReferences did not return the expected error")
+	}
+}
+
+func TestInstantiatePlugin(t *testing.T) {
+	plugSpec := configapi.PluginSpec{PluginName: "plover"}
+	_, err := instantiatePlugin(plugSpec, utils.NewTestHandle())
+	if err == nil {
+		t.Fatalf("InstantiatePlugin did not return the expected error")
+	}
+}
+
+func TestLoadSchedulerConfig(t *testing.T) {
+	tests := []struct {
+		name       string
+		configText string
+		wantErr    bool
+	}{
+		{
+			name:       "schedulerSuccess",
+			configText: successSchedulerConfigText,
+			wantErr:    false,
+		},
+		{
+			name:       "errorBadPluginJson",
+			configText: errorBadPluginJsonText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorBadReferenceNoWeight",
+			configText: errorBadReferenceNoWeightText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorPluginReferenceJson",
+			configText: errorPluginReferenceJsonText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorTwoPickers",
+			configText: errorTwoPickersText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorConfig",
+			configText: errorConfigText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorTwoProfileHandlers",
+			configText: errorTwoProfileHandlersText,
+			wantErr:    true,
+		},
+		{
+			name:       "errorNoProfileHandlers",
+			configText: errorNoProfileHandlersText,
+			wantErr:    true,
+		},
+	}
+
+	registerNeededPlgugins()
+
+	for _, test := range tests {
+		theConfig, err := LoadConfig([]byte(test.configText), "")
+		if err != nil {
+			if test.wantErr {
+				continue
+			}
+			t.Fatalf("LoadConfig returned unexpected error: %v", err)
+		}
+		handle := utils.NewTestHandle()
+		err = LoadPluginReferences(theConfig.Plugins, handle)
+		if err != nil {
+			if test.wantErr {
+				continue
+			}
+			t.Fatalf("LoadPluginReferences returned unexpected error: %v", err)
+		}
+
+		_, err = LoadSchedulerConfig(theConfig.SchedulingProfiles, handle)
+		if err != nil {
+			if !test.wantErr {
+				t.Errorf("LoadSchedulerConfig returned an unexpected error. error %v", err)
+			}
+		} else if test.wantErr {
+			t.Errorf("LoadSchedulerConfig did not return an expected error (%s)", test.name)
+		}
+	}
+}
+
+func registerNeededPlgugins() {
+	plugins.Register(filter.LowQueueFilterType, filter.LowQueueFilterFactory)
+	plugins.Register(prefix.PrefixCachePluginType, prefix.PrefixCachePluginFactory)
+	plugins.Register(picker.MaxScorePickerType, picker.MaxScorePickerFactory)
+	plugins.Register(picker.RandomPickerType, picker.RandomPickerFactory)
+	plugins.Register(profile.SingleProfileHandlerType, profile.SingleProfileHandlerFactory)
+}
+
+// The following multi-line string constants, cause false positive lint errors (dupword)
+
+//nolint:dupword
+const successConfigText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: profileHandler
+  pluginName: test-profile-handler
+- pluginName: test-two
+  parameters:
+    hashBlockSize: 32
+- name: testPicker
+  pluginName: test-picker
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: test1
+  - pluginRef: test-two
+    weight: 50
+  - pluginRef: testPicker
+`
+
+//nolint:dupword
+const errorBadYamlText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- testing 1 2 3
+`
+
+//nolint:dupword
+const errorBadPluginReferenceText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- parameters:
+    a: 1234
+`
+
+//nolint:dupword
+const errorBadPluginReferencePluginText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: testx
+  pluginName: test-x
+- name: profileHandler
+  pluginName: test-profile-handler
+`
+
+//nolint:dupword
+const errorNoProfileHandlerText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+schedulingProfiles:
+- name: default
+`
+
+//nolint:dupword
+const errorNoProfilesText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: profileHandler
+  pluginName: test-profile-handler
+`
+
+//nolint:dupword
+const errorNoProfileNameText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- plugins:
+  - pluginRef: test1
+`
+
+//nolint:dupword
+const errorNoProfilePluginsText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+`
+
+//nolint:dupword
+const errorBadProfilePluginText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - weight: 10
+`
+
+//nolint:dupword
+const errorBadProfilePluginRefText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: plover
+`
+
+//nolint:dupword
+const errorBadProfilePluginNameText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: plover
+`
+
+//nolint:dupword
+const errorBadPluginReferenceParametersText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: asdf
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: test1
+`
+
+//nolint:dupword
+const errorDuplicatePluginText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 20
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: test1
+`
+
+//nolint:dupword
+const errorDuplicateProfileText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: test2
+  pluginName: test-one
+  parameters:
+    threshold: 20
+- name: profileHandler
+  pluginName: test-profile-handler
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: test1
+- name: default
+  plugins:
+  - pluginRef: test2
+`
+
+// compile-time type validation
+var _ framework.Filter = &test1{}
+
+type test1 struct {
+	Threshold int `json:"threshold"`
+}
+
+func (f *test1) Type() string {
+	return test1Type
+}
+
+// Filter filters out pods that doesn't meet the filter criteria.
+func (f *test1) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
+	return pods
+}
+
+// compile-time type validation
+var _ framework.Scorer = &test2{}
+var _ framework.PostCycle = &test2{}
+
+type test2 struct{}
+
+func (f *test2) Type() string {
+	return test2Type
+}
+
+func (m *test2) Score(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, _ []types.Pod) map[types.Pod]float64 {
+	return map[types.Pod]float64{}
+}
+
+func (m *test2) PostCycle(_ context.Context, _ *types.CycleState, _ *types.ProfileRunResult) {}
+
+// compile-time type validation
+var _ framework.Picker = &testPicker{}
+
+type testPicker struct{}
+
+func (p *testPicker) Type() string {
+	return testPickerType
+}
+
+func (p *testPicker) Pick(_ context.Context, _ *types.CycleState, _ []*types.ScoredPod) *types.ProfileRunResult {
+	return nil
+}
+
+// compile-time type validation
+var _ framework.ProfileHandler = &testProfileHandler{}
+
+type testProfileHandler struct{}
+
+func (p *testProfileHandler) Type() string {
+	return testProfileHandlerType
+}
+
+func (p *testProfileHandler) Pick(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, _ map[string]*framework.SchedulerProfile, _ map[string]*types.ProfileRunResult) map[string]*framework.SchedulerProfile {
+	return nil
+}
+
+func (p *testProfileHandler) ProcessResults(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, _ map[string]*types.ProfileRunResult) (*types.SchedulingResult, error) {
+	return nil, nil
+}
+
+func registerTestPlugins() {
+	plugins.Register(test1Type,
+		func(name string, parameters json.RawMessage, handle plugins.Handle) (plugins.Plugin, error) {
+			result := test1{}
+			err := json.Unmarshal(parameters, &result)
+			return &result, err
+		},
+	)
+
+	plugins.Register(test2Type,
+		func(name string, parameters json.RawMessage, handle plugins.Handle) (plugins.Plugin, error) {
+			return &test2{}, nil
+		},
+	)
+
+	plugins.Register(testPickerType,
+		func(name string, parameters json.RawMessage, handle plugins.Handle) (plugins.Plugin, error) {
+			return &testPicker{}, nil
+		},
+	)
+
+	plugins.Register(testProfileHandlerType,
+		func(name string, parameters json.RawMessage, handle plugins.Handle) (plugins.Plugin, error) {
+			return &testProfileHandler{}, nil
+		},
+	)
+}
+
+//nolint:dupword
+const successSchedulerConfigText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: lowQueue
+  pluginName: low-queue
+  parameters:
+    threshold: 10
+- name: prefixCache
+  pluginName: prefix-cache
+  parameters:
+    hashBlockSize: 32
+- name: maxScore
+  pluginName: max-score
+- name: profileHandler
+  pluginName: single-profile
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: lowQueue
+  - pluginRef: prefixCache
+    weight: 50
+  - pluginRef: maxScore
+`
+
+//nolint:dupword
+const errorBadPluginJsonText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name:profileHandler
+  pluginName: single-profile
+- name: prefixCache
+  pluginName: prefix-cache
+  parameters:
+    hashBlockSize: asdf
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: prefixCache
+    weight: 50
+`
+
+//nolint:dupword
+const errorBadReferenceNoWeightText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: profileHandler
+  pluginName: single-profile
+- name: prefixCache
+  pluginName: prefix-cache
+  parameters:
+    hashBlockSize: 32
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: prefixCache
+`
+
+//nolint:dupword
+const errorPluginReferenceJsonText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: lowQueue
+  pluginName: low-queue
+  parameters:
+    threshold: qwer
+- name: profileHandler
+  pluginName: single-profile
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: lowQueue
+`
+
+//nolint:dupword
+const errorTwoPickersText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: profileHandler
+  pluginName: single-profile
+- name: maxScore
+  pluginName: max-score
+- name: random
+  pluginName: random
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: maxScore
+  - pluginRef: random
+`
+
+//nolint:dupword
+const errorConfigText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: lowQueue
+  pluginName: low-queue
+  parameters:
+    threshold: 10
+`
+
+//nolint:dupword
+const errorTwoProfileHandlersText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: profileHandler
+  pluginName: single-profile
+- name: secondProfileHandler
+  pluginName: single-profile
+- name: maxScore
+  pluginName: max-score
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: maxScore
+`
+
+//nolint:dupword
+const errorNoProfileHandlersText = `
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: maxScore
+  pluginName: max-score
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: maxScore
+`
diff --git a/pkg/epp/requestcontrol/director_test.go b/pkg/epp/requestcontrol/director_test.go
index a7ca55a..9f16ed1 100644
--- a/pkg/epp/requestcontrol/director_test.go
+++ b/pkg/epp/requestcontrol/director_test.go
@@ -54,11 +54,11 @@ func (m *mockSaturationDetector) IsSaturated(_ context.Context) bool {
 }
 
 type mockScheduler struct {
-	scheduleResults map[string]*schedulingtypes.Result
+	scheduleResults *schedulingtypes.SchedulingResult
 	scheduleErr     error
 }
 
-func (m *mockScheduler) Schedule(ctx context.Context, req *schedulingtypes.LLMRequest) (map[string]*schedulingtypes.Result, error) {
+func (m *mockScheduler) Schedule(_ context.Context, _ *schedulingtypes.LLMRequest, _ []schedulingtypes.Pod) (*schedulingtypes.SchedulingResult, error) {
 	return m.scheduleResults, m.scheduleErr
 }
 
@@ -126,17 +126,20 @@ func TestDirector_HandleRequest(t *testing.T) {
 	}
 	ds.PodUpdateOrAddIfNotExist(testPod)
 
-	defaultSuccessfulScheduleResults := map[string]*schedulingtypes.Result{
-		"testProfile": {
-			TargetPod: &schedulingtypes.ScoredPod{
-				Pod: &schedulingtypes.PodMetrics{
-					Pod: &backend.Pod{
-						Address:        "192.168.1.100",
-						NamespacedName: k8stypes.NamespacedName{Name: "pod1", Namespace: "default"},
+	defaultSuccessfulScheduleResults := &schedulingtypes.SchedulingResult{
+		ProfileResults: map[string]*schedulingtypes.ProfileRunResult{
+			"testProfile": {
+				TargetPod: &schedulingtypes.ScoredPod{
+					Pod: &schedulingtypes.PodMetrics{
+						Pod: &backend.Pod{
+							Address:        "192.168.1.100",
+							NamespacedName: k8stypes.NamespacedName{Name: "pod1", Namespace: "default"},
+						},
 					},
 				},
 			},
 		},
+		PrimaryProfileName: "testProfile",
 	}
 
 	tests := []struct {
@@ -265,6 +268,27 @@ func TestDirector_HandleRequest(t *testing.T) {
 			},
 			wantMutatedBodyModel: "resolved-target-model-A",
 		},
+		{
+			name: "nonexistent target defined, use default inference model",
+			schedulerMockSetup: func(m *mockScheduler) {
+				m.scheduleResults = defaultSuccessfulScheduleResults
+			},
+			wantReqCtx: &handlers.RequestContext{
+				Model:               "food-review-1",
+				ResolvedTargetModel: "food-review-1",
+				TargetPod: &backend.Pod{
+					NamespacedName: types.NamespacedName{Namespace: "default", Name: "pod1"},
+					Address:        "192.168.1.100",
+				},
+				TargetEndpoint: "192.168.1.100:8000",
+			},
+			wantMutatedBodyModel: "food-review-1",
+			reqBodyMap: map[string]interface{}{
+				"model":  "food-review-1",
+				"prompt": "test prompt",
+			},
+			mockSaturationDetector: &mockSaturationDetector{isSaturated: false},
+		},
 		{
 
 			name: "request dropped (sheddable, saturated)",
@@ -295,24 +319,6 @@ func TestDirector_HandleRequest(t *testing.T) {
 			},
 			wantErrCode: errutil.BadRequest,
 		},
-		{
-			name: "invalid model defined, expect err",
-			reqBodyMap: map[string]interface{}{
-				"model":  "non-existent-model",
-				"prompt": "test prompt",
-			},
-			mockSaturationDetector: &mockSaturationDetector{isSaturated: false},
-			wantErrCode:            errutil.BadConfiguration,
-		},
-		{
-			name: "invalid target defined, expect err",
-			reqBodyMap: map[string]interface{}{
-				"model":  "food-review-1",
-				"prompt": "test prompt",
-			},
-			mockSaturationDetector: &mockSaturationDetector{isSaturated: false},
-			wantErrCode:            errutil.BadConfiguration,
-		},
 		{
 			name: "scheduler returns error",
 			reqBodyMap: map[string]interface{}{
@@ -344,7 +350,7 @@ func TestDirector_HandleRequest(t *testing.T) {
 			if test.schedulerMockSetup != nil {
 				test.schedulerMockSetup(mockSched)
 			}
-			director := NewDirector(ds, mockSched, test.mockSaturationDetector)
+			director := NewDirectorWithConfig(ds, mockSched, test.mockSaturationDetector, NewConfig())
 
 			reqCtx := &handlers.RequestContext{
 				Request: &handlers.Request{
@@ -515,14 +521,13 @@ func pointer(v int32) *int32 {
 
 func TestDirector_HandleResponse(t *testing.T) {
 	pr1 := &testPostResponse{
-		NameRes: "pr1",
+		TypeRes: "pr1",
 	}
 
 	ctx := logutil.NewTestLoggerIntoContext(context.Background())
 	ds := datastore.NewDatastore(t.Context(), nil)
 	mockSched := &mockScheduler{}
-	director := NewDirector(ds, mockSched, nil).
-		WithPostResponsePlugins(pr1)
+	director := NewDirectorWithConfig(ds, mockSched, nil, NewConfig().WithPostResponsePlugins(pr1))
 
 	reqCtx := &handlers.RequestContext{
 		Request: &handlers.Request{
@@ -554,12 +559,12 @@ func TestDirector_HandleResponse(t *testing.T) {
 }
 
 type testPostResponse struct {
-	NameRes                 string
+	TypeRes                 string
 	lastRespOnResponse      *Response
 	lastTargetPodOnResponse string
 }
 
-func (p *testPostResponse) Name() string { return p.NameRes }
+func (p *testPostResponse) Type() string { return p.TypeRes }
 
 func (p *testPostResponse) PostResponse(_ context.Context, _ *schedulingtypes.LLMRequest, response *Response, targetPod *backend.Pod) {
 	p.lastRespOnResponse = response
diff --git a/pkg/epp/scheduling/framework/plugins/filter/filter_test.go b/pkg/epp/scheduling/framework/plugins/filter/filter_test.go
index 27f6943..1f07c1d 100644
--- a/pkg/epp/scheduling/framework/plugins/filter/filter_test.go
+++ b/pkg/epp/scheduling/framework/plugins/filter/filter_test.go
@@ -35,11 +35,11 @@ var _ framework.Filter = &filterAll{}
 
 type filterAll struct{}
 
-func (f *filterAll) Name() string {
-	return "filter all"
+func (f *filterAll) Type() string {
+	return "filter-all"
 }
 
-func (f *filterAll) Filter(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (f *filterAll) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
 	return []types.Pod{}
 }
 
@@ -138,7 +138,7 @@ func TestFilter(t *testing.T) {
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			got := test.filter.Filter(context.Background(), test.req, types.NewCycleState(), test.input)
+			got := test.filter.Filter(context.Background(), types.NewCycleState(), test.req, test.input)
 
 			if diff := cmp.Diff(test.output, got); diff != "" {
 				t.Errorf("Unexpected output (-want +got): %v", diff)
@@ -203,10 +203,10 @@ func TestLoRASoftAffinityDistribution(t *testing.T) {
 	expectedAvailabilityPercent := 100 - expectedAffinityPercent
 
 	// initialize LoraAffinityFilter
-	LoraAffinityFilter := NewLoraAffinityFilter()
+	LoraAffinityFilter := NewLoraAffinityFilter(config.Conf.LoraAffinityThreshold)
 
 	for range numIterations {
-		result := LoraAffinityFilter.Filter(context.Background(), req, types.NewCycleState(), pods)
+		result := LoraAffinityFilter.Filter(context.Background(), types.NewCycleState(), req, pods)
 
 		// Check which type of pod was returned
 		if len(result) != 1 {
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go
index 596625d..2409850 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/indexer_test.go
@@ -22,24 +22,23 @@ import (
 )
 
 func TestIndexer_AddAndGet(t *testing.T) {
-	cache := newIndexer(2)
+	i := newIndexer(2)
 
 	hash1 := BlockHash(1)
 	server := ServerID{Namespace: "default", Name: "server1"}
-
 	// Add an entry to the cache
-	cache.Add([]BlockHash{hash1}, server)
+	i.Add([]BlockHash{hash1}, server)
 
 	// Retrieve the entry
-	assert.Equal(t, 1, cache.ll.Len(), "Cache size should be 1 after adding an entry")
-	servers := cache.Get(hash1)
+	assert.Equal(t, 1, i.podToLRU[server].Len(), "Cache size should be 1 after adding an entry")
+	servers := i.Get(hash1)
 	assert.Contains(t, servers, server, "Cache should contain the added server")
 
 	// Add another entry to the cache, the cache size should be incremented to 2.
-	cache.Add([]BlockHash{BlockHash(2)}, server)
-	assert.Equal(t, 2, cache.ll.Len(), "Cache size should  be 2 after adding an entry")
+	i.Add([]BlockHash{BlockHash(2)}, server)
+	assert.Equal(t, 2, i.podToLRU[server].Len(), "Cache size should  be 2 after adding an entry")
 
 	// Add another entry to the cache, which should evict the first one due to max size.
-	cache.Add([]BlockHash{BlockHash(3)}, server)
-	assert.Equal(t, 2, cache.ll.Len(), "Cache size should still be 2 after adding an entry")
+	i.Add([]BlockHash{BlockHash(3)}, server)
+	assert.Equal(t, 2, i.podToLRU[server].Len(), "Cache size should still be 2 after adding an entry")
 }
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
index 1d77dbc..73f02c3 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
@@ -18,6 +18,10 @@ package prefix
 
 import (
 	"context"
+	"fmt"
+	"math"
+	"math/rand"
+	"strings"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
@@ -27,10 +31,11 @@ import (
 )
 
 func TestPrefixPlugin(t *testing.T) {
+
 	config := Config{
 		HashBlockSize:          4,
 		MaxPrefixBlocksToMatch: DefaultMaxPrefixBlocks,
-		LRUIndexerCapacity:     DefaultLRUIndexerCapacity,
+		LRUCapacityPerServer:   DefaultLRUCapacityPerServer,
 	}
 	plugin := New(config)
 
@@ -44,7 +49,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaaaa",
 	}
 	cycleState1 := types.NewCycleState()
-	scores := plugin.Score(context.Background(), req1, cycleState1, pods)
+	scores := plugin.Score(context.Background(), cycleState1, req1, pods)
 	state, err := plugin.getPrefixState(cycleState1)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -56,7 +61,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
 	// Simulate pod1 was picked.
-	plugin.PostCycle(context.Background(), cycleState1, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(context.Background(), cycleState1, &types.ProfileRunResult{TargetPod: pod1})
 
 	// Second request doesn't share any prefix with first one. It should be added to the cache but
 	// the pod score should be 0.
@@ -65,7 +70,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "bbbbbb",
 	}
 	cycleState2 := types.NewCycleState()
-	scores = plugin.Score(context.Background(), req2, cycleState2, pods)
+	scores = plugin.Score(context.Background(), cycleState2, req2, pods)
 	state, err = plugin.getPrefixState(cycleState2)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -77,7 +82,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
 	// Simulate pod2 was picked.
-	plugin.PostCycle(context.Background(), cycleState2, &types.Result{TargetPod: pod2})
+	plugin.PostCycle(context.Background(), cycleState2, &types.ProfileRunResult{TargetPod: pod2})
 
 	// Third request shares partial prefix with first one.
 	req3 := &types.LLMRequest{
@@ -85,7 +90,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbb",
 	}
 	cycleState3 := types.NewCycleState()
-	scores = plugin.Score(context.Background(), req3, cycleState3, pods)
+	scores = plugin.Score(context.Background(), cycleState3, req3, pods)
 	state, err = plugin.getPrefixState(cycleState3)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -96,7 +101,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(2)/float64(3), scores[pod1], "score should be 2/3 - the model and the first prefix block match")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostCycle(context.Background(), cycleState3, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(context.Background(), cycleState3, &types.ProfileRunResult{TargetPod: pod1})
 
 	// 4th request is same as req3 except the model is different, still no match.
 	req4 := &types.LLMRequest{
@@ -104,7 +109,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbb",
 	}
 	cycleState4 := types.NewCycleState()
-	scores = plugin.Score(context.Background(), req4, cycleState4, pods)
+	scores = plugin.Score(context.Background(), cycleState4, req4, pods)
 	state, err = plugin.getPrefixState(cycleState4)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -115,7 +120,7 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, float64(0), scores[pod1], "score for pod1")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostCycle(context.Background(), cycleState4, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(context.Background(), cycleState4, &types.ProfileRunResult{TargetPod: pod1})
 
 	// 5th request shares partial prefix with 3rd one.
 	req5 := &types.LLMRequest{
@@ -123,7 +128,7 @@ func TestPrefixPlugin(t *testing.T) {
 		Prompt:      "aaaabbbbcccc",
 	}
 	cycleState5 := types.NewCycleState()
-	scores = plugin.Score(context.Background(), req5, cycleState5, pods)
+	scores = plugin.Score(context.Background(), cycleState5, req5, pods)
 	state, err = plugin.getPrefixState(cycleState5)
 	assert.NoError(t, err)
 	t.Logf("Hashes %+v, cached servers: %+v", state.PrefixHashes, state.PrefixCacheServers)
@@ -134,5 +139,63 @@ func TestPrefixPlugin(t *testing.T) {
 	assert.Equal(t, 0.75, scores[pod1], "score should be 0.75 - the model and the first 2 prefix blocks match")
 	assert.Equal(t, float64(0), scores[pod2], "score for pod2")
 
-	plugin.PostCycle(context.Background(), cycleState5, &types.Result{TargetPod: pod1})
+	plugin.PostCycle(context.Background(), cycleState5, &types.ProfileRunResult{TargetPod: pod1})
+}
+
+// TestPrefixPluginStress is a stress test for the prefix scoring plugin, using prompts of increasing length.
+func BenchmarkPrefixPluginStress(b *testing.B) {
+	blockSize := 4
+	maxPrefixBlocks := 50000
+	config := Config{
+		HashBlockSize:          blockSize,
+		MaxPrefixBlocksToMatch: maxPrefixBlocks,
+		LRUCapacityPerServer:   DefaultLRUCapacityPerServer,
+	}
+
+	plugin := New(config)
+	types.NewCycleState()
+	var promptLen []int
+	for i := 1; i <= 1024; i++ {
+		promptLen = append(promptLen, i)
+	}
+	promptLen = append(promptLen, 2048, 4096, 8192, 10000, 20000, 50000)
+
+	for _, i := range promptLen {
+		// Generate increasing-length random prompts
+		prompt := randomPrompt(4 + i)
+		pod := &types.PodMetrics{
+			Pod: &backend.Pod{
+				NamespacedName: k8stypes.NamespacedName{
+					Name: fmt.Sprintf("random-pod-%d", i),
+				},
+			},
+		}
+
+		pods := []types.Pod{pod}
+		req := &types.LLMRequest{
+			TargetModel: "model-stress",
+			Prompt:      prompt,
+		}
+
+		// First cycle: simulate scheduling and insert prefix info into the cache
+		cycleState := types.NewCycleState()
+		plugin.Score(context.Background(), cycleState, req, pods)
+		plugin.PostCycle(context.Background(), cycleState, &types.ProfileRunResult{TargetPod: pod})
+
+		// Second cycle: validate internal state
+		state, err := plugin.getPrefixState(cycleState)
+		assert.NoError(b, err)
+		expectedHashes := int(math.Min(float64(maxPrefixBlocks+1), float64(len(req.Prompt)/blockSize+1))) // the extra one is for the model.
+		assert.Equal(b, expectedHashes, len(state.PrefixHashes), "number of hashes is incorrect")
+	}
+}
+
+// randomPrompt generates a pseudo-random string of length n using lowercase letters.
+func randomPrompt(n int) string {
+	runes := []rune("abcdefghijklmnopqrstuvwxyz")
+	var sb strings.Builder
+	for i := 0; i < n; i++ {
+		sb.WriteRune(runes[rand.Intn(len(runes))])
+	}
+	return sb.String()
 }
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go b/pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go
index 7ed4e3c..c0eeb52 100644
--- a/pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/kvcache_test.go
@@ -83,7 +83,7 @@ func TestKvCacheScorer(t *testing.T) {
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
 			scorer := &KVCacheScorer{}
-			scores := scorer.Score(context.Background(), &types.LLMRequest{}, types.NewCycleState(), test.pods)
+			scores := scorer.Score(context.Background(), types.NewCycleState(), &types.LLMRequest{}, test.pods)
 
 			for i, pod := range test.pods {
 				expectedScore := test.expectedScoresPod[i]
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/queue_test.go b/pkg/epp/scheduling/framework/plugins/scorer/queue_test.go
index 70422ff..a9a8115 100644
--- a/pkg/epp/scheduling/framework/plugins/scorer/queue_test.go
+++ b/pkg/epp/scheduling/framework/plugins/scorer/queue_test.go
@@ -73,7 +73,7 @@ func TestQueueScorer(t *testing.T) {
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			scores := scorer.Score(context.Background(), &types.LLMRequest{}, types.NewCycleState(), test.pods)
+			scores := scorer.Score(context.Background(), types.NewCycleState(), &types.LLMRequest{}, test.pods)
 
 			for i, pod := range test.pods {
 				expectedScore := test.expectedScoresPod[i]
diff --git a/pkg/epp/scheduling/framework/scheduler_profile_test.go b/pkg/epp/scheduling/framework/scheduler_profile_test.go
index a7ad081..7fbc9e8 100644
--- a/pkg/epp/scheduling/framework/scheduler_profile_test.go
+++ b/pkg/epp/scheduling/framework/scheduler_profile_test.go
@@ -30,21 +30,21 @@ import (
 
 func TestSchedulePlugins(t *testing.T) {
 	tp1 := &testPlugin{
-		NameRes:   "test1",
+		TypeRes:   "test1",
 		ScoreRes:  0.3,
 		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}, {Name: "pod3"}},
 	}
 	tp2 := &testPlugin{
-		NameRes:   "test2",
+		TypeRes:   "test2",
 		ScoreRes:  0.8,
 		FilterRes: []k8stypes.NamespacedName{{Name: "pod1"}, {Name: "pod2"}},
 	}
 	tp_filterAll := &testPlugin{
-		NameRes:   "filter all",
+		TypeRes:   "filter all",
 		FilterRes: []k8stypes.NamespacedName{},
 	}
 	pickerPlugin := &testPlugin{
-		NameRes: "picker",
+		TypeRes: "picker",
 		PickRes: k8stypes.NamespacedName{Name: "pod1"},
 	}
 
@@ -144,7 +144,7 @@ func TestSchedulePlugins(t *testing.T) {
 			wantPod := &types.PodMetrics{
 				Pod: &backend.Pod{NamespacedName: test.wantTargetPod, Labels: make(map[string]string)},
 			}
-			wantRes := &types.Result{
+			wantRes := &types.ProfileRunResult{
 				TargetPod: wantPod,
 			}
 
@@ -155,24 +155,24 @@ func TestSchedulePlugins(t *testing.T) {
 			for _, plugin := range test.profile.filters {
 				tp, _ := plugin.(*testPlugin)
 				if tp.FilterCallCount != 1 {
-					t.Errorf("Plugin %s Filter() called %d times, expected 1", plugin.Name(), tp.FilterCallCount)
+					t.Errorf("Plugin %s Filter() called %d times, expected 1", plugin.Type(), tp.FilterCallCount)
 				}
 			}
 			for _, plugin := range test.profile.scorers {
 				tp, _ := plugin.Scorer.(*testPlugin)
 				if tp.ScoreCallCount != 1 {
-					t.Errorf("Plugin %s Score() called %d times, expected 1", plugin.Name(), tp.ScoreCallCount)
+					t.Errorf("Plugin %s Score() called %d times, expected 1", plugin.Type(), tp.ScoreCallCount)
 				}
 				if test.numPodsToScore != tp.NumOfScoredPods {
-					t.Errorf("Plugin %s Score() called with %d pods, expected %d", plugin.Name(), tp.NumOfScoredPods, test.numPodsToScore)
+					t.Errorf("Plugin %s Score() called with %d pods, expected %d", plugin.Type(), tp.NumOfScoredPods, test.numPodsToScore)
 				}
 			}
 			tp, _ := test.profile.picker.(*testPlugin)
 			if tp.NumOfPickerCandidates != test.numPodsToScore {
-				t.Errorf("Picker plugin %s Pick() called with %d candidates, expected %d", tp.Name(), tp.NumOfPickerCandidates, tp.NumOfScoredPods)
+				t.Errorf("Picker plugin %s Pick() called with %d candidates, expected %d", tp.Type(), tp.NumOfPickerCandidates, tp.NumOfScoredPods)
 			}
 			if tp.PickCallCount != 1 {
-				t.Errorf("Picker plugin %s Pick() called %d times, expected 1", tp.Name(), tp.PickCallCount)
+				t.Errorf("Picker plugin %s Pick() called %d times, expected 1", tp.Type(), tp.PickCallCount)
 			}
 			if tp.WinnderPodScore != test.targetPodScore {
 				t.Errorf("winnder pod score %v, expected %v", tp.WinnderPodScore, test.targetPodScore)
@@ -180,7 +180,7 @@ func TestSchedulePlugins(t *testing.T) {
 			for _, plugin := range test.profile.postCyclePlugins {
 				tp, _ := plugin.(*testPlugin)
 				if tp.PostScheduleCallCount != 1 {
-					t.Errorf("Plugin %s PostSchedule() called %d times, expected 1", plugin.Name(), tp.PostScheduleCallCount)
+					t.Errorf("Plugin %s PostSchedule() called %d times, expected 1", plugin.Type(), tp.PostScheduleCallCount)
 				}
 			}
 		})
@@ -195,7 +195,7 @@ var _ PostCycle = &testPlugin{}
 
 // testPlugin is an implementation useful in unit tests.
 type testPlugin struct {
-	NameRes               string
+	TypeRes               string
 	ScoreCallCount        int
 	NumOfScoredPods       int
 	ScoreRes              float64
@@ -208,15 +208,15 @@ type testPlugin struct {
 	WinnderPodScore       float64
 }
 
-func (tp *testPlugin) Name() string { return tp.NameRes }
+func (tp *testPlugin) Type() string { return tp.TypeRes }
 
-func (tp *testPlugin) Filter(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) []types.Pod {
+func (tp *testPlugin) Filter(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) []types.Pod {
 	tp.FilterCallCount++
 	return findPods(pods, tp.FilterRes...)
 
 }
 
-func (tp *testPlugin) Score(_ context.Context, _ *types.LLMRequest, _ *types.CycleState, pods []types.Pod) map[types.Pod]float64 {
+func (tp *testPlugin) Score(_ context.Context, _ *types.CycleState, _ *types.LLMRequest, pods []types.Pod) map[types.Pod]float64 {
 	tp.ScoreCallCount++
 	scoredPods := make(map[types.Pod]float64, len(pods))
 	for _, pod := range pods {
@@ -226,7 +226,7 @@ func (tp *testPlugin) Score(_ context.Context, _ *types.LLMRequest, _ *types.Cyc
 	return scoredPods
 }
 
-func (tp *testPlugin) Pick(_ context.Context, _ *types.CycleState, scoredPods []*types.ScoredPod) *types.Result {
+func (tp *testPlugin) Pick(_ context.Context, _ *types.CycleState, scoredPods []*types.ScoredPod) *types.ProfileRunResult {
 	tp.PickCallCount++
 	tp.NumOfPickerCandidates = len(scoredPods)
 
@@ -238,10 +238,10 @@ func (tp *testPlugin) Pick(_ context.Context, _ *types.CycleState, scoredPods []
 		}
 	}
 
-	return &types.Result{TargetPod: winnerPod}
+	return &types.ProfileRunResult{TargetPod: winnerPod}
 }
 
-func (tp *testPlugin) PostCycle(_ context.Context, _ *types.CycleState, res *types.Result) {
+func (tp *testPlugin) PostCycle(_ context.Context, _ *types.CycleState, res *types.ProfileRunResult) {
 	tp.PostScheduleCallCount++
 }
 
diff --git a/pkg/epp/scheduling/scheduler_test.go b/pkg/epp/scheduling/scheduler_test.go
index 413a41f..720a6b4 100644
--- a/pkg/epp/scheduling/scheduler_test.go
+++ b/pkg/epp/scheduling/scheduler_test.go
@@ -33,17 +33,17 @@ func TestSchedule(t *testing.T) {
 	tests := []struct {
 		name    string
 		req     *types.LLMRequest
-		input   []*backendmetrics.FakePodMetrics
-		wantRes map[string]*types.Result
+		input   []backendmetrics.PodMetrics
+		wantRes *types.SchedulingResult
 		err     bool
 	}{
 		{
-			name: "no pods in datastore",
+			name: "no candidate pods",
 			req: &types.LLMRequest{
 				TargetModel: "any-model",
 				RequestId:   uuid.NewString(),
 			},
-			input:   []*backendmetrics.FakePodMetrics{},
+			input:   []backendmetrics.PodMetrics{},
 			wantRes: nil,
 			err:     true,
 		},
@@ -55,8 +55,8 @@ func TestSchedule(t *testing.T) {
 			},
 			// pod2 will be picked because it has relatively low queue size, with the requested
 			// model being active, and has low KV cache.
-			input: []*backendmetrics.FakePodMetrics{
-				{
+			input: []backendmetrics.PodMetrics{
+				&backendmetrics.FakePodMetrics{
 					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}},
 					Metrics: &backendmetrics.MetricsState{
 						WaitingQueueSize:    0,
@@ -68,7 +68,7 @@ func TestSchedule(t *testing.T) {
 						},
 					},
 				},
-				{
+				&backendmetrics.FakePodMetrics{
 					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}},
 					Metrics: &backendmetrics.MetricsState{
 						WaitingQueueSize:    3,
@@ -80,7 +80,7 @@ func TestSchedule(t *testing.T) {
 						},
 					},
 				},
-				{
+				&backendmetrics.FakePodMetrics{
 					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}},
 					Metrics: &backendmetrics.MetricsState{
 						WaitingQueueSize:    10,
@@ -92,32 +92,35 @@ func TestSchedule(t *testing.T) {
 					},
 				},
 			},
-			wantRes: map[string]*types.Result{
-				"default": {
-					TargetPod: &types.ScoredPod{
-						Pod: &types.PodMetrics{
-							Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}, Labels: make(map[string]string)},
-							MetricsState: &backendmetrics.MetricsState{
-								WaitingQueueSize:    3,
-								KVCacheUsagePercent: 0.1,
-								MaxActiveModels:     2,
-								ActiveModels: map[string]int{
-									"foo":      1,
-									"critical": 1,
+			wantRes: &types.SchedulingResult{
+				ProfileResults: map[string]*types.ProfileRunResult{
+					"default": {
+						TargetPod: &types.ScoredPod{
+							Pod: &types.PodMetrics{
+								Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}, Labels: make(map[string]string)},
+								MetricsState: &backendmetrics.MetricsState{
+									WaitingQueueSize:    3,
+									KVCacheUsagePercent: 0.1,
+									MaxActiveModels:     2,
+									ActiveModels: map[string]int{
+										"foo":      1,
+										"critical": 1,
+									},
+									WaitingModels: map[string]int{},
 								},
-								WaitingModels: map[string]int{},
 							},
 						},
 					},
 				},
+				PrimaryProfileName: "default",
 			},
 		},
 	}
 
 	for _, test := range tests {
 		t.Run(test.name, func(t *testing.T) {
-			scheduler := NewScheduler(&fakeDataStore{pods: test.input})
-			got, err := scheduler.Schedule(context.Background(), test.req)
+			scheduler := NewScheduler()
+			got, err := scheduler.Schedule(context.Background(), test.req, types.ToSchedulerPodMetrics(test.input))
 			if test.err != (err != nil) {
 				t.Errorf("Unexpected error, got %v, want %v", err, test.err)
 			}
@@ -128,15 +131,3 @@ func TestSchedule(t *testing.T) {
 		})
 	}
 }
-
-type fakeDataStore struct {
-	pods []*backendmetrics.FakePodMetrics
-}
-
-func (fds *fakeDataStore) PodGetAll() []backendmetrics.PodMetrics {
-	pm := make([]backendmetrics.PodMetrics, 0, len(fds.pods))
-	for _, pod := range fds.pods {
-		pm = append(pm, pod)
-	}
-	return pm
-}
diff --git a/pkg/epp/server/server_test.go b/pkg/epp/server/server_test.go
new file mode 100644
index 0000000..3696f5a
--- /dev/null
+++ b/pkg/epp/server/server_test.go
@@ -0,0 +1,192 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package server
+
+import (
+	"context"
+	"fmt"
+	"testing"
+
+	pb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers"
+	testutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/testing"
+	"sigs.k8s.io/gateway-api-inference-extension/test/utils"
+)
+
+const (
+	bufSize                    = 1024 * 1024
+	podName                    = "pod1"
+	podAddress                 = "1.2.3.4"
+	poolPort                   = int32(5678)
+	destinationEndpointHintKey = "test-target"
+	namespace                  = "ns1"
+)
+
+func TestServer(t *testing.T) {
+	theHeaderValue := "body"
+	requestHeader := "x-test"
+
+	expectedRequestHeaders := map[string]string{destinationEndpointHintKey: fmt.Sprintf("%s:%d", podAddress, poolPort),
+		"Content-Length": "42", ":method": "POST", requestHeader: theHeaderValue}
+	expectedResponseHeaders := map[string]string{"x-went-into-resp-headers": "true", ":method": "POST", requestHeader: theHeaderValue}
+	expectedSchedulerHeaders := map[string]string{":method": "POST", requestHeader: theHeaderValue}
+
+	t.Run("server", func(t *testing.T) {
+		tsModel := "food-review"
+		model := testutil.MakeInferenceModel("v1").
+			CreationTimestamp(metav1.Unix(1000, 0)).
+			ModelName(tsModel).ObjRef()
+
+		director := &testDirector{}
+		ctx, cancel, ds, _ := utils.PrepareForTestStreamingServer([]*v1alpha2.InferenceModel{model},
+			[]*v1.Pod{{ObjectMeta: metav1.ObjectMeta{Name: podName}}}, "test-pool1", namespace, poolPort)
+
+		streamingServer := handlers.NewStreamingServer(namespace, destinationEndpointHintKey, ds, director)
+
+		testListener, errChan := utils.SetupTestStreamingServer(t, ctx, ds, streamingServer)
+		process, conn := utils.GetStreamingServerClient(ctx, t)
+		defer conn.Close()
+
+		// Send request headers - no response expected
+		headers := utils.BuildEnvoyGRPCHeaders(map[string]string{requestHeader: theHeaderValue, ":method": "POST"}, true)
+		request := &pb.ProcessingRequest{
+			Request: &pb.ProcessingRequest_RequestHeaders{
+				RequestHeaders: headers,
+			},
+		}
+		err := process.Send(request)
+		if err != nil {
+			t.Error("Error sending request headers", err)
+		}
+
+		// Send request body
+		requestBody := "{\"model\":\"food-review\",\"prompt\":\"Is banana tasty?\"}"
+		expectedBody := "{\"model\":\"v1\",\"prompt\":\"Is banana tasty?\"}"
+		request = &pb.ProcessingRequest{
+			Request: &pb.ProcessingRequest_RequestBody{
+				RequestBody: &pb.HttpBody{
+					Body:        []byte(requestBody),
+					EndOfStream: true,
+				},
+			},
+		}
+		err = process.Send(request)
+		if err != nil {
+			t.Error("Error sending request body", err)
+		}
+
+		// Receive request headers and check
+		responseReqHeaders, err := process.Recv()
+		if err != nil {
+			t.Error("Error receiving response", err)
+		} else {
+			if responseReqHeaders == nil || responseReqHeaders.GetRequestHeaders() == nil ||
+				responseReqHeaders.GetRequestHeaders().Response == nil ||
+				responseReqHeaders.GetRequestHeaders().Response.HeaderMutation == nil ||
+				responseReqHeaders.GetRequestHeaders().Response.HeaderMutation.SetHeaders == nil {
+				t.Error("Invalid request headers response")
+			} else if !utils.CheckEnvoyGRPCHeaders(t, responseReqHeaders.GetRequestHeaders().Response, expectedRequestHeaders) {
+				t.Error("Incorrect request headers")
+			}
+		}
+
+		// Receive request body and check
+		responseReqBody, err := process.Recv()
+		if err != nil {
+			t.Error("Error receiving response", err)
+		} else {
+			if responseReqBody == nil || responseReqBody.GetRequestBody() == nil ||
+				responseReqBody.GetRequestBody().Response == nil ||
+				responseReqBody.GetRequestBody().Response.BodyMutation == nil ||
+				responseReqBody.GetRequestBody().Response.BodyMutation.GetStreamedResponse() == nil {
+				t.Error("Invalid request body response")
+			} else {
+				body := responseReqBody.GetRequestBody().Response.BodyMutation.GetStreamedResponse().Body
+				if string(body) != expectedBody {
+					t.Errorf("Incorrect body %s expected %s", string(body), expectedBody)
+				}
+			}
+		}
+
+		// Check headers passed to the scheduler
+		if len(director.requestHeaders) != 2 {
+			t.Errorf("Incorrect number of request headers %d instead of 2", len(director.requestHeaders))
+		}
+		for expectedKey, expectedValue := range expectedSchedulerHeaders {
+			got, ok := director.requestHeaders[expectedKey]
+			if !ok {
+				t.Errorf("Missing header %s", expectedKey)
+			} else if got != expectedValue {
+				t.Errorf("Incorrect value for header %s, want %s got %s", expectedKey, expectedValue, got)
+			}
+		}
+
+		// Send response headers
+		headers = utils.BuildEnvoyGRPCHeaders(map[string]string{requestHeader: theHeaderValue, ":method": "POST"}, false)
+		request = &pb.ProcessingRequest{
+			Request: &pb.ProcessingRequest_ResponseHeaders{
+				ResponseHeaders: headers,
+			},
+		}
+		err = process.Send(request)
+		if err != nil {
+			t.Error("Error sending response", err)
+		}
+
+		// Receive response headers and check
+		response, err := process.Recv()
+		if err != nil {
+			t.Error("Error receiving response", err)
+		} else {
+			if response == nil || response.GetResponseHeaders() == nil || response.GetResponseHeaders().Response == nil ||
+				response.GetResponseHeaders().Response.HeaderMutation == nil ||
+				response.GetResponseHeaders().Response.HeaderMutation.SetHeaders == nil {
+				t.Error("Invalid response")
+			} else if !utils.CheckEnvoyGRPCHeaders(t, response.GetResponseHeaders().Response, expectedResponseHeaders) {
+				t.Error("Incorrect response headers")
+			}
+		}
+
+		cancel()
+		<-errChan
+		testListener.Close()
+	})
+}
+
+type testDirector struct {
+	requestHeaders map[string]string
+}
+
+func (ts *testDirector) HandleRequest(ctx context.Context, reqCtx *handlers.RequestContext) (*handlers.RequestContext, error) {
+	ts.requestHeaders = reqCtx.Request.Headers
+
+	reqCtx.Request.Body["model"] = "v1"
+	reqCtx.TargetEndpoint = fmt.Sprintf("%s:%d", podAddress, poolPort)
+	return reqCtx, nil
+}
+
+func (ts *testDirector) HandleResponse(ctx context.Context, reqCtx *handlers.RequestContext) (*handlers.RequestContext, error) {
+	return reqCtx, nil
+}
+
+func (ts *testDirector) GetRandomPod() *backend.Pod {
+	return nil
+}
diff --git a/test/e2e/epp/e2e_suite_test.go b/test/e2e/epp/e2e_suite_test.go
index 6595923..04f4e89 100644
--- a/test/e2e/epp/e2e_suite_test.go
+++ b/test/e2e/epp/e2e_suite_test.go
@@ -18,6 +18,7 @@ package epp
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"os"
 	"strings"
@@ -70,6 +71,8 @@ const (
 	envoyPort = "8081"
 	// inferExtName is the name of the inference extension test resources.
 	inferExtName = "vllm-llama3-8b-instruct-epp"
+	// metricsReaderSecretName is the name of the metrics reader secret which stores sa token to read epp metrics.
+	metricsReaderSecretName = "inference-gateway-sa-metrics-reader-secret"
 	// clientManifest is the manifest for the client test resources.
 	clientManifest = "../../testdata/client.yaml"
 	// modelServerSecretManifest is the manifest for the model server secret resource.
@@ -82,6 +85,8 @@ const (
 	inferExtManifest = "../../testdata/inferencepool-e2e.yaml"
 	// envoyManifest is the manifest for the envoy proxy test resources.
 	envoyManifest = "../../testdata/envoy.yaml"
+	// metricsRbacManifest is the manifest for the rbac resources for testing metrics.
+	metricsRbacManifest = "../../testdata/metrics-rbac.yaml"
 	// modelServerManifestFilepathEnvVar is the env var that holds absolute path to the manifest for the model server test resource.
 	modelServerManifestFilepathEnvVar = "MANIFEST_PATH"
 )
@@ -133,6 +138,7 @@ func setupInfra() {
 	createInferExt(cli, inferExtManifest)
 	createClient(cli, clientManifest)
 	createEnvoy(cli, envoyManifest)
+	createMetricsRbac(cli, metricsRbacManifest)
 	// Run this step last, as it requires additional time for the model server to become ready.
 	createModelServer(cli, modelServerManifestArray, modelServerManifestPath)
 }
@@ -259,6 +265,30 @@ func createClient(k8sClient client.Client, filePath string) {
 	testutils.PodReady(ctx, k8sClient, pod, readyTimeout, interval)
 }
 
+// createMetricsRbac creates the metrics RBAC resources from the manifest file.
+func createMetricsRbac(k8sClient client.Client, filePath string) {
+	inManifests := readYaml(filePath)
+	ginkgo.By("Replacing placeholder namespace with E2E_NS environment variable")
+	outManifests := []string{}
+	for _, m := range inManifests {
+		outManifests = append(outManifests, strings.ReplaceAll(m, "$E2E_NS", nsName))
+	}
+	ginkgo.By("Creating RBAC resources for scraping metrics from manifest: " + filePath)
+	createObjsFromYaml(k8sClient, outManifests)
+
+	// wait for sa token to exist
+	testutils.EventuallyExists(ctx, func() error {
+		token, err := getMetricsReaderToken(k8sClient)
+		if err != nil {
+			return err
+		}
+		if len(token) == 0 {
+			return errors.New("failed to get metrics reader token")
+		}
+		return nil
+	}, existsTimeout, interval)
+}
+
 // createModelServer creates the model server resources used for testing from the given filePaths.
 func createModelServer(k8sClient client.Client, modelServerManifestArray []string, deployPath string) {
 	ginkgo.By("Creating model server resources from manifest: " + deployPath)
diff --git a/test/e2e/epp/e2e_test.go b/test/e2e/epp/e2e_test.go
index b1ba47a..7bf31ac 100644
--- a/test/e2e/epp/e2e_test.go
+++ b/test/e2e/epp/e2e_test.go
@@ -17,6 +17,8 @@ limitations under the License.
 package epp
 
 import (
+	"encoding/json"
+	"errors"
 	"fmt"
 	"strconv"
 	"strings"
@@ -26,9 +28,12 @@ import (
 	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
+	corev1 "k8s.io/api/core/v1"
+	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/ptr"
-	"sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+	client "sigs.k8s.io/controller-runtime/pkg/client"
+	v1alpha2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
 	testutils "sigs.k8s.io/gateway-api-inference-extension/test/utils"
 )
 
@@ -51,38 +56,57 @@ var _ = ginkgo.Describe("InferencePool", func() {
 	ginkgo.AfterEach(func() {
 		ginkgo.By("Deleting the InferenceModel test resource.")
 		cleanupInferModelResources()
+		gomega.Eventually(func() error {
+			err := cli.Get(ctx, types.NamespacedName{Namespace: infModel.Namespace, Name: infModel.Name}, infModel)
+			if err == nil {
+				return errors.New("InferenceModel resource still exists")
+			}
+			if !k8serrors.IsNotFound(err) {
+				return nil
+			}
+			return nil
+		}, existsTimeout, interval).Should(gomega.Succeed())
 	})
 
 	ginkgo.When("The Inference Extension is running", func() {
 		ginkgo.It("Should route traffic to target model servers", func() {
 			for _, t := range []struct {
 				api              string
-				promptOrMessages string
+				promptOrMessages any
 			}{
 				{
 					api:              "/completions",
 					promptOrMessages: "Write as if you were a critic: San Francisco",
 				},
 				{
-					api:              "/chat/completions",
-					promptOrMessages: `[{"role": "user", "content": "Write as if you were a critic: San Francisco"}]`,
+					api: "/chat/completions",
+					promptOrMessages: []map[string]any{
+						{
+							"role":    "user",
+							"content": "Write as if you were a critic: San Francisco",
+						},
+					},
 				},
 				{
 					api: "/chat/completions",
-					promptOrMessages: `[{"role": "user", "content": "Write as if you were a critic: San Francisco"},` +
-						`{"role": "assistant", "content": "Okay, let's see..."},` +
-						`{"role": "user", "content": "Now summarize your thoughts."}]`,
+					promptOrMessages: []map[string]any{
+						{
+							"role":    "user",
+							"content": "Write as if you were a critic: San Francisco",
+						},
+						{"role": "assistant", "content": "Okay, let's see..."},
+						{"role": "user", "content": "Now summarize your thoughts."},
+					},
 				},
 			} {
-				ginkgo.By("Verifying connectivity through the inference extension with " +
-					t.api + " api and prompt/messages: " + t.promptOrMessages)
+				ginkgo.By(fmt.Sprintf("Verifying connectivity through the inference extension with %s api and prompt/messages: %v", t.api, t.promptOrMessages))
 
 				// Ensure the expected responses include the inferencemodel target model names.
 				var expected []string
 				for _, m := range infModel.Spec.TargetModels {
 					expected = append(expected, m.Name)
 				}
-				curlCmd := getCurlCommand(envoyName, nsName, envoyPort, modelName, curlTimeout, t.api, t.promptOrMessages)
+				curlCmd := getCurlCommand(envoyName, nsName, envoyPort, modelName, curlTimeout, t.api, t.promptOrMessages, false)
 
 				actual := make(map[string]int)
 				gomega.Eventually(func() error {
@@ -106,11 +130,103 @@ var _ = ginkgo.Describe("InferencePool", func() {
 					if !cmp.Equal(got, expected, cmpopts.SortSlices(func(a, b string) bool { return a < b })) {
 						return fmt.Errorf("actual (%v) != expected (%v); resp=%q", got, expected, resp)
 					}
-
 					return nil
 				}, readyTimeout, curlInterval).Should(gomega.Succeed())
 			}
 		})
+
+		ginkgo.It("Should expose EPP metrics after generating traffic", func() {
+			// Define the metrics we expect to see
+			expectedMetrics := []string{
+				"inference_model_request_total",
+				"inference_model_request_error_total",
+				"inference_model_request_duration_seconds",
+				// TODO: normalized_time_per_output_token_seconds is not actually recorded yet
+				// "normalized_time_per_output_token_seconds",
+				"inference_model_request_sizes",
+				"inference_model_response_sizes",
+				"inference_model_input_tokens",
+				"inference_model_output_tokens",
+				"inference_pool_average_kv_cache_utilization",
+				"inference_pool_average_queue_size",
+				"inference_pool_per_pod_queue_size",
+				"inference_model_running_requests",
+				"inference_pool_ready_pods",
+				"inference_extension_info",
+			}
+
+			// Generate traffic by sending requests through the inference extension
+			ginkgo.By("Generating traffic through the inference extension")
+			curlCmd := getCurlCommand(envoyName, nsName, envoyPort, modelName, curlTimeout, "/completions", "Write as if you were a critic: San Francisco", true)
+
+			// Run the curl command multiple times to generate some metrics data
+			for i := 0; i < 5; i++ {
+				_, err := testutils.ExecCommandInPod(ctx, cfg, scheme, kubeCli, nsName, "curl", "curl", curlCmd)
+				gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			}
+
+			// modify the curl command to generate some error metrics
+			curlCmd[len(curlCmd)-1] = "invalid input"
+			for i := 0; i < 5; i++ {
+				_, err := testutils.ExecCommandInPod(ctx, cfg, scheme, kubeCli, nsName, "curl", "curl", curlCmd)
+				gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			}
+
+			// Now scrape metrics from the EPP endpoint via the curl pod
+			ginkgo.By("Scraping metrics from the EPP endpoint")
+
+			// Get Pod IP instead of Service
+			podList := &corev1.PodList{}
+			err := cli.List(ctx, podList, client.InNamespace(nsName), client.MatchingLabels{"app": inferExtName})
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			gomega.Expect(podList.Items).NotTo(gomega.BeEmpty())
+			podIP := podList.Items[0].Status.PodIP
+			gomega.Expect(podIP).NotTo(gomega.BeEmpty())
+
+			// Get the authorization token for reading metrics
+			token := ""
+			gomega.Eventually(func() error {
+				token, err = getMetricsReaderToken(cli)
+				if err != nil {
+					return err
+				}
+				if token == "" {
+					return errors.New("token not found")
+				}
+				return nil
+			}, existsTimeout, interval).Should(gomega.Succeed())
+
+			// Construct the metric scraping curl command using Pod IP
+			metricScrapeCmd := []string{
+				"curl",
+				"-i",
+				"--max-time",
+				strconv.Itoa((int)(curlTimeout.Seconds())),
+				"-H",
+				"Authorization: Bearer " + token,
+				fmt.Sprintf("http://%s:%d/metrics", podIP, 9090),
+			}
+
+			ginkgo.By("Verifying that all expected metrics are present.")
+			gomega.Eventually(func() error {
+				// Execute the metrics scrape command inside the curl pod
+				resp, err := testutils.ExecCommandInPod(ctx, cfg, scheme, kubeCli, nsName, "curl", "curl", metricScrapeCmd)
+				if err != nil {
+					return err
+				}
+				// Verify that we got a 200 OK responsecurl
+				if !strings.Contains(resp, "200 OK") {
+					return fmt.Errorf("did not get 200 OK: %s", resp)
+				}
+				// Check if all expected metrics are present in the metrics output
+				for _, metric := range expectedMetrics {
+					if !strings.Contains(resp, metric) {
+						return fmt.Errorf("expected metric %s not found in metrics output", metric)
+					}
+				}
+				return nil
+			}, readyTimeout, curlInterval).Should(gomega.Succeed())
+		})
 	})
 })
 
@@ -130,16 +246,38 @@ func newInferenceModel(ns string) *v1alpha2.InferenceModel {
 		Obj()
 }
 
+func getMetricsReaderToken(k8sClient client.Client) (string, error) {
+	secret := &corev1.Secret{}
+	err := k8sClient.Get(ctx, types.NamespacedName{Namespace: nsName, Name: metricsReaderSecretName}, secret)
+	if err != nil {
+		return "", err
+	}
+	return string(secret.Data["token"]), nil
+}
+
 // getCurlCommand returns the command, as a slice of strings, for curl'ing
 // the test model server at the given name, namespace, port, and model name.
-func getCurlCommand(name, ns, port, model string, timeout time.Duration, api string, promptOrMessages string) []string {
-	var body string
+func getCurlCommand(name, ns, port, model string, timeout time.Duration, api string, promptOrMessages any, streaming bool) []string {
+	body := map[string]any{
+		"model":       model,
+		"max_tokens":  100,
+		"temperature": 0,
+	}
+	body["model"] = model
 	switch api {
 	case "/completions":
-		body = fmt.Sprintf(`{"model": "%s", "prompt": "%s", "max_tokens": 100, "temperature": 0}`, model, promptOrMessages)
+		body["prompt"] = promptOrMessages
 	case "/chat/completions":
-		body = fmt.Sprintf(`{"model": "%s", "messages": %s, "max_tokens": 100, "temperature": 0}`, model, promptOrMessages)
+		body["messages"] = promptOrMessages
+	}
+	if streaming {
+		body["stream"] = true
+		body["stream_options"] = map[string]any{
+			"include_usage": true,
+		}
 	}
+	b, err := json.Marshal(body)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 	return []string{
 		"curl",
 		"-i",
@@ -149,6 +287,6 @@ func getCurlCommand(name, ns, port, model string, timeout time.Duration, api str
 		"-H",
 		"Content-Type: application/json",
 		"-d",
-		body,
+		string(b),
 	}
 }
diff --git a/test/integration/epp/hermetic_test.go b/test/integration/epp/hermetic_test.go
index 0b2c8dd..db62529 100644
--- a/test/integration/epp/hermetic_test.go
+++ b/test/integration/epp/hermetic_test.go
@@ -925,7 +925,7 @@ func BeforeSuite() func() {
 	// Adjust from defaults
 	serverRunner.PoolNamespacedName = types.NamespacedName{Name: testPoolName, Namespace: testNamespace}
 	serverRunner.Datastore = datastore.NewDatastore(context.Background(), pmf)
-	scheduler := scheduling.NewScheduler(serverRunner.Datastore)
+	scheduler := scheduling.NewScheduler()
 
 	sdConfig := &saturationdetector.Config{
 		QueueDepthThreshold:       saturationdetector.DefaultQueueDepthThreshold,
@@ -934,7 +934,7 @@ func BeforeSuite() func() {
 	}
 	detector := saturationdetector.NewDetector(sdConfig, serverRunner.Datastore, logger.WithName("saturation-detector"))
 	serverRunner.SaturationDetector = detector
-	serverRunner.Director = requestcontrol.NewDirector(serverRunner.Datastore, scheduler, detector)
+	serverRunner.Director = requestcontrol.NewDirectorWithConfig(serverRunner.Datastore, scheduler, detector, requestcontrol.NewConfig())
 	serverRunner.SecureServing = false
 
 	if err := serverRunner.SetupWithManager(context.Background(), mgr); err != nil {
diff --git a/test/testdata/configloader_1_test.yaml b/test/testdata/configloader_1_test.yaml
new file mode 100644
index 0000000..bd66b45
--- /dev/null
+++ b/test/testdata/configloader_1_test.yaml
@@ -0,0 +1,22 @@
+apiVersion: inference.networking.x-k8s.io/v1alpha1
+kind: EndpointPickerConfig
+plugins:
+- name: test1
+  pluginName: test-one
+  parameters:
+    threshold: 10
+- name: profileHandler
+  pluginName: test-profile-handler
+- pluginName: test-two
+  parameters:
+    hashBlockSize: 32
+- name: testPicker
+  pluginName: test-picker
+
+schedulingProfiles:
+- name: default
+  plugins:
+  - pluginRef: test1
+  - pluginRef: test-two
+    weight: 50
+  - pluginRef: testPicker
diff --git a/test/testdata/envoy.yaml b/test/testdata/envoy.yaml
index 3fff859..38418f9 100644
--- a/test/testdata/envoy.yaml
+++ b/test/testdata/envoy.yaml
@@ -170,6 +170,16 @@ data:
                 max_pending_requests: 40000
                 max_requests: 40000
                 max_retries: 1024
+          health_checks:
+            - timeout: 2s
+              interval: 10s
+              unhealthy_threshold: 3
+              healthy_threshold: 2
+              reuse_connection: true
+              grpc_health_check:
+                service_name: "envoy.service.ext_proc.v3.ExternalProcessor"
+              tls_options:
+                alpn_protocols: ["h2"]  
           # This ensures that envoy accepts untrusted certificates. We tried to explicitly
           # set TrustChainVerification to ACCEPT_UNSTRUSTED, but that actually didn't work
           # and what worked is setting the common_tls_context to empty.                
@@ -197,7 +207,6 @@ data:
                         socket_address:
                           address: vllm-llama3-8b-instruct-epp.$E2E_NS
                           port_value: 9002
-                    health_status: HEALTHY
                     load_balancing_weight: 1
 ---
 apiVersion: apps/v1
diff --git a/test/testdata/inferencepool-e2e.yaml b/test/testdata/inferencepool-e2e.yaml
index 0cd8934..261b00b 100644
--- a/test/testdata/inferencepool-e2e.yaml
+++ b/test/testdata/inferencepool-e2e.yaml
@@ -109,7 +109,7 @@ rules:
   - subjectaccessreviews
   verbs:
   - create
---- 
+---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
@@ -119,5 +119,6 @@ subjects:
   name: default
   namespace: $E2E_NS
 roleRef:
+  apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: pod-read
diff --git a/test/testdata/metrics-rbac.yaml b/test/testdata/metrics-rbac.yaml
new file mode 100644
index 0000000..0518916
--- /dev/null
+++ b/test/testdata/metrics-rbac.yaml
@@ -0,0 +1,37 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: inference-gateway-metrics-reader
+rules:
+- nonResourceURLs:
+  - /metrics
+  verbs:
+  - get
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: inference-gateway-sa-metrics-reader
+  namespace: $E2E_NS
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: inference-gateway-sa-metrics-reader-role-binding
+subjects:
+- kind: ServiceAccount
+  name: inference-gateway-sa-metrics-reader
+  namespace: $E2E_NS
+roleRef:
+  kind: ClusterRole
+  name: inference-gateway-metrics-reader
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: inference-gateway-sa-metrics-reader-secret
+  namespace: $E2E_NS
+  annotations:
+    kubernetes.io/service-account.name: inference-gateway-sa-metrics-reader
+type: kubernetes.io/service-account-token
\ No newline at end of file
diff --git a/test/utils/handle.go b/test/utils/handle.go
new file mode 100644
index 0000000..3358ad8
--- /dev/null
+++ b/test/utils/handle.go
@@ -0,0 +1,60 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package utils
+
+import "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+
+// testHandle is an implmentation of plugins.Handle for test purposes
+type testHandle struct {
+	plugins plugins.HandlePlugins
+}
+
+func (h *testHandle) Plugins() plugins.HandlePlugins {
+	return h.plugins
+}
+
+type testHandlePlugins struct {
+	thePlugins map[string]plugins.Plugin
+}
+
+func (h *testHandlePlugins) Plugin(name string) plugins.Plugin {
+	return h.thePlugins[name]
+}
+
+func (h *testHandlePlugins) AddPlugin(name string, plugin plugins.Plugin) {
+	h.thePlugins[name] = plugin
+}
+
+func (h *testHandlePlugins) GetAllPlugins() []plugins.Plugin {
+	result := make([]plugins.Plugin, 0)
+	for _, plugin := range h.thePlugins {
+		result = append(result, plugin)
+	}
+	return result
+}
+
+func (h *testHandlePlugins) GetAllPluginsWithNames() map[string]plugins.Plugin {
+	return h.thePlugins
+}
+
+func NewTestHandle() plugins.Handle {
+	return &testHandle{
+		plugins: &testHandlePlugins{
+			thePlugins: map[string]plugins.Plugin{},
+		},
+	}
+}
diff --git a/test/utils/server.go b/test/utils/server.go
new file mode 100644
index 0000000..f3d0a5a
--- /dev/null
+++ b/test/utils/server.go
@@ -0,0 +1,190 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package utils
+
+import (
+	"context"
+	"net"
+	"testing"
+	"time"
+
+	corev3 "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
+	pb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
+	"google.golang.org/grpc/test/bufconn"
+	v1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/fake"
+	"sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/datastore"
+	testutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/testing"
+)
+
+const bufSize = 1024 * 1024
+
+var testListener *bufconn.Listener
+
+func PrepareForTestStreamingServer(models []*v1alpha2.InferenceModel, pods []*v1.Pod, poolName string, namespace string,
+	poolPort int32) (context.Context, context.CancelFunc, datastore.Datastore, *metrics.FakePodMetricsClient) {
+	ctx, cancel := context.WithCancel(context.Background())
+
+	pmc := &metrics.FakePodMetricsClient{}
+	pmf := metrics.NewPodMetricsFactory(pmc, time.Second)
+	ds := datastore.NewDatastore(ctx, pmf)
+
+	initObjs := []client.Object{}
+	for _, model := range models {
+		initObjs = append(initObjs, model)
+		ds.ModelSetIfOlder(model)
+	}
+	for _, pod := range pods {
+		initObjs = append(initObjs, pod)
+		ds.PodUpdateOrAddIfNotExist(pod)
+	}
+
+	scheme := runtime.NewScheme()
+	_ = clientgoscheme.AddToScheme(scheme)
+	_ = v1alpha2.Install(scheme)
+	fakeClient := fake.NewClientBuilder().
+		WithScheme(scheme).
+		WithObjects(initObjs...).
+		Build()
+	pool := testutil.MakeInferencePool(poolName).Namespace(namespace).ObjRef()
+	pool.Spec.TargetPortNumber = poolPort
+	_ = ds.PoolSet(context.Background(), fakeClient, pool)
+
+	return ctx, cancel, ds, pmc
+}
+
+func SetupTestStreamingServer(t *testing.T, ctx context.Context, ds datastore.Datastore,
+	streamingServer pb.ExternalProcessorServer) (*bufconn.Listener, chan error) {
+	testListener = bufconn.Listen(bufSize)
+
+	errChan := make(chan error)
+	go func() {
+		err := LaunchTestGRPCServer(streamingServer, ctx, testListener)
+		if err != nil {
+			t.Error("Error launching listener", err)
+		}
+		errChan <- err
+	}()
+
+	time.Sleep(2 * time.Second)
+	return testListener, errChan
+}
+
+func testDialer(context.Context, string) (net.Conn, error) {
+	return testListener.Dial()
+}
+
+func GetStreamingServerClient(ctx context.Context, t *testing.T) (pb.ExternalProcessor_ProcessClient, *grpc.ClientConn) {
+	opts := []grpc.DialOption{
+		grpc.WithTransportCredentials(insecure.NewCredentials()),
+		grpc.WithContextDialer(testDialer),
+	}
+	conn, err := grpc.NewClient("passthrough://bufconn", opts...)
+	if err != nil {
+		t.Error(err)
+		return nil, nil
+	}
+
+	extProcClient := pb.NewExternalProcessorClient(conn)
+	process, err := extProcClient.Process(ctx)
+	if err != nil {
+		t.Error(err)
+		return nil, nil
+	}
+
+	return process, conn
+}
+
+// LaunchTestGRPCServer actually starts the server (enables testing)
+func LaunchTestGRPCServer(s pb.ExternalProcessorServer, ctx context.Context, listener net.Listener) error {
+	grpcServer := grpc.NewServer()
+
+	pb.RegisterExternalProcessorServer(grpcServer, s)
+
+	// Shutdown on context closed.
+	// Terminate the server on context closed.
+	go func() {
+		<-ctx.Done()
+		grpcServer.GracefulStop()
+	}()
+
+	if err := grpcServer.Serve(listener); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func CheckEnvoyGRPCHeaders(t *testing.T, response *pb.CommonResponse, expectedHeaders map[string]string) bool {
+	headers := response.HeaderMutation.SetHeaders
+	for expectedKey, expectedValue := range expectedHeaders {
+		found := false
+		for _, header := range headers {
+			if header.Header.Key == expectedKey {
+				if expectedValue != string(header.Header.RawValue) {
+					t.Errorf("Incorrect value for header %s, want %s got %s", expectedKey, expectedValue,
+						string(header.Header.RawValue))
+					return false
+				}
+				found = true
+				break
+			}
+		}
+		if !found {
+			t.Errorf("Missing header %s", expectedKey)
+			return false
+		}
+	}
+
+	for _, header := range headers {
+		expectedValue, ok := expectedHeaders[header.Header.Key]
+		if !ok {
+			t.Errorf("Unexpected header %s", header.Header.Key)
+			return false
+		} else if expectedValue != string(header.Header.RawValue) {
+			t.Errorf("Incorrect value for header %s, want %s got %s", header.Header.Key, expectedValue,
+				string(header.Header.RawValue))
+			return false
+		}
+	}
+	return true
+}
+
+func BuildEnvoyGRPCHeaders(headers map[string]string, rawValue bool) *pb.HttpHeaders {
+	headerValues := make([]*corev3.HeaderValue, 0)
+	for key, value := range headers {
+		header := &corev3.HeaderValue{Key: key}
+		if rawValue {
+			header.RawValue = []byte(value)
+		} else {
+			header.Value = value
+		}
+		headerValues = append(headerValues, header)
+	}
+	return &pb.HttpHeaders{
+		Headers: &corev3.HeaderMap{
+			Headers: headerValues,
+		},
+	}
+}
diff --git a/test/utils/utils.go b/test/utils/utils.go
index 1ec0fba..ba74069 100644
--- a/test/utils/utils.go
+++ b/test/utils/utils.go
@@ -59,6 +59,24 @@ func DeleteClusterResources(ctx context.Context, cli client.Client) error {
 	if err != nil && !apierrors.IsNotFound(err) {
 		return err
 	}
+	metricsReaderBinding := &rbacv1.ClusterRoleBinding{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "inference-gateway-sa-metrics-reader-role-binding",
+		},
+	}
+	err = cli.Delete(ctx, metricsReaderBinding, client.PropagationPolicy(metav1.DeletePropagationForeground))
+	if err != nil && !apierrors.IsNotFound(err) {
+		return err
+	}
+	metricsReaderRole := &rbacv1.ClusterRole{
+		ObjectMeta: metav1.ObjectMeta{
+			Name: "inference-gateway-metrics-reader",
+		},
+	}
+	err = cli.Delete(ctx, metricsReaderRole, client.PropagationPolicy(metav1.DeletePropagationForeground))
+	if err != nil && !apierrors.IsNotFound(err) {
+		return err
+	}
 	model := &apiextv1.CustomResourceDefinition{
 		ObjectMeta: metav1.ObjectMeta{
 			Name: "inferencemodels.inference.networking.x-k8s.io",
@@ -106,6 +124,10 @@ func DeleteNamespacedResources(ctx context.Context, cli client.Client, ns string
 	if err != nil && !apierrors.IsNotFound(err) {
 		return err
 	}
+	err = cli.DeleteAllOf(ctx, &corev1.ServiceAccount{}, client.InNamespace(ns), client.PropagationPolicy(metav1.DeletePropagationForeground))
+	if err != nil && !apierrors.IsNotFound(err) {
+		return err
+	}
 	err = cli.DeleteAllOf(ctx, &v1alpha2.InferencePool{}, client.InNamespace(ns), client.PropagationPolicy(metav1.DeletePropagationForeground))
 	if err != nil && !apierrors.IsNotFound(err) {
 		return err