Repository: kubernetes-sigs/gateway-api-inference-extension

Issue:
Conformance test become quite flaky with ` --cleanup-base-resources=false ` flag
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

**What happened**:
Conformance test with GKE become quite flaky with ` --cleanup-base-resources=false ` flag.
The successful ratio is much higher with ` --cleanup-base-resources=true ` flag.

**What you expected to happen**:
Conformance test should not be flaky with ` --cleanup-base-resources=false ` flag.

**How to reproduce it (as minimally and precisely as possible)**:
I set up the env following https://gateway-api-inference-extension.sigs.k8s.io/guides/ with GKE. 
And then ran conformance test via
```
go test -timeout 45m ./conformance -v -args -debug \
    --gateway-class=gke-l7-regional-external-managed \
    --allow-crds-mismatch=true \
    --cleanup-base-resources=false \
    --report-output=./test-report.yaml &> test_log.txt
```

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Inference extension version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration: GKE 
- Install tools:
- Others:


Hints from issue comments:
@zetxqx: /assign

Solution patch:
diff --git a/.github/ISSUE_TEMPLATE/new-release.md b/.github/ISSUE_TEMPLATE/new-release.md
index d88b7f8..98fc3f4 100644
--- a/.github/ISSUE_TEMPLATE/new-release.md
+++ b/.github/ISSUE_TEMPLATE/new-release.md
@@ -26,6 +26,7 @@ This document defines the process for releasing Gateway API Inference Extension.
    ```shell
    export MAJOR=0
    export MINOR=1
+   export PATCH=0
    export REMOTE=origin
    ```
 
@@ -63,7 +64,7 @@ This document defines the process for releasing Gateway API Inference Extension.
      git checkout -b release-${MAJOR}.${MINOR}
      ```
 
-   - For a Major or Minor Release:
+   - For a Major, Minor or Patch Release:
      A release branch should already exist. In this case, check out the existing branch:
 
      ```shell
@@ -81,13 +82,13 @@ This document defines the process for releasing Gateway API Inference Extension.
    For a release candidate:
 
     ```shell
-    git commit -s -m "Updates artifacts for v${MAJOR}.${MINOR}.0-rc.${RC} release"
+    git commit -s -m "Updates artifacts for v${MAJOR}.${MINOR}.${PATCH}-rc.${RC} release"
     ```
 
-   For a major or minor release:
+   For a major, minor or patch release:
 
     ```shell
-    git commit -s -m "Updates artifacts for v${MAJOR}.${MINOR}.0 release"
+    git commit -s -m "Updates artifacts for v${MAJOR}.${MINOR}.${PATCH} release"
     ```
 
 6. Push your release branch to the Gateway API Inference Extension remote.
@@ -101,13 +102,13 @@ This document defines the process for releasing Gateway API Inference Extension.
    For a release candidate:
 
     ```shell
-    git tag -s -a v${MAJOR}.${MINOR}.0-rc.${RC} -m 'Gateway API Inference Extension v${MAJOR}.${MINOR}.0-rc.${RC} Release Candidate'
+    git tag -s -a v${MAJOR}.${MINOR}.${PATCH}-rc.${RC} -m 'Gateway API Inference Extension v${MAJOR}.${MINOR}.${PATCH}-rc.${RC} Release Candidate'
     ```
 
-   For a major or minor release:
+   For a major, minor or patch release:
 
     ```shell
-    git tag -s -a v${MAJOR}.${MINOR}.0 -m 'Gateway API Inference Extension v${MAJOR}.${MINOR}.0 Release'
+    git tag -s -a v${MAJOR}.${MINOR}.${PATCH} -m 'Gateway API Inference Extension v${MAJOR}.${MINOR}.${PATCH} Release'
     ```
 
    **Note:** A PGP key must be [registered] to your GitHub account.
@@ -117,18 +118,18 @@ This document defines the process for releasing Gateway API Inference Extension.
    For a release candidate:
 
     ```shell
-    git push ${REMOTE} v${MAJOR}.${MINOR}.0-rc.${RC}
+    git push ${REMOTE} v${MAJOR}.${MINOR}.${PATCH}-rc.${RC}
     ```
 
-   For a major or minor release:
+   For a major, minor or patch release:
 
     ```shell
-    git push ${REMOTE} v${MAJOR}.${MINOR}.0
+    git push ${REMOTE} v${MAJOR}.${MINOR}.${PATCH}
     ```
 
 9. Pushing the tag triggers Prow to build and publish the container image to the [staging registry][].
 10. Submit a PR against [k8s.io][] to add the staging image tag and SHA to [`k8s-staging-gateway-api-inference-extension/images.yaml`][yaml]. This will
-    promote the image to the production registry, e.g. `registry.k8s.io/gateway-api-inference-extension/epp:v${MAJOR}.${MINOR}.0`.
+    promote the image to the production registry, e.g. `registry.k8s.io/gateway-api-inference-extension/epp:v${MAJOR}.${MINOR}.${PATCH}`.
     **Note:** Add a link to this issue when the PR is merged.
 11. Test the steps in the tagged quickstart guide after the PR merges, for example: `https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/v0.1.0-rc.1/pkg/README.md`.
 12. Create a [new release][]:
@@ -146,7 +147,7 @@ Use the following steps to announce the release.
 1. Send an announcement email to `wg-serving@kubernetes.io` and `dev@kubernetes.io` with the subject:
 
    ```shell
-   [ANNOUNCE] Gateway API Inference Extension v${MAJOR}.${MINOR}.0 is released
+   [ANNOUNCE] Gateway API Inference Extension v${MAJOR}.${MINOR}.${PATCH} is released
    ```
 
 2. Add a link to the release announcement in this issue. <!-- link to an example email once we have one -->
diff --git a/Makefile b/Makefile
index a2563a6..ba66ef4 100644
--- a/Makefile
+++ b/Makefile
@@ -24,7 +24,6 @@ SHELL = /usr/bin/env bash -o pipefail
 GIT_COMMIT_SHA ?= "$(shell git rev-parse HEAD 2>/dev/null)"
 GIT_TAG ?= $(shell git describe --tags --dirty --always)
 PLATFORMS ?= linux/amd64
-PUBLISH_PLATFORMS ?= linux/amd64,linux/arm64
 DOCKER_BUILDX_CMD ?= docker buildx
 IMAGE_BUILD_CMD ?= $(DOCKER_BUILDX_CMD) build
 IMAGE_BUILD_EXTRA_OPTS ?=
@@ -74,11 +73,6 @@ SYNCER_IMAGE_BUILD_EXTRA_OPTS += -t $(SYNCER_IMAGE_EXTRA_TAG)
 BBR_IMAGE_BUILD_EXTRA_OPTS += -t $(BBR_IMAGE_EXTRA_TAG)
 endif
 
-# Allow `make MULTI=true ...` in CI to switch to multi-arch.
-ifdef MULTI
-  PLATFORMS := $(PUBLISH_PLATFORMS)
-endif
-
 
 # The name of the kind cluster to use for the "kind-load" target.
 KIND_CLUSTER ?= kind
@@ -211,7 +205,7 @@ image-build: ## Build the EPP image using Docker Buildx.
 
 .PHONY: image-push
 image-push: PUSH=--push ## Build the EPP image and push it to $IMAGE_REPO.
-image-push: MULTI=true image-build
+image-push: image-build
 
 .PHONY: image-load
 image-load: LOAD=--load ## Build the EPP image and load it in the local Docker registry.
diff --git a/cmd/epp/runner/runner.go b/cmd/epp/runner/runner.go
index c9c5f60..7d92c78 100644
--- a/cmd/epp/runner/runner.go
+++ b/cmd/epp/runner/runner.go
@@ -78,15 +78,6 @@ var (
 		"enable-pprof",
 		runserver.DefaultEnablePprof,
 		"Enables pprof handlers. Defaults to true. Set to false to disable pprof handlers.")
-	destinationEndpointHintKey = flag.String(
-		"destination-endpoint-hint-key",
-		runserver.DefaultDestinationEndpointHintKey,
-		"Header and response metadata key used by Envoy to route to the appropriate pod. This must match Envoy configuration.")
-	destinationEndpointHintMetadataNamespace = flag.String(
-		"destination-endpoint-hint-metadata-namespace",
-		runserver.DefaultDestinationEndpointHintMetadataNamespace,
-		"The key for the outer namespace struct in the metadata field of the extproc response that is used to wrap the"+
-			"target endpoint. If not set, then an outer namespace struct should not be created.")
 	poolName = flag.String(
 		"pool-name",
 		runserver.DefaultPoolName,
@@ -113,6 +104,20 @@ var (
 		"The path to the certificate for secure serving. The certificate and private key files "+
 			"are assumed to be named tls.crt and tls.key, respectively. If not set, and secureServing is enabled, "+
 			"then a self-signed certificate is used.")
+	// header/metadata flags
+	destinationEndpointHintKey = flag.String(
+		"destination-endpoint-hint-key",
+		runserver.DefaultDestinationEndpointHintKey,
+		"Header and response metadata key used by Envoy to route to the appropriate pod. This must match Envoy configuration.")
+	destinationEndpointHintMetadataNamespace = flag.String(
+		"destination-endpoint-hint-metadata-namespace",
+		runserver.DefaultDestinationEndpointHintMetadataNamespace,
+		"The key for the outer namespace struct in the metadata field of the extproc response that is used to wrap the"+
+			"target endpoint. If not set, then an outer namespace struct should not be created.")
+	fairnessIDHeaderKey = flag.String(
+		"fairness-id-header-key",
+		runserver.DefaultFairnessIDHeaderKey,
+		"The header key used to pass the fairness ID to be used in Flow Control.")
 	// metric flags
 	totalQueuedRequestsMetric = flag.String(
 		"total-queued-requests-metric",
@@ -337,6 +342,7 @@ func (r *Runner) Run(ctx context.Context) error {
 		GrpcPort:                                 *grpcPort,
 		DestinationEndpointHintMetadataNamespace: *destinationEndpointHintMetadataNamespace,
 		DestinationEndpointHintKey:               *destinationEndpointHintKey,
+		FairnessIDHeaderKey:                      *fairnessIDHeaderKey,
 		PoolNamespacedName:                       poolNamespacedName,
 		Datastore:                                datastore,
 		SecureServing:                            *secureServing,
@@ -387,6 +393,7 @@ func (r *Runner) registerInTreePlugins() {
 	plugins.Register(profile.SingleProfileHandlerType, profile.SingleProfileHandlerFactory)
 	plugins.Register(scorer.KvCacheUtilizationScorerType, scorer.KvCacheUtilizationScorerFactory)
 	plugins.Register(scorer.QueueScorerType, scorer.QueueScorerFactory)
+	plugins.Register(scorer.LoraAffinityScorerType, scorer.LoraAffinityScorerFactory)
 	// register filter for test purpose only (used in conformance tests)
 	plugins.Register(testfilter.HeaderBasedTestingFilterType, testfilter.HeaderBasedTestingFilterFactory)
 }
diff --git a/config/charts/inferencepool/templates/epp-deployment.yaml b/config/charts/inferencepool/templates/epp-deployment.yaml
index 5ab9e2d..23957a3 100644
--- a/config/charts/inferencepool/templates/epp-deployment.yaml
+++ b/config/charts/inferencepool/templates/epp-deployment.yaml
@@ -33,6 +33,8 @@ spec:
         - "9002"
         - --grpc-health-port
         - "9003"
+        - --zap-encoder
+        - "json"
         - --metrics-port
         - "9090"
         - --config-file
diff --git a/config/charts/inferencepool/templates/rbac.yaml b/config/charts/inferencepool/templates/rbac.yaml
index a8d4d0e..a8d891c 100644
--- a/config/charts/inferencepool/templates/rbac.yaml
+++ b/config/charts/inferencepool/templates/rbac.yaml
@@ -42,6 +42,9 @@ rules:
 - apiGroups: ["inference.networking.x-k8s.io"]
   resources: ["inferenceobjectives", "inferencepools"]
   verbs: ["get", "watch", "list"]
+- apiGroups: ["inference.networking.k8s.io"]
+  resources: ["inferencepools"]
+  verbs: ["get", "watch", "list"]
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["get", "watch", "list"]
diff --git a/conformance/conformance.go b/conformance/conformance.go
index eb18e8f..6aebd69 100644
--- a/conformance/conformance.go
+++ b/conformance/conformance.go
@@ -30,7 +30,6 @@ import (
 	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
@@ -39,7 +38,6 @@ import (
 	clientsetscheme "k8s.io/client-go/kubernetes/scheme"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	k8sconfig "sigs.k8s.io/controller-runtime/pkg/client/config"
-
 	gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
 	confflags "sigs.k8s.io/gateway-api/conformance/utils/flags"
 	apikubernetes "sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
@@ -49,7 +47,6 @@ import (
 
 	inferencev1 "sigs.k8s.io/gateway-api-inference-extension/api/v1"
 	inferencev1alpha2 "sigs.k8s.io/gateway-api-inference-extension/apix/v1alpha2"
-
 	"sigs.k8s.io/gateway-api-inference-extension/conformance/resources"
 	"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
 	inferenceconfig "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/config"
diff --git a/conformance/resources/resourcename.go b/conformance/resources/resourcename.go
index 8ade3fb..4c0c7b4 100644
--- a/conformance/resources/resourcename.go
+++ b/conformance/resources/resourcename.go
@@ -32,7 +32,8 @@ const (
 	PrimaryModelServerDeploymentName   = "primary-inference-model-server-deployment"
 	SecondaryModelServerDeploymentName = "secondary-inference-model-server-deployment"
 
-	ModelServerPodReplicas = 3
+	ModelServerPodReplicas    = 3
+	EndPointPickerPodReplicas = 1
 )
 
 var (
@@ -44,4 +45,7 @@ var (
 
 	PrimaryEppDeploymentNN   = types.NamespacedName{Name: "primary-app-endpoint-picker", Namespace: AppBackendNamespace}
 	SecondaryEppDeploymentNN = types.NamespacedName{Name: "secondary-app-endpoint-picker", Namespace: AppBackendNamespace}
+
+	PrimaryEppServiceNN   = types.NamespacedName{Name: "primary-endpoint-picker-svc", Namespace: AppBackendNamespace}
+	SecondaryEppServiceNN = types.NamespacedName{Name: "secondary-endpoint-picker-svc", Namespace: AppBackendNamespace}
 )
diff --git a/conformance/utils/config/timing.go b/conformance/utils/config/timing.go
index 03cf078..6ec6c54 100644
--- a/conformance/utils/config/timing.go
+++ b/conformance/utils/config/timing.go
@@ -39,6 +39,9 @@ type InferenceExtensionTimeoutConfig struct {
 
 	// HTTPRouteConditionTimeout represents the maximum time to wait for an HTTPRoute to have a specific condition.
 	HTTPRouteDeletionReconciliationTimeout time.Duration
+
+	// ServiceUpdateTimeout represents the maximum time to wait for a service to be updated.
+	ServiceUpdateTimeout time.Duration
 }
 
 // DefaultInferenceExtensionTimeoutConfig returns a new InferenceExtensionTimeoutConfig with default values.
@@ -54,5 +57,6 @@ func DefaultInferenceExtensionTimeoutConfig() InferenceExtensionTimeoutConfig {
 		InferencePoolMustHaveConditionInterval: 10 * time.Second,
 		GatewayObjectPollInterval:              5 * time.Second,
 		HTTPRouteDeletionReconciliationTimeout: 5 * time.Second,
+		ServiceUpdateTimeout:                   10 * time.Second,
 	}
 }
diff --git a/conformance/utils/kubernetes/helpers.go b/conformance/utils/kubernetes/helpers.go
index 219b423..54d0260 100644
--- a/conformance/utils/kubernetes/helpers.go
+++ b/conformance/utils/kubernetes/helpers.go
@@ -26,8 +26,8 @@ import (
 	"time"
 
 	"github.com/stretchr/testify/require"
-	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
+	discoveryv1 "k8s.io/api/discovery/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
@@ -38,6 +38,7 @@ import (
 	gatewayk8sutils "sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
 
 	inferenceapi "sigs.k8s.io/gateway-api-inference-extension/api/v1"
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/resources"
 	"sigs.k8s.io/gateway-api-inference-extension/conformance/utils/config"
 )
 
@@ -328,42 +329,74 @@ func GetPodsWithLabel(t *testing.T, c client.Reader, namespace string, labels ma
 	return pods.Items, waitErr
 }
 
-// DeleteDeployment deletes the specified Deployment and waits until it is no longer
-// present in the cluster.
-func DeleteDeployment(t *testing.T, c client.Client, timeoutConfig gatewayapiconfig.TimeoutConfig, deploymentRef types.NamespacedName) error {
+// MakeServiceUnavailable modifies a Service's selector to make it temporarily unavailable.
+// It returns a cleanup function to restore the original selector and ensure the test state is clean.
+func MakeServiceUnavailable(t *testing.T, c client.Client, serviceRef types.NamespacedName, timeout time.Duration) (func(), error) {
 	t.Helper()
 
-	deploymentToDelete := &appsv1.Deployment{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      deploymentRef.Name,
-			Namespace: deploymentRef.Namespace,
-		},
+	ctx := context.Background()
+	svc := &corev1.Service{}
+
+	t.Logf("Making Service %s/%s unavailable by modifying its selector...", serviceRef.Namespace, serviceRef.Name)
+	if err := c.Get(ctx, serviceRef, svc); err != nil {
+		return nil, fmt.Errorf("failed to get Service %s/%s: %w", serviceRef.Namespace, serviceRef.Name, err)
+	}
+	originalSelector := svc.Spec.Selector
+	svc.Spec.Selector = map[string]string{"app": "do-not-match-for-testing"}
+	if err := c.Update(ctx, svc); err != nil {
+		return nil, fmt.Errorf("failed to update selector for Service %s/%s: %w", serviceRef.Namespace, serviceRef.Name, err)
 	}
 
-	t.Logf("Deleting Deployment %s/%s...", deploymentRef.Namespace, deploymentRef.Name)
-	if err := c.Delete(context.Background(), deploymentToDelete); err != nil {
-		// If the resource is already gone, we don't consider it an error.
-		if !apierrors.IsNotFound(err) {
-			return fmt.Errorf("failed to delete Deployment %s/%s: %w", deploymentRef.Namespace, deploymentRef.Name, err)
-		}
+	t.Logf("Waiting for EndpointSlices of Service %s/%s to become empty...", serviceRef.Namespace, serviceRef.Name)
+	err := waitNumberOfEndpointsForService(ctx, c, serviceRef, 0, timeout)
+	if err != nil {
+		return nil, fmt.Errorf("timed out waiting for EndpointSlices of Service %s/%s to become empty: %w", serviceRef.Namespace, serviceRef.Name, err)
 	}
+	t.Logf("Successfully modified selector for Service %s/%s", serviceRef.Namespace, serviceRef.Name)
 
-	// Wait for the Deployment to be fully removed.
-	waitErr := wait.PollUntilContextTimeout(context.Background(), 1*time.Second, timeoutConfig.DeleteTimeout, true, func(ctx context.Context) (bool, error) {
-		var dep appsv1.Deployment
-		err := c.Get(ctx, deploymentRef, &dep)
-		if apierrors.IsNotFound(err) {
-			return true, nil
+	cleanupFunc := func() {
+		t.Helper()
+		t.Logf("Restoring original selector for Service %s/%s...", serviceRef.Namespace, serviceRef.Name)
+
+		restorationCtx := context.Background()
+		svcToRestore := &corev1.Service{}
+
+		if err := c.Get(restorationCtx, serviceRef, svcToRestore); err != nil {
+			t.Fatalf("Cleanup failed: could not get Service %s/%s for restoration: %v", serviceRef.Namespace, serviceRef.Name, err)
 		}
+
+		svcToRestore.Spec.Selector = originalSelector
+		if err := c.Update(restorationCtx, svcToRestore); err != nil {
+			t.Fatalf("Cleanup failed: could not restore original selector for Service %s/%s: %v", serviceRef.Namespace, serviceRef.Name, err)
+		}
+
+		t.Logf("Waiting for EndpointSlices of Service %s/%s to be restored...", serviceRef.Namespace, serviceRef.Name)
+		err := waitNumberOfEndpointsForService(restorationCtx, c, serviceRef, resources.EndPointPickerPodReplicas, timeout)
 		if err != nil {
-			return false, fmt.Errorf("error waiting for Deployment %s/%s to be deleted: %w", deploymentRef.Namespace, deploymentRef.Name, err)
+			t.Fatalf("Cleanup failed: timed out waiting for EndpointSlices of Service %s/%s to be restored: %v", serviceRef.Namespace, serviceRef.Name, err)
+		}
+		t.Logf("Successfully restored selector for Service %s/%s", serviceRef.Namespace, serviceRef.Name)
+	}
+	return cleanupFunc, nil
+}
+
+func waitNumberOfEndpointsForService(ctx context.Context, c client.Client, serviceRef types.NamespacedName, wantNum int, timeout time.Duration) error {
+	err := wait.PollUntilContextTimeout(ctx, 1*time.Second, timeout, true, func(ctx context.Context) (bool, error) {
+		endpointSliceList := &discoveryv1.EndpointSliceList{}
+		if err := c.List(ctx, endpointSliceList,
+			client.InNamespace(serviceRef.Namespace),
+			client.MatchingLabels{discoveryv1.LabelServiceName: serviceRef.Name},
+		); err != nil {
+			return false, fmt.Errorf("failed to list EndpointSlices for Service %s: %w", serviceRef.Name, err)
+		}
+		totalEndpoints := 0
+		for _, slice := range endpointSliceList.Items {
+			totalEndpoints += len(slice.Endpoints)
+		}
+		if totalEndpoints == wantNum {
+			return true, nil
 		}
 		return false, nil
 	})
-
-	if waitErr != nil {
-		return fmt.Errorf("timed out waiting for Deployment %s/%s to be deleted: %w", deploymentRef.Namespace, deploymentRef.Name, waitErr)
-	}
-	t.Logf("Successfully deleted Deployment %s/%s", deploymentRef.Namespace, deploymentRef.Name)
-	return nil
+	return err
 }
diff --git a/go.mod b/go.mod
index a7d6848..56fce1f 100644
--- a/go.mod
+++ b/go.mod
@@ -12,7 +12,7 @@ require (
 	github.com/hashicorp/golang-lru/v2 v2.0.7
 	github.com/onsi/ginkgo/v2 v2.23.4
 	github.com/onsi/gomega v1.38.0
-	github.com/prometheus/client_golang v1.22.0
+	github.com/prometheus/client_golang v1.23.0
 	github.com/prometheus/client_model v0.6.2
 	github.com/prometheus/common v0.65.0
 	github.com/stretchr/testify v1.10.0
@@ -84,7 +84,7 @@ require (
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
-	github.com/prometheus/procfs v0.15.1 // indirect
+	github.com/prometheus/procfs v0.16.1 // indirect
 	github.com/spf13/cobra v1.9.1 // indirect
 	github.com/spf13/pflag v1.0.6 // indirect
 	github.com/stoewer/go-strcase v1.3.0 // indirect
diff --git a/go.sum b/go.sum
index 58fb085..9a0d8a4 100644
--- a/go.sum
+++ b/go.sum
@@ -142,14 +142,14 @@ github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRI
 github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g=
 github.com/prashantv/gostub v1.1.0/go.mod h1:A5zLQHz7ieHGG7is6LLXLz7I8+3LZzsrV0P1IAHhP5U=
-github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
-github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
+github.com/prometheus/client_golang v1.23.0 h1:ust4zpdl9r4trLY/gSjlm07PuiBq2ynaXXlptpfy8Uc=
+github.com/prometheus/client_golang v1.23.0/go.mod h1:i/o0R9ByOnHX0McrTMTyhYvKE4haaf2mW08I+jGAjEE=
 github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
 github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
 github.com/prometheus/common v0.65.0 h1:QDwzd+G1twt//Kwj/Ww6E9FQq1iVMmODnILtW1t2VzE=
 github.com/prometheus/common v0.65.0/go.mod h1:0gZns+BLRQ3V6NdaerOhMbwwRbNh9hkGINtQAsP5GS8=
-github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
-github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
+github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
+github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
 github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
 github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
diff --git a/pkg/epp/flowcontrol/controller/internal/processor.go b/pkg/epp/flowcontrol/controller/internal/processor.go
index 0154dc7..cef5b35 100644
--- a/pkg/epp/flowcontrol/controller/internal/processor.go
+++ b/pkg/epp/flowcontrol/controller/internal/processor.go
@@ -26,8 +26,8 @@ import (
 	"time"
 
 	"github.com/go-logr/logr"
-
 	"sigs.k8s.io/controller-runtime/pkg/log"
+
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/contracts"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/types"
diff --git a/pkg/epp/handlers/request.go b/pkg/epp/handlers/request.go
index 5789b3c..b528ed7 100644
--- a/pkg/epp/handlers/request.go
+++ b/pkg/epp/handlers/request.go
@@ -57,6 +57,13 @@ func (s *StreamingServer) HandleRequestHeaders(ctx context.Context, reqCtx *Requ
 		} else {
 			reqCtx.Request.Headers[header.Key] = header.Value
 		}
+		if header.Key == s.fairnessIDHeaderKey {
+			reqCtx.FairnessID = reqCtx.Request.Headers[header.Key]
+			// remove the fairness ID header from the request headers,
+			// this is not data that should be manipulated or sent to the backend.
+			// It is only used for flow control.
+			delete(reqCtx.Request.Headers, header.Key)
+		}
 	}
 	return nil
 }
diff --git a/pkg/epp/handlers/server.go b/pkg/epp/handlers/server.go
index 3059660..62ffc8c 100644
--- a/pkg/epp/handlers/server.go
+++ b/pkg/epp/handlers/server.go
@@ -44,10 +44,11 @@ const (
 	bodyByteLimit = 62000
 )
 
-func NewStreamingServer(destinationEndpointHintMetadataNamespace, destinationEndpointHintKey string, datastore Datastore, director Director) *StreamingServer {
+func NewStreamingServer(destinationEndpointHintMetadataNamespace, destinationEndpointHintKey, fairnessIDHeaderKey string, datastore Datastore, director Director) *StreamingServer {
 	return &StreamingServer{
 		destinationEndpointHintMetadataNamespace: destinationEndpointHintMetadataNamespace,
 		destinationEndpointHintKey:               destinationEndpointHintKey,
+		fairnessIDHeaderKey:                      fairnessIDHeaderKey,
 		director:                                 director,
 		datastore:                                datastore,
 	}
@@ -72,6 +73,7 @@ type StreamingServer struct {
 	// The key acting as the outer namespace struct in the metadata extproc response to communicate
 	// back the picked endpoints.
 	destinationEndpointHintMetadataNamespace string
+	fairnessIDHeaderKey                      string
 	datastore                                Datastore
 	director                                 Director
 }
@@ -85,6 +87,7 @@ type RequestContext struct {
 	TargetEndpoint            string
 	Model                     string
 	ResolvedTargetModel       string
+	FairnessID                string
 	RequestReceivedTimestamp  time.Time
 	ResponseCompleteTimestamp time.Time
 	RequestSize               int
@@ -192,8 +195,8 @@ func (s *StreamingServer) Process(srv extProcPb.ExternalProcessor_ProcessServer)
 
 		switch v := req.Request.(type) {
 		case *extProcPb.ProcessingRequest_RequestHeaders:
-			if requestId := requtil.ExtractHeaderValue(v, requtil.RequestIdHeaderKey); len(requestId) > 0 {
-				logger = logger.WithValues(requtil.RequestIdHeaderKey, requestId)
+			if requestID := requtil.ExtractHeaderValue(v, requtil.RequestIdHeaderKey); len(requestID) > 0 {
+				logger = logger.WithValues(requtil.RequestIdHeaderKey, requestID)
 				loggerTrace = logger.V(logutil.TRACE)
 				ctx = log.IntoContext(ctx, logger)
 			}
diff --git a/pkg/epp/requestcontrol/director.go b/pkg/epp/requestcontrol/director.go
index 65232be..d2073eb 100644
--- a/pkg/epp/requestcontrol/director.go
+++ b/pkg/epp/requestcontrol/director.go
@@ -139,7 +139,7 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 	logger.V(logutil.DEBUG).Info("LLM request assembled")
 
 	// --- 2. Admission Control check --
-	if err := d.admitRequest(ctx, requestCriticality); err != nil {
+	if err := d.admitRequest(ctx, requestCriticality, reqCtx.FairnessID); err != nil {
 		return reqCtx, err
 	}
 
@@ -166,9 +166,11 @@ func (d *Director) HandleRequest(ctx context.Context, reqCtx *handlers.RequestCo
 
 // admitRequest handles admission control to decide whether or not to accept the request
 // based on the request criticality and system saturation state.
-func (d *Director) admitRequest(ctx context.Context, requestCriticality v1alpha2.Criticality) error {
+func (d *Director) admitRequest(ctx context.Context, requestCriticality v1alpha2.Criticality, fairnessID string) error {
 	logger := log.FromContext(ctx)
 
+	logger.V(logutil.TRACE).Info("Entering Flow Control", "criticality", requestCriticality, "fairnessID", fairnessID)
+
 	if requestCriticality == v1alpha2.Critical {
 		logger.V(logutil.DEBUG).Info("Critical request bypassing saturation check.")
 		return nil
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
index 068f580..381ec97 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
@@ -55,6 +55,12 @@ const (
 	PrefixCachePluginType = "prefix-cache-scorer"
 )
 
+var DefaultConfig = Config{
+	HashBlockSize:          DefaultHashBlockSize,
+	MaxPrefixBlocksToMatch: DefaultMaxPrefixBlocks,
+	LRUCapacityPerServer:   DefaultLRUCapacityPerServer,
+}
+
 type Config struct {
 	// The input prompt is broken into sizes of HashBlockSize to calculate block hashes . Requests
 	// with length shorter than the block size will be ignored.
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go b/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go
new file mode 100644
index 0000000..d995bae
--- /dev/null
+++ b/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity.go
@@ -0,0 +1,89 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"context"
+	"encoding/json"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+const (
+	LoraAffinityScorerType = "lora-affinity-scorer"
+)
+
+// compile-time type assertion
+var _ framework.Scorer = &LoraAffinityScorer{}
+
+// LoraAffinityScorerFactory defines the factory function for LoraAffinityScorer.
+func LoraAffinityScorerFactory(name string, _ json.RawMessage, _ plugins.Handle) (plugins.Plugin, error) {
+	return NewLoraAffinityScorer().WithName(name), nil
+}
+
+// NewLoraAffinityScorer initializes a new LoraAffinityScorer and returns its pointer.
+func NewLoraAffinityScorer() *LoraAffinityScorer {
+	return &LoraAffinityScorer{
+		tn: plugins.TypedName{Type: LoraAffinityScorerType, Name: LoraAffinityScorerType},
+	}
+}
+
+// LoraAffinityScorer scores list of candidate pods based on Lora affinity and availability.
+type LoraAffinityScorer struct {
+	tn plugins.TypedName
+}
+
+// TypedName returns the type and name tuple of this plugin instance.
+func (s *LoraAffinityScorer) TypedName() plugins.TypedName {
+	return s.tn
+}
+
+// WithName sets the name of the scorer.
+func (s *LoraAffinityScorer) WithName(name string) *LoraAffinityScorer {
+	s.tn.Name = name
+	return s
+}
+
+func (s *LoraAffinityScorer) Score(_ context.Context, _ *types.CycleState, request *types.LLMRequest, pods []types.Pod) map[types.Pod]float64 {
+	scores := make(map[types.Pod]float64, len(pods))
+
+	// Assign a score to each pod for loading the target adapter.
+	for _, pod := range pods {
+		_, active := pod.GetMetrics().ActiveModels[request.TargetModel]
+		_, waiting := pod.GetMetrics().WaitingModels[request.TargetModel]
+
+		// Determine the model server's suitability score based on adapter load status and capacity.
+		switch {
+		// Ideal: The adapter is already active on this model server.
+		case active:
+			scores[pod] = 1.0
+		// Good: The model server has capacity to load at least one more adapter.
+		case len(pod.GetMetrics().ActiveModels)+len(pod.GetMetrics().WaitingModels) < pod.GetMetrics().MaxActiveModels:
+			scores[pod] = 0.8
+		// Moderate: The adapter is already in the queue to be loaded on this model server.
+		case waiting:
+			scores[pod] = 0.6
+		// Unsuitable: The model server has reached its maximum capacity and cannot load the adapter.
+		default:
+			scores[pod] = 0.0
+		}
+	}
+
+	return scores
+}
diff --git a/pkg/epp/scheduling/scheduler.go b/pkg/epp/scheduling/scheduler.go
index ef0c522..12c1883 100644
--- a/pkg/epp/scheduling/scheduler.go
+++ b/pkg/epp/scheduling/scheduler.go
@@ -23,6 +23,7 @@ import (
 	"time"
 
 	"sigs.k8s.io/controller-runtime/pkg/log"
+
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/metrics"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
diff --git a/pkg/epp/server/runserver.go b/pkg/epp/server/runserver.go
index 5005323..2cbcbec 100644
--- a/pkg/epp/server/runserver.go
+++ b/pkg/epp/server/runserver.go
@@ -46,6 +46,7 @@ type ExtProcServerRunner struct {
 	GrpcPort                                 int
 	DestinationEndpointHintMetadataNamespace string
 	DestinationEndpointHintKey               string
+	FairnessIDHeaderKey                      string
 	PoolNamespacedName                       types.NamespacedName
 	Datastore                                datastore.Datastore
 	SecureServing                            bool
@@ -63,24 +64,25 @@ type ExtProcServerRunner struct {
 
 // Default values for CLI flags in main
 const (
-	DefaultGrpcPort                                 = 9002                             // default for --grpc-port
-	DefaultGrpcHealthPort                           = 9003                             // default for --grpc-health-port
-	DefaultMetricsPort                              = 9090                             // default for --metrics-port
-	DefaultDestinationEndpointHintMetadataNamespace = "envoy.lb"                       // default for --destinationEndpointHintMetadataNamespace
-	DefaultDestinationEndpointHintKey               = "x-gateway-destination-endpoint" // default for --destination-endpoint-hint-key
-	DefaultPoolName                                 = ""                               // required but no default
-	DefaultPoolNamespace                            = "default"                        // default for --pool-namespace
-	DefaultRefreshMetricsInterval                   = 50 * time.Millisecond            // default for --refresh-metrics-interval
-	DefaultRefreshPrometheusMetricsInterval         = 5 * time.Second                  // default for --refresh-prometheus-metrics-interval
-	DefaultSecureServing                            = true                             // default for --secure-serving
-	DefaultHealthChecking                           = false                            // default for --health-checking
-	DefaultEnablePprof                              = true                             // default for --enable-pprof
-	DefaultTotalQueuedRequestsMetric                = "vllm:num_requests_waiting"      // default for --total-queued-requests-metric
-	DefaultKvCacheUsagePercentageMetric             = "vllm:gpu_cache_usage_perc"      // default for --kv-cache-usage-percentage-metric
-	DefaultLoraInfoMetric                           = "vllm:lora_requests_info"        // default for --lora-info-metric
-	DefaultCertPath                                 = ""                               // default for --cert-path
-	DefaultConfigFile                               = ""                               // default for --config-file
-	DefaultConfigText                               = ""                               // default for --config-text
+	DefaultGrpcPort                                 = 9002                              // default for --grpc-port
+	DefaultGrpcHealthPort                           = 9003                              // default for --grpc-health-port
+	DefaultMetricsPort                              = 9090                              // default for --metrics-port
+	DefaultDestinationEndpointHintMetadataNamespace = "envoy.lb"                        // default for --destinationEndpointHintMetadataNamespace
+	DefaultDestinationEndpointHintKey               = "x-gateway-destination-endpoint"  // default for --destination-endpoint-hint-key
+	DefaultFairnessIDHeaderKey                      = "x-gateway-inference-fairness-id" // default for --fairness-id-header-key
+	DefaultPoolName                                 = ""                                // required but no default
+	DefaultPoolNamespace                            = "default"                         // default for --pool-namespace
+	DefaultRefreshMetricsInterval                   = 50 * time.Millisecond             // default for --refresh-metrics-interval
+	DefaultRefreshPrometheusMetricsInterval         = 5 * time.Second                   // default for --refresh-prometheus-metrics-interval
+	DefaultSecureServing                            = true                              // default for --secure-serving
+	DefaultHealthChecking                           = false                             // default for --health-checking
+	DefaultEnablePprof                              = true                              // default for --enable-pprof
+	DefaultTotalQueuedRequestsMetric                = "vllm:num_requests_waiting"       // default for --total-queued-requests-metric
+	DefaultKvCacheUsagePercentageMetric             = "vllm:gpu_cache_usage_perc"       // default for --kv-cache-usage-percentage-metric
+	DefaultLoraInfoMetric                           = "vllm:lora_requests_info"         // default for --lora-info-metric
+	DefaultCertPath                                 = ""                                // default for --cert-path
+	DefaultConfigFile                               = ""                                // default for --config-file
+	DefaultConfigText                               = ""                                // default for --config-text
 	DefaultMetricsStalenessThreshold                = 2 * time.Second
 )
 
@@ -91,6 +93,7 @@ func NewDefaultExtProcServerRunner() *ExtProcServerRunner {
 		GrpcPort:                                 DefaultGrpcPort,
 		DestinationEndpointHintKey:               DefaultDestinationEndpointHintKey,
 		DestinationEndpointHintMetadataNamespace: DefaultDestinationEndpointHintMetadataNamespace,
+		FairnessIDHeaderKey:                      DefaultFairnessIDHeaderKey,
 		PoolNamespacedName:                       types.NamespacedName{Name: DefaultPoolName, Namespace: DefaultPoolNamespace},
 		SecureServing:                            DefaultSecureServing,
 		HealthChecking:                           DefaultHealthChecking,
@@ -159,6 +162,7 @@ func (r *ExtProcServerRunner) AsRunnable(logger logr.Logger) manager.Runnable {
 		extProcServer := handlers.NewStreamingServer(
 			r.DestinationEndpointHintMetadataNamespace,
 			r.DestinationEndpointHintKey,
+			r.FairnessIDHeaderKey,
 			r.Datastore,
 			r.Director,
 		)
diff --git a/site-src/guides/epp-configuration/config-text.md b/site-src/guides/epp-configuration/config-text.md
index ceb0ed6..7c33aae 100644
--- a/site-src/guides/epp-configuration/config-text.md
+++ b/site-src/guides/epp-configuration/config-text.md
@@ -261,3 +261,13 @@ available to serve new request).
 
 - *Type*: queue-scorer
 - *Parameters*: none
+
+
+#### **LoraAffinityScorer**
+
+Scores list of candidate pods based on the LoRA adapters loaded on the pod. 
+Pods with the adapter already loaded or able to be actively loaded will be 
+scored higher (since it's more available to serve new request).
+
+- *Type*: lora-affinity-scorer
+- *Parameters*: none
diff --git a/site-src/guides/index.md b/site-src/guides/index.md
index 884e497..a1b10ed 100644
--- a/site-src/guides/index.md
+++ b/site-src/guides/index.md
@@ -88,7 +88,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
    Deploy the sample InferenceModel which is configured to forward traffic to the `food-review-1` [LoRA adapter](https://docs.vllm.ai/en/latest/features/lora.html) of the sample model server.
 
    ```bash
-   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencemodel.yaml
+   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferenceobjective.yaml
    ```
 
 ### Deploy the InferencePool and Endpoint Picker Extension
@@ -269,7 +269,7 @@ This quickstart guide is intended for engineers familiar with k8s and model serv
 
       ```bash
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencepool-resources.yaml --ignore-not-found
-      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferencemodel.yaml --ignore-not-found
+      kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/inferenceobjective.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/cpu-deployment.yaml --ignore-not-found
       kubectl delete -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/gpu-deployment.yaml --ignore-not-found
       kubectl delete secret hf-token --ignore-not-found
diff --git a/site-src/performance/regression-testing/index.md b/site-src/performance/regression-testing/index.md
index 83e7a3b..e4974c1 100644
--- a/site-src/performance/regression-testing/index.md
+++ b/site-src/performance/regression-testing/index.md
@@ -71,7 +71,7 @@ Optionally, you can also run benchmarks against the `ShareGPT` dataset for addit
 
 Update deployments for multi-LoRA support:  
 - vLLM Deployment: `./config/manifests/regression-testing/vllm/multi-lora-deployment.yaml`  
-- InferenceModel: `./config/manifests/inferencemodel.yaml`
+- InferenceObjective.yaml: `./config/manifests/inferenceobjective.yaml.yaml`
 
 Refer to example manifest:  
 `./config/manifests/regression-testing/multi-lora-regression.yaml`



Test patch:
diff --git a/conformance/tests/epp_unavailable_fail_open.go b/conformance/tests/epp_unavailable_fail_open.go
index 3d0c560..36e0fa2 100644
--- a/conformance/tests/epp_unavailable_fail_open.go
+++ b/conformance/tests/epp_unavailable_fail_open.go
@@ -22,13 +22,14 @@ import (
 
 	"github.com/stretchr/testify/require"
 	"k8s.io/apimachinery/pkg/types"
-	testfilter "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/test/filter"
 	"sigs.k8s.io/gateway-api/conformance/utils/suite"
 	"sigs.k8s.io/gateway-api/pkg/features"
 
 	"sigs.k8s.io/gateway-api-inference-extension/conformance/resources"
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/utils/config"
 	k8sutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/kubernetes"
 	trafficutils "sigs.k8s.io/gateway-api-inference-extension/conformance/utils/traffic"
+	testfilter "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/test/filter"
 )
 
 func init() {
@@ -86,9 +87,11 @@ var EppUnAvailableFailOpen = suite.ConformanceTest{
 		})
 
 		t.Run("Phase 2: Verify fail-open behavior after EPP becomes unavailable", func(t *testing.T) {
-			t.Log("Simulating an EPP failure by deleting its deployment...")
-			deleteErr := k8sutils.DeleteDeployment(t, s.Client, s.TimeoutConfig, resources.SecondaryEppDeploymentNN)
-			require.NoError(t, deleteErr, "Failed to delete the EPP deployment")
+			t.Logf("Making EPP service %v unavailable...", resources.PrimaryEppServiceNN)
+			timeconfig := config.DefaultInferenceExtensionTimeoutConfig()
+			restore, err := k8sutils.MakeServiceUnavailable(t, s.Client, resources.PrimaryEppServiceNN, timeconfig.ServiceUpdateTimeout)
+			t.Cleanup(restore)
+			require.NoError(t, err, "Failed to make the EPP service %v unavailable", resources.PrimaryEppServiceNN)
 
 			t.Log("Sending request again, expecting success to verify fail-open...")
 			trafficutils.MakeRequestAndExpectSuccess(
diff --git a/conformance/tests/httproute_invalid_inferencepool_ref.go b/conformance/tests/httproute_invalid_inferencepool_ref.go
index 97841e4..1e5eaf4 100644
--- a/conformance/tests/httproute_invalid_inferencepool_ref.go
+++ b/conformance/tests/httproute_invalid_inferencepool_ref.go
@@ -21,11 +21,12 @@ import (
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
-	"sigs.k8s.io/gateway-api-inference-extension/conformance/resources"
 	gatewayv1 "sigs.k8s.io/gateway-api/apis/v1"
 	"sigs.k8s.io/gateway-api/conformance/utils/kubernetes"
 	"sigs.k8s.io/gateway-api/conformance/utils/suite"
 	"sigs.k8s.io/gateway-api/pkg/features"
+
+	"sigs.k8s.io/gateway-api-inference-extension/conformance/resources"
 )
 
 func init() {
diff --git a/pkg/epp/flowcontrol/controller/internal/processor_test.go b/pkg/epp/flowcontrol/controller/internal/processor_test.go
index aeb46b6..09f1206 100644
--- a/pkg/epp/flowcontrol/controller/internal/processor_test.go
+++ b/pkg/epp/flowcontrol/controller/internal/processor_test.go
@@ -53,9 +53,9 @@ import (
 	"github.com/go-logr/logr"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
-
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/log/zap"
+
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/contracts"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/contracts/mocks"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/flowcontrol/framework"
diff --git a/pkg/epp/handlers/request_test.go b/pkg/epp/handlers/request_test.go
new file mode 100644
index 0000000..796082c
--- /dev/null
+++ b/pkg/epp/handlers/request_test.go
@@ -0,0 +1,70 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package handlers
+
+import (
+	"context"
+	"testing"
+
+	configPb "github.com/envoyproxy/go-control-plane/envoy/config/core/v3"
+	extProcPb "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
+)
+
+func TestHandleRequestHeaders(t *testing.T) {
+	t.Parallel()
+
+	// Setup a mock server and request context
+	server := &StreamingServer{
+		fairnessIDHeaderKey: "test-fairness-id",
+	}
+
+	reqCtx := &RequestContext{
+		Request: &Request{
+			Headers: make(map[string]string),
+		},
+	}
+
+	req := &extProcPb.ProcessingRequest_RequestHeaders{
+		RequestHeaders: &extProcPb.HttpHeaders{
+			Headers: &configPb.HeaderMap{
+				Headers: []*configPb.HeaderValue{
+					{
+						Key:   "x-test-header",
+						Value: "test-value",
+					},
+					{
+						Key:   "test-fairness-id",
+						Value: "test-fairness-id-value",
+					},
+				},
+			},
+			EndOfStream: false,
+		},
+	}
+
+	err := server.HandleRequestHeaders(context.Background(), reqCtx, req)
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+
+	if reqCtx.FairnessID != "test-fairness-id-value" {
+		t.Errorf("expected fairness ID to be 'test-fairness-id-value', got %s", reqCtx.FairnessID)
+	}
+	if reqCtx.Request.Headers["test-fairness-id"] == "test-fairness-id-value" {
+		t.Errorf("expected fairness ID header to be removed from request headers, but it was not")
+	}
+}
diff --git a/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity_test.go b/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity_test.go
new file mode 100644
index 0000000..418bafd
--- /dev/null
+++ b/pkg/epp/scheduling/framework/plugins/scorer/lora_affinity_test.go
@@ -0,0 +1,174 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	k8stypes "k8s.io/apimachinery/pkg/types"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
+	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+func TestLoraAffinityScorer(t *testing.T) {
+	tests := []struct {
+		name              string
+		request           *types.LLMRequest
+		pods              []types.Pod
+		expectedScoresPod map[string]float64 // Map of pod name to expected score
+	}{
+		{
+			name:    "Target model is active",
+			request: &types.LLMRequest{TargetModel: "active-model-1"},
+			pods: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-1": 1},
+						WaitingModels:   map[string]int{},
+						MaxActiveModels: 5,
+					},
+				},
+			},
+			expectedScoresPod: map[string]float64{
+				"pod1": 1.0,
+			},
+		},
+		{
+			name:    "Target model is waiting",
+			request: &types.LLMRequest{TargetModel: "active-model-1"},
+			pods: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-2": 2},
+						WaitingModels:   map[string]int{"active-model-1": 1},
+						MaxActiveModels: 2,
+					},
+				},
+			},
+			expectedScoresPod: map[string]float64{
+				"pod1": 0.6,
+			},
+		},
+		{
+			name:    "Pods have no space for new model",
+			request: &types.LLMRequest{TargetModel: "active-model-1"},
+			pods: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-2": 2},
+						WaitingModels:   map[string]int{"active-model-3": 1},
+						MaxActiveModels: 2,
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{},
+						WaitingModels:   map[string]int{},
+						MaxActiveModels: 0,
+					},
+				},
+			},
+			expectedScoresPod: map[string]float64{
+				"pod1": 0.0,
+				"pod2": 0.0,
+			},
+		},
+		{
+			name:    "Multiple pods with mixed active and waiting models",
+			request: &types.LLMRequest{TargetModel: "active-model-1"},
+			pods: []types.Pod{
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod1"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-1": 1},
+						WaitingModels:   map[string]int{},
+						MaxActiveModels: 5,
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-2": 4},
+						WaitingModels:   map[string]int{"active-model-1": 1},
+						MaxActiveModels: 5,
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-2": 1},
+						WaitingModels:   map[string]int{},
+						MaxActiveModels: 2,
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod4"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-3": 1},
+						WaitingModels:   map[string]int{"active-model-1": 1},
+						MaxActiveModels: 2,
+					},
+				},
+				&types.PodMetrics{
+					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod5"}},
+					MetricsState: &backendmetrics.MetricsState{
+						ActiveModels:    map[string]int{"active-model-4": 1, "active-model-5": 1},
+						WaitingModels:   map[string]int{},
+						MaxActiveModels: 2,
+					},
+				},
+			},
+			expectedScoresPod: map[string]float64{
+				"pod1": 1.0,
+				"pod2": 0.8,
+				"pod3": 0.8,
+				"pod4": 0.6,
+				"pod5": 0.0,
+			},
+		},
+		{
+			name:              "Empty pods slice",
+			request:           &types.LLMRequest{TargetModel: "modelA"},
+			pods:              []types.Pod{},
+			expectedScoresPod: map[string]float64{}, // No pods, no scores
+		},
+	}
+
+	for _, test := range tests {
+		t.Run(test.name, func(t *testing.T) {
+			scorer := &LoraAffinityScorer{}
+			scores := scorer.Score(context.Background(), types.NewCycleState(), test.request, test.pods)
+
+			for _, pod := range test.pods {
+				expectedScore, ok := test.expectedScoresPod[pod.GetPod().NamespacedName.Name]
+				if !ok {
+					t.Fatalf("Expected score not found for pod %s in test %s", pod.GetPod().NamespacedName, test.name)
+				}
+				assert.InDelta(t, expectedScore, scores[pod], 0.0001, "Pod %s should have score %f", pod.GetPod().NamespacedName.Name, expectedScore)
+			}
+			assert.Len(t, scores, len(test.expectedScoresPod), "Number of scored pods should match expected")
+		})
+	}
+}
diff --git a/pkg/epp/scheduling/scheduler_test.go b/pkg/epp/scheduling/scheduler_test.go
index ea12375..aea9c57 100644
--- a/pkg/epp/scheduling/scheduler_test.go
+++ b/pkg/epp/scheduling/scheduler_test.go
@@ -27,43 +27,27 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
 	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics" // Import config for thresholds
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
 )
 
 // Tests the default scheduler configuration and expected behavior.
 func TestSchedule(t *testing.T) {
-	loraAffinityFilter := filter.NewLoraAffinityFilter(filter.DefaultLoraAffinityThreshold)
-	leastQueueFilter := filter.NewLeastQueueFilter()
-	leastKvCacheFilter := filter.NewLeastKVCacheFilter()
-
-	lowLatencyFilter := &filter.DecisionTreeFilter{
-		Current: filter.NewLowQueueFilter(filter.DefaultQueueingThresholdLoRA),
-		NextOnSuccess: &filter.DecisionTreeFilter{
-			Current: loraAffinityFilter,
-			NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-				Current: leastQueueFilter,
-				NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-					Current: leastKvCacheFilter,
-				},
-			},
-		},
-		NextOnFailure: &filter.DecisionTreeFilter{
-			Current: leastQueueFilter,
-			NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-				Current: loraAffinityFilter,
-				NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-					Current: leastKvCacheFilter,
-				},
-			},
-		},
-	}
+	kvCacheUtilizationScorer := scorer.NewKVCacheUtilizationScorer()
+	queueingScorer := scorer.NewQueueScorer()
+	prefixCacheScorer := prefix.New(prefix.DefaultConfig)
+	loraAffinityScorer := scorer.NewLoraAffinityScorer()
 
 	defaultProfile := framework.NewSchedulerProfile().
-		WithFilters(lowLatencyFilter).
-		WithPicker(picker.NewRandomPicker(picker.DefaultMaxNumOfEndpoints))
+		WithScorers(framework.NewWeightedScorer(kvCacheUtilizationScorer, 1),
+			framework.NewWeightedScorer(queueingScorer, 1),
+			framework.NewWeightedScorer(prefixCacheScorer, 1),
+			framework.NewWeightedScorer(loraAffinityScorer, 1),
+		).
+		WithPicker(picker.NewMaxScorePicker(picker.DefaultMaxNumOfEndpoints))
 
 	profileHandler := profile.NewSingleProfileHandler()
 
@@ -110,8 +94,8 @@ func TestSchedule(t *testing.T) {
 				&types.PodMetrics{
 					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}},
 					MetricsState: &backendmetrics.MetricsState{
-						WaitingQueueSize:    3,
-						KVCacheUsagePercent: 0.1,
+						WaitingQueueSize:    0,
+						KVCacheUsagePercent: 0.2,
 						MaxActiveModels:     2,
 						ActiveModels: map[string]int{
 							"foo":      1,
@@ -123,7 +107,7 @@ func TestSchedule(t *testing.T) {
 					Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod3"}},
 					MetricsState: &backendmetrics.MetricsState{
 						WaitingQueueSize:    10,
-						KVCacheUsagePercent: 0.2,
+						KVCacheUsagePercent: 0.8,
 						MaxActiveModels:     2,
 						ActiveModels: map[string]int{
 							"foo": 1,
@@ -139,8 +123,8 @@ func TestSchedule(t *testing.T) {
 								Pod: &types.PodMetrics{
 									Pod: &backend.Pod{NamespacedName: k8stypes.NamespacedName{Name: "pod2"}},
 									MetricsState: &backendmetrics.MetricsState{
-										WaitingQueueSize:    3,
-										KVCacheUsagePercent: 0.1,
+										WaitingQueueSize:    0,
+										KVCacheUsagePercent: 0.2,
 										MaxActiveModels:     2,
 										ActiveModels: map[string]int{
 											"foo":      1,
@@ -148,6 +132,7 @@ func TestSchedule(t *testing.T) {
 										},
 									},
 								},
+								Score: 2.8,
 							},
 						},
 					},
diff --git a/pkg/epp/server/server_test.go b/pkg/epp/server/server_test.go
index de93430..fc075de 100644
--- a/pkg/epp/server/server_test.go
+++ b/pkg/epp/server/server_test.go
@@ -38,6 +38,7 @@ const (
 	podAddress                 = "1.2.3.4"
 	poolPort                   = int32(5678)
 	destinationEndpointHintKey = "test-target"
+	fairnessIDHeaderKey        = "x-fairness-id"
 	namespace                  = "ns1"
 )
 
@@ -60,14 +61,18 @@ func TestServer(t *testing.T) {
 		ctx, cancel, ds, _ := utils.PrepareForTestStreamingServer([]*v1alpha2.InferenceObjective{model},
 			[]*v1.Pod{{ObjectMeta: metav1.ObjectMeta{Name: podName}}}, "test-pool1", namespace, poolPort)
 
-		streamingServer := handlers.NewStreamingServer(namespace, destinationEndpointHintKey, ds, director)
+		streamingServer := handlers.NewStreamingServer(namespace, destinationEndpointHintKey, fairnessIDHeaderKey, ds, director)
 
 		testListener, errChan := utils.SetupTestStreamingServer(t, ctx, ds, streamingServer)
 		process, conn := utils.GetStreamingServerClient(ctx, t)
 		defer conn.Close()
 
 		// Send request headers - no response expected
-		headers := utils.BuildEnvoyGRPCHeaders(map[string]string{requestHeader: theHeaderValue, ":method": "POST"}, true)
+		headers := utils.BuildEnvoyGRPCHeaders(map[string]string{
+			requestHeader:       theHeaderValue,
+			":method":           "POST",
+			fairnessIDHeaderKey: "a-very-interesting-fairness-id",
+		}, true)
 		request := &pb.ProcessingRequest{
 			Request: &pb.ProcessingRequest_RequestHeaders{
 				RequestHeaders: headers,
diff --git a/test/integration/epp/hermetic_test.go b/test/integration/epp/hermetic_test.go
index 8da489d..ba17b64 100644
--- a/test/integration/epp/hermetic_test.go
+++ b/test/integration/epp/hermetic_test.go
@@ -67,9 +67,10 @@ import (
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/saturationdetector"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework"
-	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/filter"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/multi/prefix"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/picker"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/profile"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/framework/plugins/scorer"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/server"
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 	epptestutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/testing"
@@ -249,15 +250,15 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			),
 		},
 		{
-			name:     "select no lora despite active model, avoid excessive queue size",
+			name:     "select lora despite higher kv cache usage",
 			requests: integrationutils.GenerateStreamedRequestSet(logger, "test3", modelSQLLora, nil),
 			// Pod 2 will be picked despite NOT having the requested model active as it is above the affinity for queue size.
 			// Also it is critical, so we should still admit the request despite all queue sizes being greater than the queue
 			// size threshold.
 			pods: newPodStates(
 				podState{index: 0, queueSize: 10, kvCacheUsage: 0.2, activeModels: []string{"foo", "bar"}},
-				podState{index: 1, queueSize: 200, kvCacheUsage: 0.1, activeModels: []string{"foo", modelSQLLoraTarget}},
-				podState{index: 2, queueSize: 6, kvCacheUsage: 0.2, activeModels: []string{"foo"}},
+				podState{index: 1, queueSize: 10, kvCacheUsage: 0.4, activeModels: []string{"foo", modelSQLLoraTarget}},
+				podState{index: 2, queueSize: 10, kvCacheUsage: 0.3, activeModels: []string{"foo"}},
 			),
 			wantMetrics: map[string]string{
 				"inference_model_request_total": inferenceObjectiveRequestTotal([]label{
@@ -267,7 +268,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			},
 			wantErr: false,
 			wantResponses: integrationutils.NewRequestBufferedResponse(
-				"192.168.1.3:8000",
+				"192.168.1.2:8000",
 				fmt.Sprintf(`{"max_tokens":100,"model":%q,"prompt":"test3","temperature":0}`, modelSQLLoraTarget),
 				&configPb.HeaderValueOption{
 					Header: &configPb.HeaderValue{
@@ -301,7 +302,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			// Pod 1 will be picked because it has relatively low queue size and low KV cache.
 			pods: newPodStates(
 				podState{index: 0, queueSize: 4, kvCacheUsage: 0.2, activeModels: []string{"foo", "bar", modelSheddableTarget}},
-				podState{index: 1, queueSize: 0, kvCacheUsage: 0.85, activeModels: []string{"foo", modelSheddableTarget}},
+				podState{index: 1, queueSize: 4, kvCacheUsage: 0.85, activeModels: []string{"foo", modelSheddableTarget}},
 				podState{index: 2, queueSize: 10, kvCacheUsage: 0.9, activeModels: []string{"foo", modelSheddableTarget}},
 			),
 			wantMetrics: map[string]string{
@@ -312,7 +313,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			},
 			wantErr: false,
 			wantResponses: integrationutils.NewRequestBufferedResponse(
-				"192.168.1.2:8000",
+				"192.168.1.1:8000",
 				fmt.Sprintf(`{"max_tokens":100,"model":%q,"prompt":"test5","temperature":0}`, modelSheddableTarget),
 				&configPb.HeaderValueOption{
 					Header: &configPb.HeaderValue{
@@ -353,7 +354,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			// Pod 1 will be picked because it has relatively low queue size and low KV cache.
 			pods: newPodStates(
 				podState{index: 0, queueSize: 4, kvCacheUsage: 0.2, activeModels: []string{"foo", "bar", modelSheddableTarget}},
-				podState{index: 1, queueSize: 0, kvCacheUsage: 0.85, activeModels: []string{"foo", modelSheddableTarget}},
+				podState{index: 1, queueSize: 4, kvCacheUsage: 0.85, activeModels: []string{"foo", modelSheddableTarget}},
 				podState{index: 2, queueSize: 10, kvCacheUsage: 0.9, activeModels: []string{"foo", modelSheddableTarget}},
 			),
 			wantMetrics: map[string]string{
@@ -364,7 +365,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			},
 			wantErr: false,
 			wantResponses: integrationutils.NewRequestBufferedResponse(
-				"192.168.1.2:8000",
+				"192.168.1.1:8000",
 				fmt.Sprintf(`{"max_tokens":100,"model":%q,"prompt":"test6","temperature":0}`, modelSheddableTarget),
 				&configPb.HeaderValueOption{
 					Header: &configPb.HeaderValue{
@@ -402,9 +403,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 					},
 				},
 			},
-			// pod 0: selected
-			// pod 1: excluded; above KV cache threshold
-			// pod 2: excluded; above queue size threshold
+			// pod 0: selected due to low queue size and kv cache usage
 			pods: newPodStates(
 				podState{index: 0, queueSize: 4, kvCacheUsage: 0.2, activeModels: []string{"foo", "bar", modelSheddableTarget}},
 				podState{index: 1, queueSize: 0, kvCacheUsage: 0.85, activeModels: []string{"foo", modelSheddableTarget}},
@@ -418,7 +417,7 @@ func TestFullDuplexStreamed_KubeInferenceObjectiveRequest(t *testing.T) {
 			},
 			wantErr: false,
 			wantResponses: integrationutils.NewRequestBufferedResponse(
-				"192.168.1.2:8000",
+				"192.168.1.1:8000",
 				fmt.Sprintf(`{"max_tokens":100,"model":%q,"prompt":"test6","temperature":0}`, modelDirect),
 				&configPb.HeaderValueOption{
 					Header: &configPb.HeaderValue{
@@ -1090,35 +1089,18 @@ func BeforeSuite() func() {
 	serverRunner.PoolNamespacedName = types.NamespacedName{Name: testPoolName, Namespace: testNamespace}
 	serverRunner.Datastore = datastore.NewDatastore(context.Background(), pmf)
 
-	loraAffinityFilter := filter.NewLoraAffinityFilter(filter.DefaultLoraAffinityThreshold)
-	leastQueueFilter := filter.NewLeastQueueFilter()
-	leastKvCacheFilter := filter.NewLeastKVCacheFilter()
-
-	lowLatencyFilter := &filter.DecisionTreeFilter{
-		Current: filter.NewLowQueueFilter(filter.DefaultQueueingThresholdLoRA),
-		NextOnSuccess: &filter.DecisionTreeFilter{
-			Current: loraAffinityFilter,
-			NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-				Current: leastQueueFilter,
-				NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-					Current: leastKvCacheFilter,
-				},
-			},
-		},
-		NextOnFailure: &filter.DecisionTreeFilter{
-			Current: leastQueueFilter,
-			NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-				Current: loraAffinityFilter,
-				NextOnSuccessOrFailure: &filter.DecisionTreeFilter{
-					Current: leastKvCacheFilter,
-				},
-			},
-		},
-	}
+	kvCacheUtilizationScorer := scorer.NewKVCacheUtilizationScorer()
+	queueingScorer := scorer.NewQueueScorer()
+	prefixCacheScorer := prefix.New(prefix.DefaultConfig)
+	loraAffinityScorer := scorer.NewLoraAffinityScorer()
 
 	defaultProfile := framework.NewSchedulerProfile().
-		WithFilters(lowLatencyFilter).
-		WithPicker(picker.NewRandomPicker(picker.DefaultMaxNumOfEndpoints))
+		WithScorers(framework.NewWeightedScorer(kvCacheUtilizationScorer, 1),
+			framework.NewWeightedScorer(queueingScorer, 1),
+			framework.NewWeightedScorer(prefixCacheScorer, 1),
+			framework.NewWeightedScorer(loraAffinityScorer, 1),
+		).
+		WithPicker(picker.NewMaxScorePicker(picker.DefaultMaxNumOfEndpoints))
 
 	profileHandler := profile.NewSingleProfileHandler()