Repository: kubernetes-sigs/gateway-api-inference-extension
Pull request #423: scheduling changes for lora affinity load balancing
Author: login: kaushikmitr
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: ok-to-test
description: Indicates a non-member PR verified by an org member that is safe to test.
Created: 2025-02-27T21:37:39Z | Merged: 2025-03-04T18:37:44Z

This pull request includes several changes to the deployment configuration, metrics collection, and scheduling logic. The most important changes include updating metrics collection to include waiting adapters, and implementing a new pod selection strategy that balances load while considering model affinity.

### Scheduling Logic Enhancements:
* [`pkg/epp/scheduling/filter.go`](diffhunk://#diff-39a77c3f30c969fdb254c6c00de96eb242c30bb66489d56fc572b9e549c97363L186-R240): Replaced the `loRAAffinityPredicate` function with a new `loRASoftAffinityPredicate` function that prioritizes pods with existing model affinity while allowing for load balancing through randomization (as long as there is room to fit another adapter in the pod).
* [`pkg/epp/scheduling/scheduler.go`](diffhunk://#diff-655ce8fe0ad8bc5a9dfde966addceb6c339c38862365246e81192d162cd84d7bL39-R43): Updated the scheduling configuration to use the new `loRASoftAffinityPredicate` function and increased the `queueingThresholdLoRA` value from 50 to 128. Added a new `loraAffinityThreshold` constant to indicate the probability of preferring pods with model affinity. [[1]](diffhunk://#diff-655ce8fe0ad8bc5a9dfde966addceb6c339c38862365246e81192d162cd84d7bL39-R43) [[2]](diffhunk://#diff-655ce8fe0ad8bc5a9dfde966addceb6c339c38862365246e81192d162cd84d7bL57-R60) [[3]](diffhunk://#diff-655ce8fe0ad8bc5a9dfde966addceb6c339c38862365246e81192d162cd84d7bL80-L87)

### Deployment Configuration Changes:
* [`config/manifests/vllm/deployment.yaml`](diffhunk://#diff-17dd2df9b967d3063b6c2ac8b7d3a6e8ba9b456fbbf0ce98030e323f22cc5c52L6-R6): Added new command-line arguments for `--compilation-config`, `--max-num-seqs`, and `--max-lora-rank`. Added a new environment variable `VLLM_USE_V1`. [[1]](diffhunk://#diff-17dd2df9b967d3063b6c2ac8b7d3a6e8ba9b456fbbf0ce98030e323f22cc5c52L6-R6) [[2]](diffhunk://#diff-17dd2df9b967d3063b6c2ac8b7d3a6e8ba9b456fbbf0ce98030e323f22cc5c52R27-R43)

### Metrics Collection Updates:
* [`pkg/epp/backend/vllm/metrics.go`](diffhunk://#diff-6675e8527ef13a6df8cab88b46ac78a848104abb1fe6bf0a846276d91b1b07afR40): Added a new metric `LoraRequestInfoWaitingAdaptersMetricName` and updated the `promToPodMetrics` and `getLatestLoraMetric` functions to handle waiting adapters. Also pick the previous running + waiting adapters if there are no current running or waiting adapters [[1]](diffhunk://#diff-6675e8527ef13a6df8cab88b46ac78a848104abb1fe6bf0a846276d91b1b07afR40) [[2]](diffhunk://#diff-6675e8527ef13a6df8cab88b46ac78a848104abb1fe6bf0a846276d91b1b07afR140-R147) [[3]](diffhunk://#diff-6675e8527ef13a6df8cab88b46ac78a848104abb1fe6bf0a846276d91b1b07afL164-R206)


Diff:
diff --git a/docs/proposals/003-model-server-protocol/README.md b/docs/proposals/003-model-server-protocol/README.md
index 2ab557f7af..02efbe5cb5 100644
--- a/docs/proposals/003-model-server-protocol/README.md
+++ b/docs/proposals/003-model-server-protocol/README.md
@@ -47,3 +47,4 @@ The model server MUST expose the following LoRA adapter metrics via the same Pro
   requested adapter. Example: `"max_lora": "8"`.
   * `running_lora_adapters`: A comma separated list of adapters that are currently loaded in GPU
     memory and ready to serve requests. Example: `"running_lora_adapters": "adapter1, adapter2"`
+  * `waiting_lora_adapters`: A comma separated list of adapters that are waiting to be served. Example: `"waiting_lora_adapters": "adapter1, adapter2"`
diff --git a/pkg/epp/backend/vllm/metrics.go b/pkg/epp/backend/vllm/metrics.go
index 4973c93eb1..5b36b930e0 100644
--- a/pkg/epp/backend/vllm/metrics.go
+++ b/pkg/epp/backend/vllm/metrics.go
@@ -34,9 +34,13 @@ import (
 	logutil "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/util/logging"
 )
 
+// Metric names used in the vLLM metrics implementation.
+// Refer to the protocol doc for more details:
+// https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/docs/proposals/003-model-server-protocol
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
 	LoraRequestInfoMaxAdaptersMetricName     = "max_lora"
 	// TODO: Replace these with the num_tokens_running/waiting below once we add those to the fork.
 	RunningQueueSizeMetricName = "vllm:num_requests_running"
@@ -45,8 +49,7 @@ const (
 	RunningQueueSizeMetricName        = "vllm:num_tokens_running"
 	WaitingQueueSizeMetricName        = "vllm:num_tokens_waiting"
 	*/
-	KVCacheUsagePercentMetricName     = "vllm:gpu_cache_usage_perc"
-	KvCacheMaxTokenCapacityMetricName = "vllm:gpu_cache_max_token_capacity"
+	KVCacheUsagePercentMetricName = "vllm:gpu_cache_usage_perc"
 )
 
 type PodMetricsClientImpl struct{}
@@ -138,6 +141,14 @@ func promToPodMetrics(
 					}
 				}
 			}
+			if label.GetName() == LoraRequestInfoWaitingAdaptersMetricName {
+				if label.GetValue() != "" {
+					adapterList := strings.Split(label.GetValue(), ",")
+					for _, adapter := range adapterList {
+						updated.ActiveModels[adapter] = 0
+					}
+				}
+			}
 			if label.GetName() == LoraRequestInfoMaxAdaptersMetricName {
 				if label.GetValue() != "" {
 					updated.MaxActiveModels, err = strconv.Atoi(label.GetValue())
@@ -163,14 +174,42 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty. This happens when there are no running or waiting requests on
+		// the server, in this case it is best to use the last set of active adapters.
+		if running == "" && waiting == "" {
+			continue
+		}
+
+		// Select the metric with the latest creation timestamp.
 		if m.GetGauge().GetValue() > latestTs {
 			latestTs = m.GetGauge().GetValue()
 			latest = m
 		}
 	}
+
+	if latest == nil {
+		logger.V(logutil.TRACE).Info("Metric value Empty", "value", latest, "metric", LoraRequestInfoMetricName)
+		return nil, time.Time{}, nil
+	}
+
+	// Convert the gauge value (creation timestamp) to time.Time.
 	return latest, time.Unix(0, int64(latestTs*1000)), nil
 }
 
diff --git a/pkg/epp/scheduling/filter.go b/pkg/epp/scheduling/filter.go
index b7881468c0..d3c226732f 100644
--- a/pkg/epp/scheduling/filter.go
+++ b/pkg/epp/scheduling/filter.go
@@ -19,6 +19,8 @@ package scheduling
 import (
 	"errors"
 	"math"
+	"math/rand"
+	"time"
 
 	"github.com/go-logr/logr"
 	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/datastore"
@@ -183,18 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
-}
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityFilter(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+
+		if _, exists := pod.ActiveModels[req.ResolvedTargetModel]; exists {
+			filtered_affinity = append(filtered_affinity, pod)
+		} else if len(pod.ActiveModels) < pod.MaxActiveModels {
+			filtered_available = append(filtered_available, pod)
+		}
+	}
+
+	// Use crypto/rand for better randomization in production environments
+	randSource := rand.NewSource(time.Now().UnixNano())
+	randGen := rand.New(randSource)
+
+	// If both groups have pods, use probability to select which group to return
+	if len(filtered_affinity) > 0 && len(filtered_available) > 0 {
+		if randGen.Float64() < loraAffinityThreshold {
+			return filtered_affinity, nil
+		}
+		return filtered_available, nil
+	}
+
+	// Return whichever group has pods
+	if len(filtered_affinity) > 0 {
+		return filtered_affinity, nil
+	}
 
-// canAcceptNewLoraPredicate is a filter function to check whether a pod has room to load the adapter.
-func canAcceptNewLoraPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	return len(pod.ActiveModels) < pod.MaxActiveModels
+	return filtered_available, nil
 }
 
-func criticalRequestPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
+func criticalRequestPredicate(req *LLMRequest, _ *datastore.PodMetrics) bool {
 	return req.Critical
 }
 
diff --git a/pkg/epp/scheduling/filter_test.go b/pkg/epp/scheduling/filter_test.go
index ac765b7807..f76cece9b5 100644
--- a/pkg/epp/scheduling/filter_test.go
+++ b/pkg/epp/scheduling/filter_test.go
@@ -429,3 +429,93 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
+		tolerancePercent  = 5.0 // Allow 5% tolerance from expected distribution
+	)
+
+	// Create a test request and pods
+	req := &LLMRequest{
+		Model:               testAffinityModel,
+		ResolvedTargetModel: testAffinityModel,
+	}
+
+	// Test setup: One affinity pod and one available pod
+	pods := []*datastore.PodMetrics{
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "affinity-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels: map[string]int{
+					testAffinityModel: 1,
+				},
+			},
+		},
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "available-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels:    map[string]int{},
+			},
+		},
+	}
+
+	// Run the filter function multiple times and count the results
+	affinityCount := 0
+	availableCount := 0
+
+	// Use the actual loraAffinityThreshold as defined in the original code
+	// This test should work with whatever value is set there
+	expectedAffinityPercent := loraAffinityThreshold * 100
+	for i := 0; i < numIterations; i++ {
+		result, err := loRASoftAffinityFilter(logger, req, pods)
+		if err != nil {
+			t.Fatalf("Unexpected error: %v", err)
+		}
+
+		// Check which type of pod was returned
+		if len(result) != 1 {
+			t.Fatalf("Expected exactly one pod in result, got %d", len(result))
+		}
+
+		// Identify if the returned pod is the affinity pod or available pod
+		if _, exists := result[0].ActiveModels[testAffinityModel]; exists {
+			affinityCount++
+		} else {
+			availableCount++
+		}
+	}
+
+	// Calculate the actual percentages
+	actualAffinityPercent := float64(affinityCount) / float64(numIterations) * 100
+	actualAvailablePercent := float64(availableCount) / float64(numIterations) * 100
+
+	// Check if the distribution matches expected threshold within tolerance
+	affinityLowerBound := expectedAffinityPercent - tolerancePercent
+	affinityUpperBound := expectedAffinityPercent + tolerancePercent
+
+	availableLowerBound := actualAvailablePercent - tolerancePercent
+	availableUpperBound := actualAvailablePercent + tolerancePercent
+
+	t.Logf("Distribution results over %d iterations:", numIterations)
+	t.Logf("Expected affinity percent: %.2f%% (threshold: %.2f)", expectedAffinityPercent, loraAffinityThreshold)
+	t.Logf("Actual affinity percent: %.2f%% (%d out of %d)", actualAffinityPercent, affinityCount, numIterations)
+	t.Logf("Actual available percent: %.2f%% (%d out of %d)", actualAvailablePercent, availableCount, numIterations)
+
+	if actualAffinityPercent < affinityLowerBound || actualAffinityPercent > affinityUpperBound {
+		t.Errorf("Affinity selection percent %.2f%% outside expected range %.2f%% to %.2f%%",
+			actualAffinityPercent, affinityLowerBound, affinityUpperBound)
+	}
+	if actualAvailablePercent < availableLowerBound || actualAvailablePercent > availableUpperBound {
+		t.Errorf("Availability selection percent %.2f%% outside expected range %.2f%% to %.2f%%",
+			actualAvailablePercent, availableLowerBound, availableUpperBound)
+	}
+}
diff --git a/pkg/epp/scheduling/scheduler.go b/pkg/epp/scheduling/scheduler.go
index a969948ef5..bdddd972ef 100644
--- a/pkg/epp/scheduling/scheduler.go
+++ b/pkg/epp/scheduling/scheduler.go
@@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
+	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
+	// loraAffinityThreshold indicates the probability with which we prefer a pod with LoRA affinity over a pod without but having room to fit more LoRA adapters.
+	loraAffinityThreshold = 0.999
 )
 
 var (
@@ -54,7 +57,7 @@ var (
 		filter: leastQueuingFilterFunc,
 		nextOnSuccessOrFailure: &filter{
 			name:   "low cost LoRA",
-			filter: toFilterFunc(lowLoRACostPredicate),
+			filter: loRASoftAffinityFilter,
 			nextOnSuccessOrFailure: &filter{
 				name:   "least KV cache percent",
 				filter: leastKVCacheFilterFunc,
@@ -76,14 +79,9 @@ var (
 		name:   "low queueing filter",
 		filter: toFilterFunc((lowQueueingPodPredicate)),
 		nextOnSuccess: &filter{
-			name:          "affinity LoRA",
-			filter:        toFilterFunc(loRAAffinityPredicate),
-			nextOnSuccess: queueAndKVCacheFilter,
-			nextOnFailure: &filter{
-				name:                   "can accept LoRA Adapter",
-				filter:                 toFilterFunc(canAcceptNewLoraPredicate),
-				nextOnSuccessOrFailure: queueAndKVCacheFilter,
-			},
+			name:                   "affinity LoRA",
+			filter:                 loRASoftAffinityFilter,
+			nextOnSuccessOrFailure: queueAndKVCacheFilter,
 		},
 		nextOnFailure: queueLoRAAndKVCacheFilter,
 	}
diff --git a/test/integration/hermetic_test.go b/test/integration/hermetic_test.go
index 7755795b02..cc83650459 100644
--- a/test/integration/hermetic_test.go
+++ b/test/integration/hermetic_test.go
@@ -158,6 +158,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 					KVCacheUsagePercent: 0.2,
 					ActiveModels: map[string]int{
 						"foo": 1,
+						"bar": 1,
 					},
 				}),
 			},
@@ -200,7 +201,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 					},
 				}),
 				extprocutils.FakePodMetrics(1, datastore.Metrics{
-					WaitingQueueSize:    50,
+					WaitingQueueSize:    200,
 					KVCacheUsagePercent: 0.1,
 					ActiveModels: map[string]int{
 						"foo":            1,


Reviews:
user: login: ahg-g
body: I didn't look at the algorithm change yet, left a couple of quick comments.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T00:01:35Z
user: login: ahg-g
body: The algorithm is not using the waiting_lora_adapters metric, right?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T04:57:31Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T18:14:45Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:16:51Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:17:44Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:18:21Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:19:39Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:19:39Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:21:07Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:25:01Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:25:40Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:29:57Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:40:26Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:40:47Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:40:53Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T22:41:19Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T23:17:52Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T00:44:41Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T00:46:38Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:24:24Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:28:16Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:29:45Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:34:17Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:38:33Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T02:43:22Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T18:16:35Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T19:23:09Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:45:23Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:51:01Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T18:44:05Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T18:51:19Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T19:15:16Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T19:20:03Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T22:00:54Z
user: login: liu-cong
body: /lgtm with a few nits. Feel free to ping me to lgtm again if you want do address them
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T22:37:16Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T22:45:38Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T22:58:34Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T23:25:19Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T01:31:55Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T01:34:29Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T02:39:54Z
user: login: kaushikmitr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T18:11:18Z

Review comments:
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
path: config/manifests/vllm/deployment.yaml
user: login: ahg-g
body: what does this do?
created_at: 2025-02-27T22:08:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
+          - "3"
+          - "--max-num-seqs"
+          - "2048"
           - "--enable-lora"
           - "--max-loras"
           - "4"
           - "--max-cpu-loras"
-          - "12"
+          - "15"
+          - "--max-lora-rank"
+          - "16"
           - "--lora-modules"
           - '{"name": "tweet-summary-0", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           - '{"name": "tweet-summary-1", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           env:
+            - name: VLLM_USE_V1
+              value: "1"
path: config/manifests/vllm/deployment.yaml
user: login: ahg-g
body: The released vllm version doesn't support our metrics yet, right? if so, then we can't use it now.
created_at: 2025-02-27T23:26:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -161,14 +170,40 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty.
+		if running == "" && waiting == "" {
+			//	continue
path: pkg/epp/backend/vllm/metrics.go
user: login: ahg-g
body: commented out code
created_at: 2025-02-27T23:57:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,14 +79,9 @@ var (
 		name:   "low queueing filter",
 		filter: toFilterFunc((lowQueueingPodPredicate)),
 		nextOnSuccess: &filter{
-			name:          "affinity LoRA",
-			filter:        toFilterFunc(loRAAffinityPredicate),
-			nextOnSuccess: queueAndKVCacheFilter,
-			nextOnFailure: &filter{
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: was this never executed before?
created_at: 2025-02-28T04:43:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
path: pkg/epp/backend/vllm/metrics.go
user: login: ahg-g
body: can you please document the semantics of each metric?
created_at: 2025-02-28T04:50:06Z
author_association: CONTRIBUTOR
path: config/manifests/vllm/deployment.yaml
user: login: liu-cong
body: The changes here are not necessarily related to the algorithm change. Can we make this a separate PR? 
created_at: 2025-02-28T17:43:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
path: pkg/epp/backend/vllm/metrics.go
user: login: liu-cong
body: We can update the protocol doc and reference the protocol doc here.
created_at: 2025-02-28T17:45:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
 	LoraRequestInfoMaxAdaptersMetricName     = "max_lora"
 	// TODO: Replace these with the num_tokens_running/waiting below once we add those to the fork.
path: pkg/epp/backend/vllm/metrics.go
user: login: liu-cong
body: Can you clean up the TODOs and the metrics that are not currently used?
created_at: 2025-02-28T17:45:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: I think we should make this configurable perhaps via a flag for now. Different environments will likely need different thresholds.
created_at: 2025-02-28T17:48:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
path: pkg/epp/scheduling/filter.go
user: login: liu-cong
body: This is not a predicate, this is a filter, according to the current `filter` and `predicate` interface definition.
created_at: 2025-02-28T17:52:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
path: pkg/epp/scheduling/filter.go
user: login: liu-cong
body: pls add a warning log here and state that this should never happen
created_at: 2025-02-28T17:54:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
+		}
+
+		if _, exists := pod.ActiveModels[req.ResolvedTargetModel]; exists {
+			filtered_affinity = append(filtered_affinity, pod)
+		} else if len(pod.ActiveModels) < pod.MaxActiveModels {
path: pkg/epp/scheduling/filter.go
user: login: liu-cong
body: This is essentially the `canAcceptNewLoraPredicate` function below, are we still using `canAcceptNewLoraPredicate`?
created_at: 2025-02-28T17:55:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
+		}
+
+		if _, exists := pod.ActiveModels[req.ResolvedTargetModel]; exists {
+			filtered_affinity = append(filtered_affinity, pod)
+		} else if len(pod.ActiveModels) < pod.MaxActiveModels {
+			filtered_available = append(filtered_available, pod)
+		}
+	}
+
+	// Use crypto/rand for better randomization in production environments
+	randSource := rand.NewSource(time.Now().UnixNano())
path: pkg/epp/scheduling/filter.go
user: login: liu-cong
body: This can be a follow up, but it sounds like we can extend the current filter framework to support such probability based filtering. So instead of having one base filter, we have a list of filters with weights. This way we can keep each filter very focused, and make them more reusable
created_at: 2025-02-28T18:00:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
+	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
+	// loraAffinityThreshold indicates the probability with which we prefer a pod with LoRA affinity over a pod without but having room to fit more LoRA adapters.
+	loraAffinityThreshold = 0.999
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: do you have some insights to show why this is needed and why this value is picked?
created_at: 2025-02-28T18:01:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
path: pkg/epp/backend/vllm/metrics.go
user: login: liu-cong
body: On one hand, I can see why considering `waiting` is useful, because `waiting` loras are going to be served next. However, I have concerns of this weakening the lora affinity. `running` is bound by the `max lora`, `waiting` is not bound. If we enter an unstable state with a long `waiting`, we can lose the affinity benefit.

An improvement algorithm could be we prioritize `waiting` over `running`, what do you think?
created_at: 2025-02-28T18:14:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
+          - "3"
+          - "--max-num-seqs"
+          - "2048"
           - "--enable-lora"
           - "--max-loras"
           - "4"
           - "--max-cpu-loras"
-          - "12"
+          - "15"
+          - "--max-lora-rank"
+          - "16"
           - "--lora-modules"
           - '{"name": "tweet-summary-0", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           - '{"name": "tweet-summary-1", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           env:
+            - name: VLLM_USE_V1
+              value: "1"
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: Yes, that is why the tests are failing. I will switch back to V0
created_at: 2025-02-28T22:16:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -76,14 +79,9 @@ var (
 		name:   "low queueing filter",
 		filter: toFilterFunc((lowQueueingPodPredicate)),
 		nextOnSuccess: &filter{
-			name:          "affinity LoRA",
-			filter:        toFilterFunc(loRAAffinityPredicate),
-			nextOnSuccess: queueAndKVCacheFilter,
-			nextOnFailure: &filter{
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: It was but we had "hard" affinity, which was optimized for throughput. This helped in lowering tail latency
created_at: 2025-02-28T22:17:44Z
author_association: CONTRIBUTOR
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: I think its related in the sense that the benchmark I did relies on both changes. But still can break it to two. 
created_at: 2025-02-28T22:18:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
+		}
+
+		if _, exists := pod.ActiveModels[req.ResolvedTargetModel]; exists {
+			filtered_affinity = append(filtered_affinity, pod)
+		} else if len(pod.ActiveModels) < pod.MaxActiveModels {
path: pkg/epp/scheduling/filter.go
user: login: kaushikmitr
body: we are not using canAcceptNewLoraPredicate any more. But would be good to keep I think. 
created_at: 2025-02-28T22:19:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
+          - "3"
+          - "--max-num-seqs"
+          - "2048"
           - "--enable-lora"
           - "--max-loras"
           - "4"
           - "--max-cpu-loras"
-          - "12"
+          - "15"
+          - "--max-lora-rank"
+          - "16"
           - "--lora-modules"
           - '{"name": "tweet-summary-0", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           - '{"name": "tweet-summary-1", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           env:
+            - name: VLLM_USE_V1
+              value: "1"
path: config/manifests/vllm/deployment.yaml
user: login: ahg-g
body: I don't think that is, the integration test doesn't use this deployment yaml.

I think the test is failing because this PR introduces some randomness to the selection.
created_at: 2025-02-28T22:19:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
path: pkg/epp/backend/vllm/metrics.go
user: login: kaushikmitr
body: so using waiting + running for affinity is always superior to using just running. Because adapters get served in First come first serve basis. So we know for sure thar waiting if not available will get loaded for sure. But yes, within waiting + running prioritizing waiting over running makes sense I think, but need to test first. 
created_at: 2025-02-28T22:21:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: we may not need this if using V0. It outputs the CUDA graph for optimization.  
created_at: 2025-02-28T22:25:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -161,14 +170,40 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty.
+		if running == "" && waiting == "" {
+			//	continue
path: pkg/epp/backend/vllm/metrics.go
user: login: kaushikmitr
body: this was a bug. 
created_at: 2025-02-28T22:25:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
 	LoraRequestInfoMaxAdaptersMetricName     = "max_lora"
 	// TODO: Replace these with the num_tokens_running/waiting below once we add those to the fork.
path: pkg/epp/backend/vllm/metrics.go
user: login: kaushikmitr
body: I think the TODOs are still relevant. I will remove max token in KV Cache because its not being used. 
created_at: 2025-02-28T22:29:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
+		}
+
+		if _, exists := pod.ActiveModels[req.ResolvedTargetModel]; exists {
+			filtered_affinity = append(filtered_affinity, pod)
+		} else if len(pod.ActiveModels) < pod.MaxActiveModels {
+			filtered_available = append(filtered_available, pod)
+		}
+	}
+
+	// Use crypto/rand for better randomization in production environments
+	randSource := rand.NewSource(time.Now().UnixNano())
path: pkg/epp/scheduling/filter.go
user: login: kaushikmitr
body: agreed
created_at: 2025-02-28T22:40:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
+
+	// Pre-allocate slices with estimated capacity
+	filtered_affinity := make([]*datastore.PodMetrics, 0, len(pods))
+	filtered_available := make([]*datastore.PodMetrics, 0, len(pods))
+
+	// Categorize pods based on affinity and availability
+	for _, pod := range pods {
+		if pod == nil {
+			continue
path: pkg/epp/scheduling/filter.go
user: login: kaushikmitr
body: removed this, as this scenario is captured already upstream
created_at: 2025-02-28T22:40:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,10 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityPredicate(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
path: pkg/epp/scheduling/filter.go
user: login: kaushikmitr
body: agreed
created_at: 2025-02-28T22:40:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: I would rather levarage this to make this configurable. https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16
created_at: 2025-02-28T22:41:19Z
author_association: CONTRIBUTOR
path: config/manifests/vllm/deployment.yaml
user: login: liu-cong
body: This is just an example manifest for startup guide. 
created_at: 2025-02-28T23:17:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,15 +24,23 @@ spec:
           - "1"
           - "--port"
           - "8000"
+          - "--compilation-config"
+          - "3"
+          - "--max-num-seqs"
+          - "2048"
           - "--enable-lora"
           - "--max-loras"
           - "4"
           - "--max-cpu-loras"
-          - "12"
+          - "15"
+          - "--max-lora-rank"
+          - "16"
           - "--lora-modules"
           - '{"name": "tweet-summary-0", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           - '{"name": "tweet-summary-1", "path": "vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm", "base_model_name": "llama-2"}'
           env:
+            - name: VLLM_USE_V1
+              value: "1"
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: this one probably is failing because of V1: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_gateway-api-inference-extension/423/pull-gateway-api-inference-extension-verify-main/1895229573222633472
created_at: 2025-02-28T23:30:53Z
author_association: CONTRIBUTOR
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: agreed removed the manifest changes

created_at: 2025-03-01T00:34:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +37,7 @@ import (
 const (
 	LoraRequestInfoMetricName                = "vllm:lora_requests_info"
 	LoraRequestInfoRunningAdaptersMetricName = "running_lora_adapters"
+	LoraRequestInfoWaitingAdaptersMetricName = "waiting_lora_adapters"
path: pkg/epp/backend/vllm/metrics.go
user: login: kaushikmitr
body: added protocol link here and also updated protocol readme.md
created_at: 2025-03-01T00:41:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
+	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
+	// loraAffinityThreshold indicates the probability with which we prefer a pod with LoRA affinity over a pod without but having room to fit more LoRA adapters.
+	loraAffinityThreshold = 0.999
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: I picked it after some trail and error. This value worked well when we had skewed traffic for different adapters, helped spread out high QPS adapters while keeping low QPS adapters less spread out

created_at: 2025-03-01T00:44:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: I don't think we have time to do API change for the next release.  Given we already had to change it on different accelerator types, it's important to have this knob configurable. Exposing it as a flag seems straightforward and gives us time to gather feedback on this before making an API change.
created_at: 2025-03-01T00:44:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: I took at look, iiuc,  adding this flag is not straightforward, the way scheduler is written. If its needed for next release would rather have it in another PR. 
created_at: 2025-03-01T02:24:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -47,3 +47,5 @@ The model server MUST expose the following LoRA adapter metrics via the same Pro
   requested adapter. Example: `"max_lora": "8"`.
   * `running_lora_adapters`: A comma separated list of adapters that are currently loaded in GPU
     memory and ready to serve requests. Example: `"running_lora_adapters": "adapter1, adapter2"`
+  * `waiting_lora_adapters`: A comma separated list of adapters that are currently loaded in GPU
+    memory and ready to serve requests. Example: `"waiting_lora_adapters": "adapter1, adapter2"`
path: docs/proposals/003-model-server-protocol/README.md
user: login: ahg-g
body: update the docs, this reads exactly the same as the running one
created_at: 2025-03-01T02:28:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -161,14 +172,41 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty.
+		if running == "" && waiting == "" {
+			continue
path: pkg/epp/backend/vllm/metrics.go
user: login: ahg-g
body: Does this happen in practice? in what cases? 
created_at: 2025-03-01T02:29:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -161,14 +172,41 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty.
+		if running == "" && waiting == "" {
+			continue
path: pkg/epp/backend/vllm/metrics.go
user: login: kaushikmitr
body: When there are no running requests, that is qps = 0 for the pod. In this case it's good to use last adapters that ran to determine affinity (as a proxy for Lora metric for what's already loaded)
created_at: 2025-03-01T02:34:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: Defining a flag for each parameter is tedious, we can use a versioned configuration file, this is called ComponentConfig, ideally we do that for https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/383

Here is JobSet's config file as an example: https://github.com/kubernetes-sigs/jobset/tree/main/api/config/v1alpha1
created_at: 2025-03-01T02:38:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -161,14 +172,41 @@ func getLatestLoraMetric(logger logr.Logger, metricFamilies map[string]*dto.Metr
 		logger.V(logutil.DEFAULT).Error(nil, "Metric family not found", "name", LoraRequestInfoMetricName)
 		return nil, time.Time{}, fmt.Errorf("metric family %q not found", LoraRequestInfoMetricName)
 	}
-	var latestTs float64
+
 	var latest *dto.Metric
+	var latestTs float64
+
+	// Iterate over all metrics in the family.
 	for _, m := range loraRequests.GetMetric() {
+		var running, waiting string
+		// Read the label values for running and waiting adapters.
+		for _, lp := range m.GetLabel() {
+			switch lp.GetName() {
+			case LoraRequestInfoRunningAdaptersMetricName:
+				running = lp.GetValue()
+			case LoraRequestInfoWaitingAdaptersMetricName:
+				waiting = lp.GetValue()
+			}
+		}
+
+		// Ignore metrics with both labels empty.
path: pkg/epp/backend/vllm/metrics.go
user: login: ahg-g
body: ```suggestion
		// Ignore metrics with both labels empty. This happens when there are no running or waiting requests on 
		// the server, in this case it is best to use the last set of active adapters.
```
created_at: 2025-03-01T02:43:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
+	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
+	// loraAffinityThreshold indicates the probability with which we prefer a pod with LoRA affinity over a pod without but having room to fit more LoRA adapters.
+	loraAffinityThreshold = 0.999
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: I believe we need to update the hermetic_test case `"select active lora, low queue"`, given  the new probabilistic behavior. You can set the pods without the requested lora adapter to have no room so they will never be picked.
created_at: 2025-03-01T17:53:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,18 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
-}
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityFilter(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
path: pkg/epp/scheduling/filter.go
user: login: liu-cong
body: Can you add a unit test for this?
created_at: 2025-03-01T17:54:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: This is a new "config" CRD? If we think about heterogeneous pools in the future, then we will need different configurations per "segment" of the pool. Though by carefully design the semantics such as the default config for the pool, and each segment can override, we can still be future proof. But we need a bit more discussion on this. Will create an issue and raise this in the community meeting.

For now, I still think adding a flag is the fastest way to unblock this.

> adding this flag is not straightforward, the way scheduler is written.
If plumbing is too cumbersome, cou can define this as a flag in the current file, and then in main.go just reference it like `_ = scheduler.LoraAffinityThreshold`, this should work.
created_at: 2025-03-01T18:16:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: My general concern is that this is a rapidly evolving algorithm, those knobs may be dropped soon in another version of the algorithm.

> This is a new "config" CRD?

Not a CRD, a config file, this is the practice for all kube components as well including kube-scheduler. The config file can be mounted as a ConfigMap, it just offers a better way of structuring the parameters compared to flags which is a flat space. But even this, although versioned, may not be a good path to expose knobs given the concern I mention above.

> If we think about heterogeneous pools in the future, then we will need different configurations per "segment" of the pool. Though by carefully design the semantics such as the default config for the pool, and each segment can override, we can still be future proof.

Tuning algorithmic parameters like this one per segment of the pool is not sustainable, even tuning it for the whole pool is not. It is fine for now to have them explicitly set, but we need to evolve it in a way that the algorithm offers self tuning for the most part. 

> Though by carefully design the semantics such as the default config for the pool, and each segment can override, we can still be future proof. But we need a bit more discussion on this. Will create an issue and raise this in the community meeting.

I am also thinking about how we support setting metrics names to address https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/383, and how would that work for a heterogeneous pool of model servers (vllm + jetstream).


One compromise: just so we don't lock ourselves into a set of flags that we have to deprecate later, and unblock having knobs to tune a rapidly evolving algorithm, we can perhaps use env vars (vllm does this). Env vars might be easier to drop once we evolve into our canonical configuration form? We can prefix them with `EXPERIMENTAL_` to clearly indicate that?
created_at: 2025-03-01T19:23:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: Why are env vars easier to drop? 
created_at: 2025-03-03T17:45:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: Because setting an env var is not validated in general (i.e., if you set an env var that the binary doesn't care about, nothing will happen), while with flags it is more strict.

It is ugly either way, lack of validation is bad, but I would rather prioritize asserting that this is a temporary knob that will likely either not exist in the future or be set via a different mechanism.
created_at: 2025-03-03T17:51:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: I suggest to not block this PR on adding the env vars, and do that as a followup for this and other algorithm parameters.
created_at: 2025-03-03T18:44:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: liu-cong
body: sounds good to me. I am OK to lgtm once the integration test is fixed, and a unit test for the new filter
created_at: 2025-03-03T18:51:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: "but we need to evolve it in a way that the algorithm offers self tuning for the most part.": My concern is that we may never have a truly model-server configuration agnostic load balancing algorithm. For example, vLLM exposes settings like max_num_seq, max_lora, max_lora_rank, and max_num_batched_tokens, and the optimal load balancing strategy will depend on these parameters—many of which may not be directly available to the Gateway. While we could choose to scrape them as needed, I’m not sure if that’s the best design choice. I believe there will always be some scheduling configuration parameters that require independent tuning. For now, using environment variables works, but in the long run, we might want to make them configurable as load balancing parameters. 
created_at: 2025-03-03T19:15:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
path: pkg/epp/scheduling/scheduler.go
user: login: ahg-g
body: I agree, I am just not confident that the current set of parameters are the ones we will actually have moving forward. We are very early, and our benchmarking so far is limited to a few use cases, and so only when we have more benchmarks across different model sizes/accelerators/ datasets or deployed more widely in practice we will get the confidence of what knobs should be exposed.
created_at: 2025-03-03T19:20:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,8 +36,11 @@ const (
 	queueThresholdCritical = 5
 	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
 	// the threshold for queued requests to be considered low below which we can prioritize LoRA affinity.
-	// The value of 50 is arrived heuristicically based on experiments.
-	queueingThresholdLoRA = 50
+	// The value of 128 is arrived heuristicically based on experiments.
+	queueingThresholdLoRA = 128
+	// TODO(https://github.com/kubernetes-sigs/gateway-api-inference-extension/issues/16) Make this configurable.
+	// loraAffinityThreshold indicates the probability with which we prefer a pod with LoRA affinity over a pod without but having room to fit more LoRA adapters.
+	loraAffinityThreshold = 0.999
path: pkg/epp/scheduling/scheduler.go
user: login: kaushikmitr
body: yes, the current test might fail sometimes. I tried to make it probablistic but its more complicated than I thought. For now fixed it as you suggested by setting  the pods without the requested lora adapter to have no room so they will never be picked
created_at: 2025-03-03T21:23:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -183,18 +185,59 @@ func lowLoRACostPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
 	return ok || len(pod.ActiveModels) < pod.MaxActiveModels
 }
 
-// loRAAffinityPredicate is a filter function to check whether a pod has affinity to the lora requested.
-func loRAAffinityPredicate(req *LLMRequest, pod *datastore.PodMetrics) bool {
-	_, ok := pod.ActiveModels[req.ResolvedTargetModel]
-	return ok
-}
+// loRASoftAffinityPredicate implements a pod selection strategy that prioritizes pods
+// with existing LoRA model affinity while allowing for load balancing through randomization.
+//
+// The function works by:
+// 1. Separating pods into two groups: those with target model affinity and those with available capacity
+// 2. Using a probability threshold to sometimes select from non-affinity pods to enable load balancing
+// 3. Falling back to whatever group has pods if one group is empty
+//
+// Parameters:
+//   - logger: Logger interface for diagnostic output
+//   - req: LLM request containing the resolved target model
+//   - pods: Slice of pod metrics to filter
+//
+// Returns:
+//   - Filtered slice of pod metrics based on affinity and availability
+//   - Error if any issues occur during filtering
+func loRASoftAffinityFilter(logger logr.Logger, req *LLMRequest, pods []*datastore.PodMetrics) ([]*datastore.PodMetrics, error) {
path: pkg/epp/scheduling/filter.go
user: login: kaushikmitr
body: yes, added. 
created_at: 2025-03-03T22:00:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -142,6 +142,7 @@ func TestKubeInferenceModelRequest(t *testing.T) {
 					KVCacheUsagePercent: 0.2,
 					ActiveModels: map[string]int{
 						"foo": 1,
+						"bar": 1,
path: test/integration/hermetic_test.go
user: login: liu-cong
body: nit: Can you explain this in the comment line 121 that because no pods have room for new lora, therefore the pod with lora affinity will always be picked?
created_at: 2025-03-03T22:20:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
path: pkg/epp/scheduling/filter_test.go
user: login: liu-cong
body: nit: not used?
created_at: 2025-03-03T22:28:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
path: pkg/epp/scheduling/filter_test.go
user: login: liu-cong
body: having to run 10k times seems a lot. How long does it take? Any chance we can reduce it?
created_at: 2025-03-03T22:29:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
+		tolerancePercent  = 5.0 // Allow 5% tolerance from expected distribution
+	)
+
+	// Create a test request and pods
+	req := &LLMRequest{
+		Model:               testAffinityModel,
+		ResolvedTargetModel: testAffinityModel,
+	}
+
+	// Test setup: One affinity pod and one available pod
+	pods := []*datastore.PodMetrics{
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "affinity-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels: map[string]int{
+					testAffinityModel: 1,
+				},
+			},
+		},
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "available-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels:    map[string]int{},
+			},
+		},
+	}
+
+	// Run the filter function multiple times and count the results
+	affinityCount := 0
+	availableCount := 0
+
+	// Use the actual loraAffinityThreshold as defined in the original code
+	// This test should work with whatever value is set there
+	expectedAffinityPercent := loraAffinityThreshold * 100
+
+	for i := 0; i < numIterations; i++ {
+		result, err := loRASoftAffinityFilter(logger, req, pods)
+		if err != nil {
+			t.Fatalf("Unexpected error: %v", err)
+		}
+
+		// Check which type of pod was returned
+		if len(result) != 1 {
+			t.Fatalf("Expected exactly one pod in result, got %d", len(result))
+		}
+
+		// Identify if the returned pod is the affinity pod or available pod
+		if _, exists := result[0].ActiveModels[testAffinityModel]; exists {
path: pkg/epp/scheduling/filter_test.go
user: login: liu-cong
body: nit: I recommend checking the pod name directly, it's more readable and reliable. (imagine in the future the ActiveModels is modified during tests for some reason).
created_at: 2025-03-03T22:33:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
+		tolerancePercent  = 5.0 // Allow 5% tolerance from expected distribution
+	)
+
+	// Create a test request and pods
+	req := &LLMRequest{
+		Model:               testAffinityModel,
+		ResolvedTargetModel: testAffinityModel,
+	}
+
+	// Test setup: One affinity pod and one available pod
+	pods := []*datastore.PodMetrics{
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "affinity-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels: map[string]int{
+					testAffinityModel: 1,
+				},
+			},
+		},
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "available-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels:    map[string]int{},
+			},
+		},
+	}
+
+	// Run the filter function multiple times and count the results
+	affinityCount := 0
+	availableCount := 0
+
+	// Use the actual loraAffinityThreshold as defined in the original code
+	// This test should work with whatever value is set there
+	expectedAffinityPercent := loraAffinityThreshold * 100
+
+	for i := 0; i < numIterations; i++ {
+		result, err := loRASoftAffinityFilter(logger, req, pods)
+		if err != nil {
+			t.Fatalf("Unexpected error: %v", err)
+		}
+
+		// Check which type of pod was returned
+		if len(result) != 1 {
+			t.Fatalf("Expected exactly one pod in result, got %d", len(result))
+		}
+
+		// Identify if the returned pod is the affinity pod or available pod
+		if _, exists := result[0].ActiveModels[testAffinityModel]; exists {
+			affinityCount++
+		} else {
+			availableCount++
+		}
+	}
+
+	// Calculate the actual percentages
+	actualAffinityPercent := float64(affinityCount) / float64(numIterations) * 100
+	actualAvailablePercent := float64(availableCount) / float64(numIterations) * 100
+
+	// Check if the distribution matches expected threshold within tolerance
+	affinityLowerBound := expectedAffinityPercent - tolerancePercent
+	affinityUpperBound := expectedAffinityPercent + tolerancePercent
+
+	t.Logf("Distribution results over %d iterations:", numIterations)
+	t.Logf("Expected affinity percent: %.2f%% (threshold: %.2f)", expectedAffinityPercent, loraAffinityThreshold)
+	t.Logf("Actual affinity percent: %.2f%% (%d out of %d)", actualAffinityPercent, affinityCount, numIterations)
+	t.Logf("Actual available percent: %.2f%% (%d out of %d)", actualAvailablePercent, availableCount, numIterations)
+
+	if actualAffinityPercent < affinityLowerBound || actualAffinityPercent > affinityUpperBound {
path: pkg/epp/scheduling/filter_test.go
user: login: liu-cong
body: Just for the sake of completeness, I think we should also check the availablePercent here.
created_at: 2025-03-03T22:36:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
path: pkg/epp/scheduling/filter_test.go
user: login: kaushikmitr
body: it takes 0.29 seconds, I had it 1000 initially which took 0.16 seconds. 1000 is fine but given the probabiliy i have is very skewed 99.9%, 10000 gives assigns some pods to both cases (~9990, 100)
created_at: 2025-03-03T22:45:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
path: pkg/epp/scheduling/filter_test.go
user: login: liu-cong
body: If the issue is the 99.9% threshold, you can change the percentage in tests to make it easier to test, like 90%
created_at: 2025-03-03T22:58:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
path: pkg/epp/scheduling/filter_test.go
user: login: kaushikmitr
body: yup thats a bit complicated though (for now) as the threshold is defined in the scheduler as const and the  filter template does not take as input any arbitrary parameter. I have to update the const for the test and reset it back. I would rather test it against whats already set in the scheduler, 0.29 sec should not be too bad, if it is I can change it go 1000 (the test would still work, since it has a tolerance for 5%)
created_at: 2025-03-03T23:25:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,4 +120,4 @@ data:
           - base-model: meta-llama/Llama-2-7b-hf
             id: tweet-summary-1
             source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
-  
+  
path: config/manifests/vllm/deployment.yaml
user: login: ahg-g
body: can you please revert this file
created_at: 2025-03-04T01:31:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,4 +120,4 @@ data:
           - base-model: meta-llama/Llama-2-7b-hf
             id: tweet-summary-1
             source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
-  
+  
path: config/manifests/vllm/deployment.yaml
user: login: ahg-g
body: mostly to avoid unnecessary conflict with https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/436
created_at: 2025-03-04T01:34:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,4 +120,4 @@ data:
           - base-model: meta-llama/Llama-2-7b-hf
             id: tweet-summary-1
             source: vineetsharma/qlora-adapter-Llama-2-7b-hf-TweetSumm
-  
+  
path: config/manifests/vllm/deployment.yaml
user: login: kaushikmitr
body: done
created_at: 2025-03-04T02:39:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -429,3 +429,87 @@ func TestFilterFunc(t *testing.T) {
 		})
 	}
 }
+
+// TestLoRASoftAffinityDistribution tests that the loRASoftAffinityFilter function
+// properly distributes requests according to the loraAffinityThreshold
+func TestLoRASoftAffinityDistribution(t *testing.T) {
+	logger := logutil.NewTestLogger()
+
+	const (
+		testModelName     = "test-model"
+		testAffinityModel = "test-affinity-model"
+		numIterations     = 10000
+		tolerancePercent  = 5.0 // Allow 5% tolerance from expected distribution
+	)
+
+	// Create a test request and pods
+	req := &LLMRequest{
+		Model:               testAffinityModel,
+		ResolvedTargetModel: testAffinityModel,
+	}
+
+	// Test setup: One affinity pod and one available pod
+	pods := []*datastore.PodMetrics{
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "affinity-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels: map[string]int{
+					testAffinityModel: 1,
+				},
+			},
+		},
+		{
+			Pod: datastore.Pod{NamespacedName: types.NamespacedName{Name: "available-pod"}},
+			Metrics: datastore.Metrics{
+				MaxActiveModels: 2,
+				ActiveModels:    map[string]int{},
+			},
+		},
+	}
+
+	// Run the filter function multiple times and count the results
+	affinityCount := 0
+	availableCount := 0
+
+	// Use the actual loraAffinityThreshold as defined in the original code
+	// This test should work with whatever value is set there
+	expectedAffinityPercent := loraAffinityThreshold * 100
+
+	for i := 0; i < numIterations; i++ {
+		result, err := loRASoftAffinityFilter(logger, req, pods)
+		if err != nil {
+			t.Fatalf("Unexpected error: %v", err)
+		}
+
+		// Check which type of pod was returned
+		if len(result) != 1 {
+			t.Fatalf("Expected exactly one pod in result, got %d", len(result))
+		}
+
+		// Identify if the returned pod is the affinity pod or available pod
+		if _, exists := result[0].ActiveModels[testAffinityModel]; exists {
+			affinityCount++
+		} else {
+			availableCount++
+		}
+	}
+
+	// Calculate the actual percentages
+	actualAffinityPercent := float64(affinityCount) / float64(numIterations) * 100
+	actualAvailablePercent := float64(availableCount) / float64(numIterations) * 100
+
+	// Check if the distribution matches expected threshold within tolerance
+	affinityLowerBound := expectedAffinityPercent - tolerancePercent
+	affinityUpperBound := expectedAffinityPercent + tolerancePercent
+
+	t.Logf("Distribution results over %d iterations:", numIterations)
+	t.Logf("Expected affinity percent: %.2f%% (threshold: %.2f)", expectedAffinityPercent, loraAffinityThreshold)
+	t.Logf("Actual affinity percent: %.2f%% (%d out of %d)", actualAffinityPercent, affinityCount, numIterations)
+	t.Logf("Actual available percent: %.2f%% (%d out of %d)", actualAvailablePercent, availableCount, numIterations)
+
+	if actualAffinityPercent < affinityLowerBound || actualAffinityPercent > affinityUpperBound {
path: pkg/epp/scheduling/filter_test.go
user: login: kaushikmitr
body: done
created_at: 2025-03-04T18:11:18Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2025-02-27T21:37:49Z
body: Hi @kaushikmitr. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgateway-api-inference-extension).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

author_association: CONTRIBUTOR
user: login: netlify[bot]
created_at: 2025-02-27T21:38:02Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *gateway-api-inference-extension* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | b5d7f8f5f6488bdbad94215e6b455a33ba3d0cf2 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/gateway-api-inference-extension/deploys/67c7403f524362000864c096 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-423--gateway-api-inference-extension.netlify.app](https://deploy-preview-423--gateway-api-inference-extension.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTQyMy0tZ2F0ZXdheS1hcGktaW5mZXJlbmNlLWV4dGVuc2lvbi5uZXRsaWZ5LmFwcCJ9.aOdd6O0cxuX1M2e_j7guSCHj-hPDbUOdNIG7Zi1zUHk)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [gateway-api-inference-extension Preview](https://deploy-preview-423--gateway-api-inference-extension.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify site configuration](https://app.netlify.com/sites/gateway-api-inference-extension/configuration/notifications#deploy-webhooks)._
author_association: NONE
user: login: ahg-g
created_at: 2025-02-27T21:48:33Z
body: /ok-to-test
author_association: CONTRIBUTOR
user: login: kaushikmitr
created_at: 2025-02-28T22:27:17Z
body: > The algorithm is not using the waiting_lora_adapters metric, right?

It is, we are now checking for both waiting + running to determine affinity
author_association: CONTRIBUTOR
user: login: kaushikmitr
created_at: 2025-03-01T00:47:29Z
body: /retest 
author_association: CONTRIBUTOR
user: login: ahg-g
created_at: 2025-03-01T02:44:12Z
body: /approve
/hold

Thanks a lot, this is great, leaving it to Cong to lgtm.
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-03-01T02:44:20Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/423#issuecomment-2691870384" title="Approved">ahg-g</a>*, *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/423#" title="Author self-approved">kaushikmitr</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgateway-api-inference-extension).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/OWNERS)~~ [ahg-g]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: liu-cong
created_at: 2025-03-04T18:26:54Z
body: /lgtm

Thanks @kaushikmitr!

This PR introduces a unit test based on probabilities. We chatted offline with @kaushikmitr that with the large # of iterations, no flakiness was found with 100 test runs. So despite the test is not deterministic, in practice the test should be very reliable.
author_association: CONTRIBUTOR
user: login: liu-cong
created_at: 2025-03-04T18:27:20Z
body: /unhold
author_association: CONTRIBUTOR