Repository: kubernetes-sigs/gateway-api-inference-extension
Pull request #762: Add queue and kv-cache scorers
Author: login: liu-cong
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
Created: 2025-04-30T15:12:33Z | Merged: 2025-04-30T20:07:55Z

They will be used with prefix cache scorer to calculate a combined score to rank pods.

KV-cache scorer:
* Return scores in [0,1]
* Pods with higher kv cache utilization gets lower score, e.g., 90% utilizations gets 1-0.9=0.1

Queue scorer:
* Returns scores in [0,1]
* Pods with higher queue depth gets lower score
* The score is relative to the shortest and longest queues, the equation is `(longestQueueDepth-podQueueDepth)/(longestQueueDepth-shortestQueueDepth)`


cc @nirrozenbaum @kfswain @ahg-g 


Diff:
diff --git a/pkg/epp/scheduling/plugins/picker/max_score_picker.go b/pkg/epp/scheduling/plugins/picker/max_score_picker.go
index 1705b7dd1c..a6d7b397ca 100644
--- a/pkg/epp/scheduling/plugins/picker/max_score_picker.go
+++ b/pkg/epp/scheduling/plugins/picker/max_score_picker.go
@@ -1,3 +1,19 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+	http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
 package picker
 
 import (
diff --git a/pkg/epp/scheduling/plugins/scorer/kvcache.go b/pkg/epp/scheduling/plugins/scorer/kvcache.go
new file mode 100644
index 0000000000..0877691d12
--- /dev/null
+++ b/pkg/epp/scheduling/plugins/scorer/kvcache.go
@@ -0,0 +1,35 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type KVCacheScorer struct{}
+
+func (ss *KVCacheScorer) Name() string {
+	return "kv-cache"
+}
+
+func (ss *KVCacheScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	scores := make(map[types.Pod]float64, len(pods))
+	for _, pod := range pods {
+		scores[pod] = 1 - pod.GetMetrics().KVCacheUsagePercent
+	}
+	return scores
+}
diff --git a/pkg/epp/scheduling/plugins/scorer/kvcache_test.go b/pkg/epp/scheduling/plugins/scorer/kvcache_test.go
new file mode 100644
index 0000000000..257a58c171
--- /dev/null
+++ b/pkg/epp/scheduling/plugins/scorer/kvcache_test.go
@@ -0,0 +1,95 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
+	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+func TestKvCacheScorer(t *testing.T) {
+	tests := []struct {
+		name              string
+		pods              []types.Pod
+		expectedScoresPod map[int]float64 // Map of pod index to expected score
+	}{
+		{
+			name: "Different KV cache utilization",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.8}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.5}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.0}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 0.2, // Highest KV cache usage (0.8) gets lowest score (1-0.8=0.2)
+				1: 0.5, // Medium KV cache usage (0.5) gets medium score (1-0.5=0.5)
+				2: 1.0, // No KV cache usage (0.0) gets highest score (1-0=1.0)
+			},
+		},
+		{
+			name: "Same KV cache utilization",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.6}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.6}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 0.4, // Both get same score (1-0.6=0.4)
+				1: 0.4,
+			},
+		},
+		{
+			name: "Zero KV cache utilization",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.0}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.0}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 1.0, // No KV cache usage gets highest score
+				1: 1.0,
+			},
+		},
+		{
+			name: "Full KV cache utilization",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 1.0}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{KVCacheUsagePercent: 0.5}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 0.0, // Full KV cache (1.0) gets lowest score (1-1=0)
+				1: 0.5, // Half KV cache (0.5) gets medium score (1-0.5=0.5)
+			},
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			ctx := types.NewSchedulingContext(context.Background(), &types.LLMRequest{}, tt.pods)
+			scorer := &KVCacheScorer{}
+			scores := scorer.Score(ctx, tt.pods)
+
+			for i, pod := range tt.pods {
+				expectedScore := tt.expectedScoresPod[i]
+				assert.InDelta(t, expectedScore, scores[pod], 0.0001, "Pod %d should have score %f", i, expectedScore)
+			}
+		})
+	}
+}
diff --git a/pkg/epp/scheduling/plugins/scorer/queue.go b/pkg/epp/scheduling/plugins/scorer/queue.go
new file mode 100644
index 0000000000..3df9d41401
--- /dev/null
+++ b/pkg/epp/scheduling/plugins/scorer/queue.go
@@ -0,0 +1,61 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct{}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(pod types.Pod) float64 {
+		if maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	// Create a map to hold the scores for each pod
+	scores := make(map[types.Pod]float64, len(pods))
+	for _, pod := range pods {
+		scores[pod] = podScoreFunc(pod)
+	}
+	return scores
+}
diff --git a/pkg/epp/scheduling/plugins/scorer/queue_test.go b/pkg/epp/scheduling/plugins/scorer/queue_test.go
new file mode 100644
index 0000000000..907681b256
--- /dev/null
+++ b/pkg/epp/scheduling/plugins/scorer/queue_test.go
@@ -0,0 +1,85 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"context"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend"
+	backendmetrics "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/backend/metrics"
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+func TestQueueScorer(t *testing.T) {
+	tests := []struct {
+		name              string
+		pods              []types.Pod
+		expectedScoresPod map[int]float64 // Map of pod index to expected score
+	}{
+		{
+			name: "Different queue sizes",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 10}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 5}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 0}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 0.0, // Longest queue (10) gets lowest score
+				1: 0.5, // Medium queue (5) gets medium score
+				2: 1.0, // Shortest queue (0) gets highest score
+			},
+		},
+		{
+			name: "Same queue sizes",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 5}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 5}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 1.0, // When all pods have the same queue size, they get the same neutral score
+				1: 1.0,
+			},
+		},
+		{
+			name: "Zero queue sizes",
+			pods: []types.Pod{
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 0}},
+				&types.PodMetrics{Pod: &backend.Pod{}, Metrics: &backendmetrics.Metrics{WaitingQueueSize: 0}},
+			},
+			expectedScoresPod: map[int]float64{
+				0: 1.0,
+				1: 1.0,
+			},
+		},
+	}
+
+	scorer := &QueueScorer{}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			ctx := types.NewSchedulingContext(context.Background(), &types.LLMRequest{}, tt.pods)
+			scores := scorer.Score(ctx, tt.pods)
+
+			for i, pod := range tt.pods {
+				expectedScore := tt.expectedScoresPod[i]
+				assert.InDelta(t, expectedScore, scores[pod], 0.0001, "Pod %d should have score %f", i, expectedScore)
+			}
+		})
+	}
+}


Reviews:
user: login: kfswain
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-30T16:40:11Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T16:50:00Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T16:56:18Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:15:11Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:21:15Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:24:16Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:27:00Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:33:00Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:35:07Z
user: login: nirrozenbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:36:37Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:39:48Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T17:41:40Z
user: login: ahg-g
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T20:05:39Z
user: login: liu-cong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-01T04:02:59Z

Review comments:
diff_hunk: @@ -0,0 +1,30 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+var KvCacheScorer = &scorerWrapper{
path: pkg/epp/scheduling/plugins/scorer/kvcache.go
user: login: kfswain
body: This makes for a very clean implementation when it works. 

However, as the queue implementation shows, this isn't always the case. I feel that this indirection would make for inconsistent implementations of `Score`, causing a new reader to have to look up the wrapper & delineate between the `ScoreFunc` & `PodScoreFunc`.

For the most part the generalization is just a `for` loop iterating over each pod and calling the score function. I think that is an acceptable repeatability tradeoff in this case.
created_at: 2025-04-30T16:40:11Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,30 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+var KvCacheScorer = &scorerWrapper{
path: pkg/epp/scheduling/plugins/scorer/kvcache.go
user: login: liu-cong
body: I found there is a pattern that generally you initialize some state, and write a helper function `scoreOnePod` anyway, and then run the for loop.

This helper just makes it a bit more reusable and we don't really need to enforce using it. It's just for convenience. And all the scorers I have now needs this, so I think it's a good convenience method to reduce duplication.
created_at: 2025-04-30T16:50:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,30 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+var KvCacheScorer = &scorerWrapper{
path: pkg/epp/scheduling/plugins/scorer/kvcache.go
user: login: liu-cong
body: I removed the wrapper and just kept the scoreFunc which is more reusable.
created_at: 2025-04-30T16:56:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,36 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type KVCacheScorer struct{}
+
+func (ss *KVCacheScorer) Name() string {
+	return "kv-cache"
+}
+
+func (ss *KVCacheScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	scoreFunc := ToScoreFunc(scorePerPod)
+	return scoreFunc(ctx, pods)
+}
+
+func scorePerPod(ctx *types.SchedulingContext, pod types.Pod) float64 {
+	return 1 - pod.GetMetrics().KVCacheUsagePercent
+}
path: pkg/epp/scheduling/plugins/scorer/kvcache.go
user: login: nirrozenbaum
body: I tend to agree with @kfswain on previous comments.
I completely understand the convenience of using these helps functions, but it comes with tradeoff of readability.
for example, this file could look like this:
```go
func (ss *KVCacheScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
        result := make(map[types.Pod]float64, len(pods))
        for _, pod := range pods {
	        result[pod] = scorePod(pod)
	}
	return result
}

func scorePod(pod types.Pod) float64 {
	return 1 - pod.GetMetrics().KVCacheUsagePercent
}

```

score function is 4 lines instead of 2, but IMO much more readable.

we need to keep in mind that GIE plugins will serve as an example and basis for others who will want to implement their own plugins when extensibility becomes more relevant. 
I'm a true fan of keep it simple in this case, even if it adds 2-3 lines.
created_at: 2025-04-30T17:15:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,36 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type KVCacheScorer struct{}
+
+func (ss *KVCacheScorer) Name() string {
+	return "kv-cache"
+}
+
+func (ss *KVCacheScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	scoreFunc := ToScoreFunc(scorePerPod)
+	return scoreFunc(ctx, pods)
+}
+
+func scorePerPod(ctx *types.SchedulingContext, pod types.Pod) float64 {
+	return 1 - pod.GetMetrics().KVCacheUsagePercent
+}
path: pkg/epp/scheduling/plugins/scorer/kvcache.go
user: login: liu-cong
body: sure, I have no strong opinion here.
created_at: 2025-04-30T17:21:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: nirrozenbaum
body: if we had the NormalizeScore function in the scheduler similarly to [kube-scheduler logic](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/#normalize-scoring), would this scorer look simpler?
the max-min calculations looks like score normalization.
created_at: 2025-04-30T17:24:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,62 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: nirrozenbaum
body: nit: `}` can be in previous line
created_at: 2025-04-30T17:27:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: liu-cong
body: The complexity in this scorer really comes from the calculation of the max and min queue size. If we use a normalizeScore, then we simply shift the complexity to the normalizeScore I think?

IMO having another normalize score plugin adds more cognitive overhead. To me having Score and weights is very straightforward: 0-least preferred, 1-most preferred. 
created_at: 2025-04-30T17:33:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,61 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct{}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: nirrozenbaum
body: nit1: 
do we need this check `maxQueueSize == 0`?
is the other condition sufficient in that case?
nit2: 
if not using ctx, it can be removed
created_at: 2025-04-30T17:35:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: nirrozenbaum
body: k, was just wondering about your opinion here :)
created_at: 2025-04-30T17:36:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,61 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct{}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: liu-cong
body: good catch, cleaned up
created_at: 2025-04-30T17:39:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: liu-cong
body: We may need something like this in the future if we find a pattern across many scorer plugins. Honestly for what we have now, I don't think it's worth it.  It took me quite some time to read the scheduler NormalizeScore implementations.
created_at: 2025-04-30T17:41:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: ahg-g
body: have we established that scorers must return a value in a specific range? In kube-scheduler we set that to 0-100 IIRC, not 0-1.
created_at: 2025-04-30T20:05:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,57 @@
+/*
+Copyright 2025 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package scorer
+
+import (
+	"math"
+
+	"sigs.k8s.io/gateway-api-inference-extension/pkg/epp/scheduling/types"
+)
+
+type QueueScorer struct {
+}
+
+func (q *QueueScorer) Name() string {
+	return "queue"
+}
+
+func (q *QueueScorer) Score(ctx *types.SchedulingContext, pods []types.Pod) map[types.Pod]float64 {
+	minQueueSize := math.MaxInt
+	maxQueueSize := math.MinInt
+
+	// Iterate through the remaining pods to find min and max
+	for _, pod := range pods {
+		queueSize := pod.GetMetrics().WaitingQueueSize
+		if queueSize < minQueueSize {
+			minQueueSize = queueSize
+		}
+		if queueSize > maxQueueSize {
+			maxQueueSize = queueSize
+		}
+	}
+
+	// podScoreFunc calculates the score based on the queue size of each pod. Longer queue gets a lower score.
+	podScoreFunc := func(ctx *types.SchedulingContext, pod types.Pod) float64 {
+		if maxQueueSize == 0 || maxQueueSize == minQueueSize {
+			// If all pods have the same queue size, return a neutral score
+			return 1.0
+		}
+		return float64(maxQueueSize-pod.GetMetrics().WaitingQueueSize) / float64(maxQueueSize-minQueueSize)
+	}
+
+	return ToScoreFunc(podScoreFunc)(ctx, pods)
+}
path: pkg/epp/scheduling/plugins/scorer/queue.go
user: login: liu-cong
body: Yes it's [0, 1] https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/a6ee5590dc54fc3018a4e5951592dcf2a7b25c46/pkg/epp/scheduling/plugins/plugins.go#L53
created_at: 2025-05-01T04:02:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: netlify[bot]
created_at: 2025-04-30T15:13:03Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *gateway-api-inference-extension* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | a4162a4e9523725b00d5eb21833b3778f389db9b |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/gateway-api-inference-extension/deploys/6812605c78a95200086e22c2 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-762--gateway-api-inference-extension.netlify.app](https://deploy-preview-762--gateway-api-inference-extension.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTc2Mi0tZ2F0ZXdheS1hcGktaW5mZXJlbmNlLWV4dGVuc2lvbi5uZXRsaWZ5LmFwcCJ9.a5yCmeD_eTK46UR-BmXEx_Bx-IkW7ftdNa0mOgLOHBk)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [gateway-api-inference-extension Preview](https://deploy-preview-762--gateway-api-inference-extension.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify site configuration](https://app.netlify.com/sites/gateway-api-inference-extension/configuration/notifications#deploy-webhooks)._
author_association: NONE
user: login: kfswain
created_at: 2025-04-30T19:52:39Z
body: /lgtm
/approve
author_association: COLLABORATOR
user: login: k8s-ci-robot
created_at: 2025-04-30T19:52:46Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/762#issuecomment-2843117808" title="Approved">kfswain</a>*, *<a href="https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/762#" title="Author self-approved">liu-cong</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fgateway-api-inference-extension).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/OWNERS)~~ [kfswain]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR