Repository: kubernetes-sigs/gateway-api-inference-extension

Issue:
Pods not coming up for CPU deployment
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

**What happened**:

Pods not coming up when for CPU deployment:
```
$ k get pods
NAME                                       READY   STATUS     RESTARTS   AGE
vllm-llama3-8b-instruct-586778f79c-hb4vp   0/1     Init:0/1   0          6m15s
vllm-llama3-8b-instruct-586778f79c-qjk94   0/1     Init:0/1   0          6m15s
vllm-llama3-8b-instruct-586778f79c-w6pjh   0/1     Init:0/1   0          6m15s
```
Upon looking further, it seems the container lora is not starting:

```
$ kubectl logs vllm-llama3-8b-instruct-586778f79c-hb4vp
Defaulted container "lora" out of: lora, lora-adapter-syncer (init)
Error from server (BadRequest): container "lora" in pod "vllm-llama3-8b-instruct-586778f79c-7gmsb" is waiting to start: PodInitializing
```

**What you expected to happen**:

**How to reproduce it (as minimally and precisely as possible)**:

Try steps for CPU deployment from getting started guide: https://gateway-api-inference-extension.sigs.k8s.io/guides/

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Inference extension version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- Install tools:
- Others:


Discussion:
@nirrozenbaum: which version was used for the cpu image?
can you give it a try with v0.8.5 as you did in #1526 PR?@rahulgurnani: Yes @nirrozenbaum , that version of vlllm is working. I created the bug so that we could track it going forward if need to make some updates on our end. I don't think pinning a version of vllm is a sustainable solution. What do you think?@rahulgurnani: The lora-syncer container is an init container and is part of IGW codebase , tools/dynamic-lora-sidecar does the loading of lora adapter dynamically. I tried with deleting the CPU deployment on a running cluster and then redeploying with the latest vllm image. The CPU deployment succeded and I pods did come up. So this seems to happen only on newly created cluster. I will add more investigation notes to this bug. @nirrozenbaum: > Yes [@nirrozenbaum](https://github.com/nirrozenbaum) , that version of vlllm is working. I created the bug so that we could track it going forward if need to make some updates on our end. I don't think pinning a version of vllm is a sustainable solution. What do you think?

pinning to a version is not a long term solution.
we should keep an eye and check newer versions of vllm as those are released.
we hace issue #1501 to track it from the GPU images. I assume we can track CPU images as well (and I expect it to get solved for both in the same vllm release).
maybe for now you can just add a comment with the relevant information on that issue.@rahulgurnani: Adding logs from the lora-adapter-syncer container:

```
rahulgurnani@rahulgurnani3:~/gateway-api-inference-extension$ kubectl logs vllm-llama3-8b-instruct-586778f79c-4m87j -c lora-adapter-syncer
2025-09-05 18:18:06 - INFO - sidecar.py:118 -  Settings initialized: health check timeout=300s, interval=2s, reconcile trigger=5s
2025-09-05 18:18:06 - INFO - sidecar.py:332 -  Starting metrics server on port 8080
2025-09-05 18:18:06 - INFO - sidecar.py:335 -  Running initial reconcile for config map /config/configmap.yaml
2025-09-05 18:18:06 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:23:07 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:23:07 - INFO - sidecar.py:346 -  Starting to watch /config/configmap.yaml for changes and performing periodic reconciliation...
2025-09-05 18:23:07 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:23:12 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:23:12 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:28:12 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:28:12 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:28:17 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:28:17 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
2025-09-05 18:33:18 - ERROR - sidecar.py:286 -  vllm server at localhost:8000 not healthy
2025-09-05 18:33:18 - INFO - sidecar.py:350 -  Waiting 5s before next reconciliation...
2025-09-05 18:33:23 - INFO - sidecar.py:354 -  Periodic reconciliation triggered
2025-09-05 18:33:23 - INFO - sidecar.py:281 -  reconciling model server localhost:8000 with config stored at /config/configmap.yaml
```
It seems the container is not coming up because its waiting for vllm server at localhost:8000 to be healthy. @rahulgurnani: More investigation notes here, deploying with vllm latest without the lora syncer init container succeeds i.e. using the following CPU deployment succeeds:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b-instruct
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm-llama3-8b-instruct
  template:
    metadata:
      labels:
        app: vllm-llama3-8b-instruct
    spec:
      containers:
        - name: lora
          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.10.1" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
          imagePullPolicy: IfNotPresent
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
          - "--model"
          - "Qwen/Qwen2.5-1.5B-Instruct"
          - "--port"
          - "8000"
          - "--enable-lora"
          - "--max-loras"
          - "4"
          - "--lora-modules"
          - '{"name": "food-review-0", "path": "SriSanth2345/Qwen-1.5B-Tweet-Generations", "base_model_name": "Qwen/Qwen2.5-1.5B"}'
          - '{"name": "food-review-1", "path": "SriSanth2345/Qwen-1.5B-Tweet-Generations", "base_model_name": "Qwen/Qwen2.5-1.5B"}'
          env:
            - name: PORT
              value: "8000"
            - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
              value: "true"
            - name: VLLM_CPU_KVCACHE_SPACE
              value: "4"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          livenessProbe:
            failureThreshold: 240
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 600
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            initialDelaySeconds: 5
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1
          resources:
             limits:
               cpu: "12"
               memory: "9000Mi"
             requests:
               cpu: "12"
               memory: "9000Mi"
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: shm
            - name: adapters
              mountPath: "/adapters"
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      volumes:
        - name: data
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
        - name: adapters
          emptyDir: {}
        - name: config-volume
```@nirrozenbaum: @rahulgurnani that's a very interesting finding. 
I assume the same is correct for GPU deployment, and more generally sounds like the bug in on our side (in lora syncer) and not in vllm images.

I'm guessing that something in the API has changed after vllm version 0.8.5, and then lora-syncer is not working properly with that.

I think it might be worth opening a dedicate issue (which should remain open) to describe the issues you're seeing.
we should track this in an issue and close the issue when the problem is fixed.

Solution patch:
diff --git a/config/manifests/vllm/cpu-deployment.yaml b/config/manifests/vllm/cpu-deployment.yaml
index 3e38d4e..8b759d3 100644
--- a/config/manifests/vllm/cpu-deployment.yaml
+++ b/config/manifests/vllm/cpu-deployment.yaml
@@ -14,7 +14,7 @@ spec:
     spec:
       containers:
         - name: lora
-          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
+          image: "public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:v0.8.5" # formal images can be found in https://gallery.ecr.aws/q9t5s3a7/vllm-cpu-release-repo
           imagePullPolicy: Always
           command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
           args:
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
index a0f05c8..49ec7fa 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go
@@ -21,6 +21,7 @@ import (
 	"encoding/binary"
 	"encoding/json"
 	"fmt"
+	"sync"
 
 	"github.com/cespare/xxhash/v2"
 	k8stypes "k8s.io/apimachinery/pkg/types"
@@ -78,6 +79,7 @@ type Plugin struct {
 	config      Config
 	pluginState *plugins.PluginState
 	indexer     Indexer
+	wg          sync.WaitGroup
 }
 
 // podSet holds an pods servers that may have a specific prefix hash.
@@ -218,8 +220,11 @@ func (p *Plugin) PreRequest(ctx context.Context, request *types.LLMRequest, sche
 	// This function is just adding data, it does not need to block other operations.
 	// TODO: look into making this entire function async, none of this needs to be done in-band
 	// The PR that introduces this change is meant as a cherrypick, so it was minimally invasive.
+	// WaitGroup is added to the Plugin struct to allow waiting in tests.
+	p.wg.Add(1)
 	go func() {
 		p.indexer.Add(state.PrefixHashes, ServerID(targetPod.NamespacedName))
+		p.wg.Done()
 	}()
 
 	total := len(state.PrefixHashes)



Test patch:
diff --git a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
index d6ec43c..3fbac2c 100644
--- a/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
+++ b/pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go
@@ -71,6 +71,7 @@ func TestPrefixPlugin(t *testing.T) {
 		},
 	}
 	plugin.PreRequest(context.Background(), req1, schedulingResult, 0)
+	plugin.wg.Wait()
 
 	// Second request doesn't share any prefix with first one. It should be added to the cache but
 	// the pod score should be 0.
@@ -98,6 +99,7 @@ func TestPrefixPlugin(t *testing.T) {
 		},
 	}
 	plugin.PreRequest(context.Background(), req2, schedulingResult, 0)
+	plugin.wg.Wait()
 
 	// Third request shares partial prefix with first one.
 	req3 := &types.LLMRequest{
@@ -123,6 +125,7 @@ func TestPrefixPlugin(t *testing.T) {
 		},
 	}
 	plugin.PreRequest(context.Background(), req3, schedulingResult, 0)
+	plugin.wg.Wait()
 
 	// 4th request is same as req3 except the model is different, still no match.
 	req4 := &types.LLMRequest{
@@ -148,6 +151,7 @@ func TestPrefixPlugin(t *testing.T) {
 		},
 	}
 	plugin.PreRequest(context.Background(), req4, schedulingResult, 0)
+	plugin.wg.Wait()
 
 	// 5th request shares partial prefix with 3rd one.
 	req5 := &types.LLMRequest{
@@ -173,6 +177,7 @@ func TestPrefixPlugin(t *testing.T) {
 		},
 	}
 	plugin.PreRequest(context.Background(), req5, schedulingResult, 0)
+	plugin.wg.Wait()
 }
 
 // TestPrefixPluginStress is a stress test for the prefix scoring plugin, using prompts of increasing length.
@@ -220,6 +225,7 @@ func BenchmarkPrefixPluginStress(b *testing.B) {
 			},
 		}
 		plugin.PreRequest(context.Background(), req, schedulingResult, 0)
+		plugin.wg.Wait()
 
 		// Second cycle: validate internal state
 		state, err := plugins.ReadPluginStateKey[*SchedulingContextState](plugin.pluginState, req.RequestId, plugins.StateKey(plugin.TypedName().String()))