Repository: vllm-project/vllm
Pull request #13950: [Frontend] track server_load
Author: login: daniel-salib
Labels: name: frontend
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
Created: 2025-02-27T07:44:38Z | Merged: 2025-03-14T16:53:18Z

accurate realtime request concurrency tracking. Added the /load api to retrieve the realtime concurrency count.

benchmark_serving.py with the load tracking:
```
============ Serving Benchmark Result ============
Successful requests:                     20000     
Benchmark duration (s):                  581.50    
Total input tokens:                      4516852   
Total generated tokens:                  3751446   
Request throughput (req/s):              34.39     
Output token throughput (tok/s):         6451.36   
Total Token throughput (tok/s):          14218.98  
---------------Time to First Token----------------
Mean TTFT (ms):                          328943.69 
Median TTFT (ms):                        329329.01 
P99 TTFT (ms):                           557303.45 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.84     
Median TPOT (ms):                        31.75     
P99 TPOT (ms):                           77.55     
---------------Inter-token Latency----------------
Mean ITL (ms):                           32.21     
Median ITL (ms):                         23.08     
P99 ITL (ms):                            63.13     
==================================================
```

benchmark_serving.py without the load tracking:
```
============ Serving Benchmark Result ============
Successful requests:                     20000     
Benchmark duration (s):                  604.39    
Total input tokens:                      4516852   
Total generated tokens:                  3748047   
Request throughput (req/s):              33.09     
Output token throughput (tok/s):         6201.37   
Total Token throughput (tok/s):          13674.76  
---------------Time to First Token----------------
Mean TTFT (ms):                          346449.51 
Median TTFT (ms):                        341887.55 
P99 TTFT (ms):                           581491.11 
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.11     
Median TPOT (ms):                        32.22     
P99 TPOT (ms):                           78.94     
---------------Inter-token Latency----------------
Mean ITL (ms):                           32.74     
Median ITL (ms):                         23.24     
P99 ITL (ms):                            65.95     
==================================================
```




Diff:
diff --git a/tests/entrypoints/openai/test_basic.py b/tests/entrypoints/openai/test_basic.py
index e7bf974f13ed..a4ac80070773 100644
--- a/tests/entrypoints/openai/test_basic.py
+++ b/tests/entrypoints/openai/test_basic.py
@@ -171,3 +171,51 @@ async def test_request_wrong_content_type(server: RemoteOpenAIServer):
             extra_headers={
                 "Content-Type": "application/x-www-form-urlencoded"
             })
+
+
+@pytest.mark.parametrize(
+    "server_args",
+    [
+        pytest.param(["--enable-server-load-tracking"],
+                     id="enable-server-load-tracking")
+    ],
+    indirect=True,
+)
+@pytest.mark.asyncio
+async def test_server_load(server: RemoteOpenAIServer):
+    # Check initial server load
+    response = requests.get(server.url_for("load"))
+    assert response.status_code == HTTPStatus.OK
+    assert response.json().get("server_load") == 0
+
+    def make_long_completion_request():
+        return requests.post(
+            server.url_for("v1/completions"),
+            headers={"Content-Type": "application/json"},
+            json={
+                "prompt": "Give me a long story",
+                "max_tokens": 1000,
+                "temperature": 0,
+            },
+        )
+
+    # Start the completion request in a background thread.
+    completion_future = asyncio.create_task(
+        asyncio.to_thread(make_long_completion_request))
+
+    # Give a short delay to ensure the request has started.
+    await asyncio.sleep(0.1)
+
+    # Check server load while the completion request is running.
+    response = requests.get(server.url_for("load"))
+    assert response.status_code == HTTPStatus.OK
+    assert response.json().get("server_load") == 1
+
+    # Wait for the completion request to finish.
+    await completion_future
+    await asyncio.sleep(0.1)
+
+    # Check server load after the completion request has finished.
+    response = requests.get(server.url_for("load"))
+    assert response.status_code == HTTPStatus.OK
+    assert response.json().get("server_load") == 0
diff --git a/vllm/entrypoints/openai/api_server.py b/vllm/entrypoints/openai/api_server.py
index ec2099d4cebf..86d9337c181c 100644
--- a/vllm/entrypoints/openai/api_server.py
+++ b/vllm/entrypoints/openai/api_server.py
@@ -80,7 +80,7 @@
 from vllm.entrypoints.openai.serving_transcription import (
     OpenAIServingTranscription)
 from vllm.entrypoints.openai.tool_parsers import ToolParserManager
-from vllm.entrypoints.utils import with_cancellation
+from vllm.entrypoints.utils import load_aware_call, with_cancellation
 from vllm.logger import init_logger
 from vllm.usage.usage_lib import UsageContext
 from vllm.utils import (FlexibleArgumentParser, get_open_zmq_ipc_path,
@@ -347,6 +347,24 @@ async def health(raw_request: Request) -> Response:
     return Response(status_code=200)
 
 
+@router.get("/load")
+async def get_server_load_metrics(request: Request):
+    # This endpoint returns the current server load metrics.
+    # It tracks requests utilizing the GPU from the following routes:
+    # - /v1/chat/completions
+    # - /v1/completions
+    # - /v1/audio/transcriptions
+    # - /v1/embeddings
+    # - /pooling
+    # - /score
+    # - /v1/score
+    # - /rerank
+    # - /v1/rerank
+    # - /v2/rerank
+    return JSONResponse(
+        content={'server_load': request.app.state.server_load_metrics})
+
+
 @router.api_route("/ping", methods=["GET", "POST"])
 async def ping(raw_request: Request) -> Response:
     """Ping check. Endpoint required for SageMaker"""
@@ -400,6 +418,7 @@ async def show_version():
 @router.post("/v1/chat/completions",
              dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_chat_completion(request: ChatCompletionRequest,
                                  raw_request: Request):
     handler = chat(raw_request)
@@ -421,6 +440,7 @@ async def create_chat_completion(request: ChatCompletionRequest,
 
 @router.post("/v1/completions", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_completion(request: CompletionRequest, raw_request: Request):
     handler = completion(raw_request)
     if handler is None:
@@ -439,6 +459,7 @@ async def create_completion(request: CompletionRequest, raw_request: Request):
 
 @router.post("/v1/embeddings", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_embedding(request: EmbeddingRequest, raw_request: Request):
     handler = embedding(raw_request)
     if handler is None:
@@ -485,6 +506,7 @@ async def create_embedding(request: EmbeddingRequest, raw_request: Request):
 
 @router.post("/pooling", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_pooling(request: PoolingRequest, raw_request: Request):
     handler = pooling(raw_request)
     if handler is None:
@@ -503,6 +525,7 @@ async def create_pooling(request: PoolingRequest, raw_request: Request):
 
 @router.post("/score", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_score(request: ScoreRequest, raw_request: Request):
     handler = score(raw_request)
     if handler is None:
@@ -521,6 +544,7 @@ async def create_score(request: ScoreRequest, raw_request: Request):
 
 @router.post("/v1/score", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def create_score_v1(request: ScoreRequest, raw_request: Request):
     logger.warning(
         "To indicate that Score API is not part of standard OpenAI API, we "
@@ -531,10 +555,10 @@ async def create_score_v1(request: ScoreRequest, raw_request: Request):
 
 @router.post("/v1/audio/transcriptions")
 @with_cancellation
+@load_aware_call
 async def create_transcriptions(request: Annotated[TranscriptionRequest,
                                                    Form()],
                                 raw_request: Request):
-
     handler = transcription(raw_request)
     if handler is None:
         return base(raw_request).create_error_response(
@@ -556,6 +580,7 @@ async def create_transcriptions(request: Annotated[TranscriptionRequest,
 
 @router.post("/rerank", dependencies=[Depends(validate_json_request)])
 @with_cancellation
+@load_aware_call
 async def do_rerank(request: RerankRequest, raw_request: Request):
     handler = rerank(raw_request)
     if handler is None:
@@ -894,6 +919,9 @@ async def init_app_state(
     ) if model_config.runner_type == "transcription" else None
     state.task = model_config.task
 
+    state.enable_server_load_tracking = args.enable_server_load_tracking
+    state.server_load_metrics = 0
+
 
 def create_server_socket(addr: tuple[str, int]) -> socket.socket:
     family = socket.AF_INET
diff --git a/vllm/entrypoints/openai/cli_args.py b/vllm/entrypoints/openai/cli_args.py
index b8cc57430f85..bd66416d90cc 100644
--- a/vllm/entrypoints/openai/cli_args.py
+++ b/vllm/entrypoints/openai/cli_args.py
@@ -257,6 +257,13 @@ def make_arg_parser(parser: FlexibleArgumentParser) -> FlexibleArgumentParser:
         action='store_true',
         default=False,
         help="If set to True, enable prompt_tokens_details in usage.")
+    parser.add_argument(
+        "--enable-server-load-tracking",
+        action='store_true',
+        default=False,
+        help=
+        "If set to True, enable tracking server_load_metrics in the app state."
+    )
 
     return parser
 
diff --git a/vllm/entrypoints/utils.py b/vllm/entrypoints/utils.py
index 9af37871d57c..60cbb58af3d9 100644
--- a/vllm/entrypoints/utils.py
+++ b/vllm/entrypoints/utils.py
@@ -4,6 +4,8 @@
 import functools
 
 from fastapi import Request
+from fastapi.responses import JSONResponse, StreamingResponse
+from starlette.background import BackgroundTask, BackgroundTasks
 
 
 async def listen_for_disconnect(request: Request) -> None:
@@ -17,9 +19,9 @@ async def listen_for_disconnect(request: Request) -> None:
 def with_cancellation(handler_func):
     """Decorator that allows a route handler to be cancelled by client
     disconnections.
-    
+
     This does _not_ use request.is_disconnected, which does not work with
-    middleware. Instead this follows the pattern from 
+    middleware. Instead this follows the pattern from
     starlette.StreamingResponse, which simultaneously awaits on two tasks- one
     to wait for an http disconnect message, and the other to do the work that we
     want done. When the first task finishes, the other is cancelled.
@@ -57,3 +59,45 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def decrement_server_load(request: Request):
+    request.app.state.server_load_metrics -= 1
+
+
+def load_aware_call(func):
+
+    @functools.wraps(func)
+    async def wrapper(*args, raw_request: Request, **kwargs):
+        if not raw_request.app.state.enable_server_load_tracking:
+            return await func(*args, raw_request=raw_request, **kwargs)
+
+        raw_request.app.state.server_load_metrics += 1
+        try:
+            response = await func(*args, raw_request=raw_request, **kwargs)
+        except Exception:
+            raw_request.app.state.server_load_metrics -= 1
+            raise
+
+        if isinstance(response, (JSONResponse, StreamingResponse)):
+            if response.background is None:
+                response.background = BackgroundTask(decrement_server_load,
+                                                     raw_request)
+            elif isinstance(response.background, BackgroundTasks):
+                response.background.add_task(decrement_server_load,
+                                             raw_request)
+            elif isinstance(response.background, BackgroundTask):
+                # Convert the single BackgroundTask to BackgroundTasks
+                # and chain the decrement_server_load task to it
+                tasks = BackgroundTasks()
+                tasks.add_task(response.background.func,
+                               *response.background.args,
+                               **response.background.kwargs)
+                tasks.add_task(decrement_server_load, raw_request)
+                response.background = tasks
+        else:
+            raw_request.app.state.server_load_metrics -= 1
+
+        return response
+
+    return wrapper


Reviews:
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T20:07:53Z
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T20:11:12Z
user: login: russellb
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-27T20:32:06Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T09:19:54Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T09:20:57Z
user: login: youngkent
body: To address the perf concerns of the middleware, could you paste some benchmark data in the PR summary?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T18:42:07Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T12:13:30Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T12:13:55Z
user: login: youngkent
body: Thanks for updating and adding the latency benchmark data!
It seems latency is regressed by ~1% with middleware enabled, which is a bit higher than expected. Can we understand a bit more if the latency is added to TTFT, TPOT, or just added to the end of a http request due to async background task? (benchmark_serving.py should give some more detailed info)
If it's really stat-sig and impacting TTFT, TPOT, we might need to consider exploring a more efficient implementation without using middleware? e.g. Implement increment/decrement directly in API calls in api_server.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T17:45:10Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T05:43:06Z
user: login: robertgshaw2-redhat
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T22:53:19Z
user: login: robertgshaw2-redhat
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T22:53:39Z
user: login: simon-mo
body: I think this is a good approach now. Two questions
1. Please put this behind a feature flag so by default there is no perf regression
2. Please define the semantics of "load", which endpoint does it actually cover? I'm confused on why tokenize endpoint is being covered here as it doesn't hit GPU at all. 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-06T17:42:31Z
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T19:11:28Z
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T22:10:17Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T23:53:36Z
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T00:15:35Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T07:20:41Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T07:22:00Z
user: login: youngkent
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T07:59:17Z
user: login: youngkent
body: One more comment. Otherwise, looks good. Stamping.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T08:00:08Z
user: login: daniel-salib
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-08T08:42:45Z

Review comments:
diff_hunk: @@ -893,6 +900,9 @@ async def init_app_state(
     ) if model_config.runner_type == "transcription" else None
     state.task = model_config.task
 
+    state.concurrent_requests_lock = asyncio.Lock()
+    state.concurrent_requests = 0
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: Since this is just one counter update, I think using regular threading.Lock() might be more efficient than asyncio.Lock, avoiding async context switches.
created_at: 2025-02-27T20:07:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,43 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if request.url.path.startswith("/v1"):
+            async with request.app.state.concurrent_requests_lock:
+                request.app.state.concurrent_requests += 1
+
+            try:
+                response = await call_next(request)
+
+                async def decrement_counter():
+                    async with request.app.state.concurrent_requests_lock:
+                        request.app.state.concurrent_requests -= 1
+
+                def schedule_decrement():
+                    asyncio.create_task(decrement_counter())
+
+                # If no background task exists, attach decrement task
+                if response.background is None:
+                    response.background = BackgroundTask(schedule_decrement)
+                else:
+                    # Chain decrement after the existing background task
+                    original_bg = response.background
+
+                    async def combined_background():
+                        await original_bg()
+                        await decrement_counter()
+
+                    response.background = BackgroundTask(
+                        lambda: asyncio.create_task(combined_background()))
+                return response
+
+            except Exception as exc:
path: vllm/entrypoints/utils.py
user: login: youngkent
body: We could use a finally block to decrement the counter?
created_at: 2025-02-27T20:09:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,43 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if request.url.path.startswith("/v1"):
+            async with request.app.state.concurrent_requests_lock:
+                request.app.state.concurrent_requests += 1
+
+            try:
+                response = await call_next(request)
path: vllm/entrypoints/utils.py
user: login: youngkent
body: How will http streaming request be handled in this case? 
created_at: 2025-02-27T20:10:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -346,6 +347,11 @@ async def health(raw_request: Request) -> Response:
     return Response(status_code=200)
 
 
+@router.get("/load")
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: Can we add a unit test for the /load endpoint?
created_at: 2025-02-27T20:10:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -346,6 +347,11 @@ async def health(raw_request: Request) -> Response:
     return Response(status_code=200)
 
 
+@router.get("/load")
path: vllm/entrypoints/openai/api_server.py
user: login: russellb
body: Why is this a new API endpoint instead of exposing this as a metric in the existing metrics endpoint?
created_at: 2025-02-27T20:32:06Z
author_association: MEMBER
diff_hunk: @@ -57,3 +59,43 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if request.url.path.startswith("/v1"):
+            async with request.app.state.concurrent_requests_lock:
+                request.app.state.concurrent_requests += 1
+
+            try:
+                response = await call_next(request)
path: vllm/entrypoints/utils.py
user: login: daniel-salib
body: from what I can tell from testing, starlette will run the background task only after streaming is complete or when the connection is closed
created_at: 2025-02-28T09:19:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -346,6 +347,11 @@ async def health(raw_request: Request) -> Response:
     return Response(status_code=200)
 
 
+@router.get("/load")
path: vllm/entrypoints/openai/api_server.py
user: login: daniel-salib
body: great suggestion @russellb! I added it as a gauge metric to be returned with the rest of the prometheus metrics in the /metrics route
created_at: 2025-02-28T09:20:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,37 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if not request.url.path.startswith("/v1"):
+            return await call_next(request)
+
+        with request.app.state.server_load_metrics_lock:
+            request.app.state.server_load_metrics += 1
+
+        async def decrement():
+            with request.app.state.server_load_metrics_lock:
+                request.app.state.server_load_metrics -= 1
+
+        response = None
+        exc = None
+        try:
+            response = await call_next(request)
+        except Exception as e:
+            exc = e
+        finally:
+            if response is None:
+                await decrement()
+            else:
+                if response.background is None:
+                    response.background = BackgroundTask(decrement)
+                else:
+                    # Chain decrement after the existing background task
+                    response.background = BackgroundTask(
+                        lambda: asyncio.create_task(decrement()))
+        if exc:
path: vllm/entrypoints/utils.py
user: login: youngkent
body: We could just do try/finally without the extra handling of the exception part?
created_at: 2025-02-28T18:23:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,37 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
path: vllm/entrypoints/utils.py
user: login: youngkent
body: NIT: name it as ServerLoadMiddelware not to be consistent?
created_at: 2025-02-28T18:31:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,37 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if not request.url.path.startswith("/v1"):
+            return await call_next(request)
+
+        with request.app.state.server_load_metrics_lock:
+            request.app.state.server_load_metrics += 1
+
+        async def decrement():
+            with request.app.state.server_load_metrics_lock:
+                request.app.state.server_load_metrics -= 1
+
+        response = None
+        exc = None
+        try:
+            response = await call_next(request)
+        except Exception as e:
+            exc = e
+        finally:
+            if response is None:
+                await decrement()
+            else:
+                if response.background is None:
+                    response.background = BackgroundTask(decrement)
+                else:
+                    # Chain decrement after the existing background task
+                    response.background = BackgroundTask(
path: vllm/entrypoints/utils.py
user: login: youngkent
body: This is not chaining the task? Do you mean
  response.backgroud.add_task(decrement)
?
created_at: 2025-02-28T18:36:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -273,21 +275,28 @@ def mount_metrics(app: FastAPI):
     # We need to set PROMETHEUS_MULTIPROC_DIR environment variable
     # before prometheus_client is imported.
     # See https://prometheus.github.io/client_python/multiprocess/
-    from prometheus_client import (CollectorRegistry, make_asgi_app,
+    from prometheus_client import (CollectorRegistry, Gauge, make_asgi_app,
                                    multiprocess)
 
+    app.state.server_load_metrics_lock = threading.Lock()
+    app.state.server_load_metrics = 0
+
     prometheus_multiproc_dir_path = os.getenv("PROMETHEUS_MULTIPROC_DIR", None)
-    if prometheus_multiproc_dir_path is not None:
+    registry = CollectorRegistry() if prometheus_multiproc_dir_path else None
+    if registry:
         logger.debug("vLLM to use %s as PROMETHEUS_MULTIPROC_DIR",
                      prometheus_multiproc_dir_path)
-        registry = CollectorRegistry()
         multiprocess.MultiProcessCollector(registry)
 
-        # Add prometheus asgi middleware to route /metrics requests
-        metrics_route = Mount("/metrics", make_asgi_app(registry=registry))
-    else:
-        # Add prometheus asgi middleware to route /metrics requests
-        metrics_route = Mount("/metrics", make_asgi_app())
+    # Create the custom metric using the registry if available.
+    server_load = Gauge("server_load", "Current server load",
+                        registry=registry) if registry \
+                  else Gauge("server_load", "Current server load")
+    server_load.set_function(lambda: app.state.server_load_metrics)
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: server load counter might need to be pulled verify frequently to advice load balancing strategy.
Would pulling /metrics endpoint (generate and parsing all metrics) be inefficient and impacting service performance?
Is there a way to just pull one specific prometheus counter, or we should keep the /load endpoint? cc: @russellb 
created_at: 2025-02-28T18:40:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -273,21 +275,28 @@ def mount_metrics(app: FastAPI):
     # We need to set PROMETHEUS_MULTIPROC_DIR environment variable
     # before prometheus_client is imported.
     # See https://prometheus.github.io/client_python/multiprocess/
-    from prometheus_client import (CollectorRegistry, make_asgi_app,
+    from prometheus_client import (CollectorRegistry, Gauge, make_asgi_app,
                                    multiprocess)
 
+    app.state.server_load_metrics_lock = threading.Lock()
+    app.state.server_load_metrics = 0
+
     prometheus_multiproc_dir_path = os.getenv("PROMETHEUS_MULTIPROC_DIR", None)
-    if prometheus_multiproc_dir_path is not None:
+    registry = CollectorRegistry() if prometheus_multiproc_dir_path else None
+    if registry:
         logger.debug("vLLM to use %s as PROMETHEUS_MULTIPROC_DIR",
                      prometheus_multiproc_dir_path)
-        registry = CollectorRegistry()
         multiprocess.MultiProcessCollector(registry)
 
-        # Add prometheus asgi middleware to route /metrics requests
-        metrics_route = Mount("/metrics", make_asgi_app(registry=registry))
-    else:
-        # Add prometheus asgi middleware to route /metrics requests
-        metrics_route = Mount("/metrics", make_asgi_app())
+    # Create the custom metric using the registry if available.
+    server_load = Gauge("server_load", "Current server load",
+                        registry=registry) if registry \
+                  else Gauge("server_load", "Current server load")
+    server_load.set_function(lambda: app.state.server_load_metrics)
path: vllm/entrypoints/openai/api_server.py
user: login: daniel-salib
body: makes sense - for now I went back to the /load route to keep the metric polling as lightweight as possible

thanks!
created_at: 2025-03-03T12:13:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,37 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ConcurrentRequestsMiddleware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
+        if not request.url.path.startswith("/v1"):
+            return await call_next(request)
+
+        with request.app.state.server_load_metrics_lock:
+            request.app.state.server_load_metrics += 1
+
+        async def decrement():
+            with request.app.state.server_load_metrics_lock:
+                request.app.state.server_load_metrics -= 1
+
+        response = None
+        exc = None
+        try:
+            response = await call_next(request)
+        except Exception as e:
+            exc = e
+        finally:
+            if response is None:
+                await decrement()
+            else:
+                if response.background is None:
+                    response.background = BackgroundTask(decrement)
+                else:
+                    # Chain decrement after the existing background task
+                    response.background = BackgroundTask(
path: vllm/entrypoints/utils.py
user: login: daniel-salib
body: good catch - yes meant to use add_task
created_at: 2025-03-03T12:13:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,6 +39,12 @@ async def health() -> Response:
     return Response(status_code=200)
 
 
+@app.get("/load")
+async def get_server_load() -> Response:
+    """Health check."""
+    return Response(status_code=200)
path: vllm/entrypoints/api_server.py
user: login: youngkent
body: Is this endpoint not supported for non-openai server? If so, should we return an error by default?
created_at: 2025-03-03T17:29:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -39,6 +39,12 @@ async def health() -> Response:
     return Response(status_code=200)
 
 
+@app.get("/load")
+async def get_server_load() -> Response:
+    """Health check."""
+    return Response(status_code=200)
path: vllm/entrypoints/api_server.py
user: login: daniel-salib
body: meant to remove this - from the pattern I see its common to have some endpoints such as /ping and /version that are only defined and implemented in the openai server so I followed the pattern for /load and no longer define it in the non-openai server
created_at: 2025-03-04T05:43:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -894,6 +902,9 @@ async def init_app_state(
     ) if model_config.runner_type == "transcription" else None
     state.task = model_config.task
 
+    state.server_load_metrics_lock = threading.Lock()
path: vllm/entrypoints/openai/api_server.py
user: login: robertgshaw2-redhat
body: We do not need locks. This is not threaded code this is asyncio code so only one coroutine can run at a time
created_at: 2025-03-05T22:53:19Z
author_association: COLLABORATOR
diff_hunk: @@ -57,3 +59,31 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+class ServerLoadMiddelware(BaseHTTPMiddleware):
+
+    async def dispatch(self, request, call_next):
path: vllm/entrypoints/utils.py
user: login: robertgshaw2-redhat
body: locks are not needed here
created_at: 2025-03-05T22:53:39Z
author_association: COLLABORATOR
diff_hunk: @@ -266,6 +267,22 @@ async def validate_json_request(raw_request: Request):
         )
 
 
+def increment_server_load(request: Request):
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    request.app.state.server_load_metrics -= 1
+
+
+async def track_server_load_non_streaming(request: Request):
+    increment_server_load(request)
+    try:
+        yield
+    finally:
+        decrement_server_load(request)
path: vllm/entrypoints/openai/api_server.py
user: login: simon-mo
body: the inc and dec can be inlined
created_at: 2025-03-06T17:40:58Z
author_association: COLLABORATOR
diff_hunk: @@ -347,13 +364,23 @@ async def health(raw_request: Request) -> Response:
     return Response(status_code=200)
 
 
+@router.get("/load")
+async def get_server_load_metrics(request: Request):
+    return JSONResponse(
+        content={'server_load': request.app.state.server_load_metrics})
+
+
 @router.api_route("/ping", methods=["GET", "POST"])
 async def ping(raw_request: Request) -> Response:
     """Ping check. Endpoint required for SageMaker"""
     return await health(raw_request)
 
 
-@router.post("/tokenize", dependencies=[Depends(validate_json_request)])
+@router.post("/tokenize",
+             dependencies=[
+                 Depends(validate_json_request),
+                 Depends(track_server_load_non_streaming)
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: Yeah, it might be better to NOT count tokenizer as the load.
created_at: 2025-03-06T19:05:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -402,42 +433,66 @@ async def show_version():
 @with_cancellation
 async def create_chat_completion(request: ChatCompletionRequest,
                                  raw_request: Request):
+    increment_server_load(raw_request)
+
     handler = chat(raw_request)
     if handler is None:
+        decrement_server_load(raw_request)
         return base(raw_request).create_error_response(
             message="The model does not support Chat Completions API")
 
     generator = await handler.create_chat_completion(request, raw_request)
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: If create_chat_completion throw exception, it would not decrement server load?
created_at: 2025-03-06T19:11:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -402,42 +433,66 @@ async def show_version():
 @with_cancellation
 async def create_chat_completion(request: ChatCompletionRequest,
                                  raw_request: Request):
+    increment_server_load(raw_request)
+
     handler = chat(raw_request)
     if handler is None:
+        decrement_server_load(raw_request)
         return base(raw_request).create_error_response(
             message="The model does not support Chat Completions API")
 
     generator = await handler.create_chat_completion(request, raw_request)
path: vllm/entrypoints/openai/api_server.py
user: login: daniel-salib
body: yea good catch. I added a dep for the request to decrement on any exception now
created_at: 2025-03-06T19:58:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -398,46 +454,91 @@ async def show_version():
 
 
 @router.post("/v1/chat/completions",
-             dependencies=[Depends(validate_json_request)])
+             dependencies=[
+                 Depends(validate_json_request),
+                 Depends(handle_streaming_request_exceptions)
+             ])
 @with_cancellation
 async def create_chat_completion(request: ChatCompletionRequest,
                                  raw_request: Request):
+    increment_server_load(raw_request)
     handler = chat(raw_request)
     if handler is None:
+        decrement_server_load(raw_request)
         return base(raw_request).create_error_response(
             message="The model does not support Chat Completions API")
 
     generator = await handler.create_chat_completion(request, raw_request)
 
     if isinstance(generator, ErrorResponse):
+        if raw_request.app.state.enable_server_load_tracking:
path: vllm/entrypoints/openai/api_server.py
user: login: youngkent
body: The code is a bit more complex to read now. How about we just make a wrapper function like

```
async def create_chat_completion(...):
    if not raw_request.app.state.enable_server_load_tracking:
        return create_chat_completion(...)
    else: 
        increment_server_load(raw_request)
        try:
            response = create_chat_completion(...)
            response.background = BackgroundTask(...)
            return response
        except:
            decrement_server_load(raw_request)
```

We can further make the above common logic a method like load_aware_call(http_method, ...)
created_at: 2025-03-07T22:09:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -398,46 +454,91 @@ async def show_version():
 
 
 @router.post("/v1/chat/completions",
-             dependencies=[Depends(validate_json_request)])
+             dependencies=[
+                 Depends(validate_json_request),
+                 Depends(handle_streaming_request_exceptions)
+             ])
 @with_cancellation
 async def create_chat_completion(request: ChatCompletionRequest,
                                  raw_request: Request):
+    increment_server_load(raw_request)
     handler = chat(raw_request)
     if handler is None:
+        decrement_server_load(raw_request)
         return base(raw_request).create_error_response(
             message="The model does not support Chat Completions API")
 
     generator = await handler.create_chat_completion(request, raw_request)
 
     if isinstance(generator, ErrorResponse):
+        if raw_request.app.state.enable_server_load_tracking:
path: vllm/entrypoints/openai/api_server.py
user: login: daniel-salib
body: yea looks much cleaner - I made another pass with "load_aware_streaming_call" implemented. I only applied it to steaming calls because I thought it might be unnecessary for the non streaming requests. The non streaming calls still use the dependency injection.

LMK how it looks and if its better for everything to use load_aware_call

thanks!


created_at: 2025-03-07T23:52:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,50 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def increment_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
path: vllm/entrypoints/utils.py
user: login: youngkent
body: Not needed anymore?
created_at: 2025-03-07T23:57:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,50 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def increment_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics -= 1
+
+
+async def track_server_load_non_streaming(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        yield None
+        return
+
+    request.app.state.server_load_metrics += 1
+    try:
+        yield
+    finally:
+        request.app.state.server_load_metrics -= 1
+
+
+def load_aware_streaming_call(func):
+
+    async def wrapper(*args, raw_request: Request, **kwargs):
+        if not raw_request.app.state.enable_server_load_tracking:
+            return await func(*args, raw_request=raw_request, **kwargs)
+
+        increment_server_load(raw_request)
+        try:
+            response = await func(*args, raw_request=raw_request, **kwargs)
+            if isinstance(response, (JSONResponse, StreamingResponse)):
+                response.background = BackgroundTask(decrement_server_load,
path: vllm/entrypoints/utils.py
user: login: youngkent
body: For safety, add assertion that response.background is none? 
created_at: 2025-03-08T00:00:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,50 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def increment_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics -= 1
+
+
+async def track_server_load_non_streaming(request: Request):
path: vllm/entrypoints/utils.py
user: login: youngkent
body: Wondering if we could unify the track_server_load_non_streaming and load_aware_streaming_call into a "@load_aware" annotation, put in front of the http methods?
created_at: 2025-03-08T00:15:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,50 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def increment_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics -= 1
+
+
+async def track_server_load_non_streaming(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        yield None
+        return
+
+    request.app.state.server_load_metrics += 1
+    try:
+        yield
+    finally:
+        request.app.state.server_load_metrics -= 1
+
+
+def load_aware_streaming_call(func):
+
+    async def wrapper(*args, raw_request: Request, **kwargs):
+        if not raw_request.app.state.enable_server_load_tracking:
+            return await func(*args, raw_request=raw_request, **kwargs)
+
+        increment_server_load(raw_request)
+        try:
+            response = await func(*args, raw_request=raw_request, **kwargs)
+            if isinstance(response, (JSONResponse, StreamingResponse)):
+                response.background = BackgroundTask(decrement_server_load,
path: vllm/entrypoints/utils.py
user: login: daniel-salib
body: I went ahead and handled both cases with and without a pre-existing bg task by adding a task if it already exists
created_at: 2025-03-08T07:20:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,50 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def increment_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics += 1
+
+
+def decrement_server_load(request: Request):
+    if not request.app.state.enable_server_load_tracking:
+        return
+    request.app.state.server_load_metrics -= 1
+
+
+async def track_server_load_non_streaming(request: Request):
path: vllm/entrypoints/utils.py
user: login: daniel-salib
body: sounds good - I implemented as a decorator now and removed the dependency injection
created_at: 2025-03-08T07:22:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,36 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def decrement_server_load(request: Request):
+    request.app.state.server_load_metrics -= 1
+
+
+def load_aware_call(func):
+
+    @functools.wraps(func)
+    async def wrapper(*args, raw_request: Request, **kwargs):
+        if not raw_request.app.state.enable_server_load_tracking:
+            return await func(*args, raw_request=raw_request, **kwargs)
+
+        raw_request.app.state.server_load_metrics += 1
+        try:
+            response = await func(*args, raw_request=raw_request, **kwargs)
+        except Exception:
+            raw_request.app.state.server_load_metrics -= 1
+            raise
+
+        if isinstance(response, (JSONResponse, StreamingResponse)):
+            if response.background is None:
+                response.background = BackgroundTask(decrement_server_load,
+                                                     raw_request)
+            else:
+                response.background.add_task(decrement_server_load,
path: vllm/entrypoints/utils.py
user: login: youngkent
body: The BackgroundTask type does not have add_task method? https://github.com/encode/starlette/blob/49158732f8a96744889f5a7644a42ae08131bd15/starlette/background.py#L17
created_at: 2025-03-08T07:59:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -57,3 +59,36 @@ async def wrapper(*args, **kwargs):
         return None
 
     return wrapper
+
+
+def decrement_server_load(request: Request):
+    request.app.state.server_load_metrics -= 1
+
+
+def load_aware_call(func):
+
+    @functools.wraps(func)
+    async def wrapper(*args, raw_request: Request, **kwargs):
+        if not raw_request.app.state.enable_server_load_tracking:
+            return await func(*args, raw_request=raw_request, **kwargs)
+
+        raw_request.app.state.server_load_metrics += 1
+        try:
+            response = await func(*args, raw_request=raw_request, **kwargs)
+        except Exception:
+            raw_request.app.state.server_load_metrics -= 1
+            raise
+
+        if isinstance(response, (JSONResponse, StreamingResponse)):
+            if response.background is None:
+                response.background = BackgroundTask(decrement_server_load,
+                                                     raw_request)
+            else:
+                response.background.add_task(decrement_server_load,
path: vllm/entrypoints/utils.py
user: login: daniel-salib
body: good catch - I added handling to convert to from a single BackgroundTask -> BackgroundTasks which will us to chain multiple tasks together when needed. Did manual testing locally to confirm its working
created_at: 2025-03-08T08:42:45Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-02-27T07:44:50Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: simon-mo
created_at: 2025-02-27T18:32:22Z
body: Middleware does looks cleaner to me. Please rename the metrics to `server_load_metrics` to be more readable. I also want to (1) clarify the performance impact (2) make sure this can work when we run multiple frontend processes (@russellb @njhill) 
author_association: COLLABORATOR
user: login: russellb
created_at: 2025-02-27T20:07:22Z
body: This won't work as-is with multiple frontend processes (API servers), but this also won't be the only metric we have to fix. It's noted as one of the challenges, though we know the expected solution. It's mentioned in the design doc linked from https://github.com/vllm-project/vllm/issues/12705


author_association: MEMBER
user: login: njhill
created_at: 2025-02-27T21:32:22Z
body: +1 on making sure there's no performance impact, we've had some nasty middleware-related performance issues before.
author_association: MEMBER
user: login: daniel-salib
created_at: 2025-02-28T09:21:43Z
body: Thanks for the reviews :) Made another pass taking all the feedback into consideration
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-03T12:58:47Z
body: thanks for the review @youngkent 

 resolved all the comments and added a unit test for the /load route.
 
 Also attached the benchmark results showing the latency comparison before and after the middleware.
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-04T05:41:04Z
body: @youngkent ran benchmark_serving.py twice with and without the middleware and added to the PR description.

Is it safe to assume the discrepancy between the runs is due to random +/- error and the middleware's impact is negligible? 
author_association: CONTRIBUTOR
user: login: youngkent
created_at: 2025-03-04T17:09:06Z
body: @daniel-salib Seems the benchmark_serving.py data have wide variance, hard to tell if it's really stat-sig. We might want to increase the sample size, meaning more requests per control and test group. Can we test with 20000 requests per group?

author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-04T20:57:44Z
body: @youngkent  updated the PR description with the benchmark comparison across 20,000 requests per group
author_association: CONTRIBUTOR
user: login: youngkent
created_at: 2025-03-05T00:31:22Z
body: Thanks @daniel-salib, the variance is still a bit high, but at least there is no conclusive signal showing middleware regress the performance so far.
@simon-mo Do you want to double check the data and see if this implementation looks good to you? thanks
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-05T01:17:07Z
body: @youngkent 

I was previously using the random dataset when benchmarking - thought that may have an effect on the variance.

I updated the description with the results after I switched to --dataset-name=sharegpt --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json but still see some variance
author_association: CONTRIBUTOR
user: login: robertgshaw2-redhat
created_at: 2025-03-05T22:52:00Z
body: What type of GPU is the performance test run on?
author_association: COLLABORATOR
user: login: daniel-salib
created_at: 2025-03-05T23:09:27Z
body: Thanks for the review @robertgshaw2-redhat 

I ran the performance test on 2 x H100 GPU
author_association: CONTRIBUTOR
user: login: simon-mo
created_at: 2025-03-06T00:49:21Z
body: The serving throughput degradation is pretty serious, almost a 3% drop. We cannot merge this in as-is due to the performance impact. I added arbitrary middleware support here https://github.com/vllm-project/vllm/blob/a7ea35aa677272d36e01388f26801d673cbb680e/vllm/entrypoints/openai/cli_args.py#L185 if you really need it or alternatively figure out a better way 
author_association: COLLABORATOR
user: login: daniel-salib
created_at: 2025-03-06T11:55:38Z
body: @youngkent took a different approach that should be much better performance-wise. Updated the description to include the latest benchmarks
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-07T19:34:05Z
body: @youngkent  handling exceptions on streaming requests now - PTAL 
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-08T07:22:42Z
body: implemented @youngkent 's latest suggestions

also re-ran the benchmark and updated the PR comment to ensure no regression
author_association: CONTRIBUTOR
user: login: simon-mo
created_at: 2025-03-11T19:37:21Z
body: Pre-commit (i.e. lint) is failing. 
author_association: COLLABORATOR
user: login: daniel-salib
created_at: 2025-03-11T21:29:19Z
body: @simon-mo  rebased and the precommit is passing now
author_association: CONTRIBUTOR
user: login: daniel-salib
created_at: 2025-03-12T23:43:22Z
body: @simon-mo I see some failures in the entrypoints test but they seem unrelated to the PR:

I think we're good to merge but LMK if this is an issue. Thanks!

`
[2025-03-11T23:16:51Z] =========================== short test summary info ============================
--
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[-True] - AssertionError: vllm:iteration_tokens_total_sum expected value of 200 did not match found value 219.0
  | [2025-03-11T23:16:51Z] assert 219.0 == 200
  | [2025-03-11T23:16:51Z]  +  where 219.0 = Sample(name='vllm:iteration_tokens_total_sum', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=219.0, timestamp=None, exemplar=None).value
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[-False] - AssertionError: vllm:time_to_first_token_seconds_count expected value of 10 did not match found value 11.0
  | [2025-03-11T23:16:51Z] assert 11.0 == 10
  | [2025-03-11T23:16:51Z]  +  where 11.0 = Sample(name='vllm:time_to_first_token_seconds_count', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=11.0, timestamp=None, exemplar=None).value
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[--enable-chunked-prefill-False] - AssertionError: vllm:time_to_first_token_seconds_count expected value of 10 did not match found value 11.0
  | [2025-03-11T23:16:51Z] assert 11.0 == 10
  | [2025-03-11T23:16:51Z]  +  where 11.0 = Sample(name='vllm:time_to_first_token_seconds_count', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=11.0, timestamp=None, exemplar=None).value
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[--enable-chunked-prefill-True] - AssertionError: vllm:iteration_tokens_total_sum expected value of 200 did not match found value 219.0
  | [2025-03-11T23:16:51Z] assert 219.0 == 200
  | [2025-03-11T23:16:51Z]  +  where 219.0 = Sample(name='vllm:iteration_tokens_total_sum', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=219.0, timestamp=None, exemplar=None).value
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[--disable-frontend-multiprocessing-False] - AssertionError: vllm:time_to_first_token_seconds_count expected value of 10 did not match found value 11.0
  | [2025-03-11T23:16:51Z] assert 11.0 == 10
  | [2025-03-11T23:16:51Z]  +  where 11.0 = Sample(name='vllm:time_to_first_token_seconds_count', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=11.0, timestamp=None, exemplar=None).value
  | [2025-03-11T23:16:51Z] FAILED entrypoints/openai/test_metrics.py::test_metrics_counts[--disable-frontend-multiprocessing-True] - AssertionError: vllm:iteration_tokens_total_sum expected value of 200 did not match found value 219.0
  | [2025-03-11T23:16:51Z] assert 219.0 == 200
  | [2025-03-11T23:16:51Z]  +  where 219.0 = Sample(name='vllm:iteration_tokens_total_sum', labels={'model_name': 'TinyLlama/TinyLlama-1.1B-Chat-v1.0'}, value=219.0, timestamp=None, exemplar=None).value

`
author_association: CONTRIBUTOR
user: login: DarkLight1337
created_at: 2025-03-14T04:00:17Z
body: I think the entrypoints failures are related to this PR because they're not failing on main, please fix them.
author_association: MEMBER
user: login: daniel-salib
created_at: 2025-03-14T11:55:32Z
body: ah I found the issue, adding the extra unit test to test_metrics messed up the expected metric numbers in metric counts because the tests run on the same server instance. Decided it would be best to keep the metrics test just for prometheus metrics and felt is may be more appropriate to test the server_load arg in test_basic
author_association: CONTRIBUTOR