Repository: vllm-project/vllm
Pull request: [BugFix] Fix server crash on empty prompt
Fixes https://github.com/vllm-project/vllm/issues/7632

To reproduce, start the server with `python -m vllm.entrypoints.openai.api_server --model gpt2` and send an empty prompt:

```
$ curl http://localhost:8000/v1/completions    -H "Content-Type: application/json"    -d '{
     "model": "gpt2",
     "prompt": [""],
     "max_tokens": 20,
     "temperature": 0
}'
Internal Server Error
```

On the server side this log will show and the server will be dead:

```
INFO 08-21 14:49:19 logger.py:36] Received request cmpl-470c7a46582b4554b7926ce4559b0337-0: prompt: '', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, use_beam_search=False, length_penalty=1.0, early_stopping=False, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=20, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), prompt_token_ids: [], lora_request: None, prompt_adapter_request: None.
INFO 08-21 14:49:19 async_llm_engine.py:208] Added request cmpl-470c7a46582b4554b7926ce4559b0337-0.
ERROR 08-21 14:49:19 async_llm_engine.py:65] Engine background task failed
ERROR 08-21 14:49:19 async_llm_engine.py:65] Traceback (most recent call last):
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
ERROR 08-21 14:49:19 async_llm_engine.py:65]     return_value = task.result()
ERROR 08-21 14:49:19 async_llm_engine.py:65]                    ^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 930, in run_engine_loop
ERROR 08-21 14:49:19 async_llm_engine.py:65]     result = task.result()
ERROR 08-21 14:49:19 async_llm_engine.py:65]              ^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 873, in engine_step
ERROR 08-21 14:49:19 async_llm_engine.py:65]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 08-21 14:49:19 async_llm_engine.py:65]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 301, in step_async
ERROR 08-21 14:49:19 async_llm_engine.py:65]     virtual_engine].schedule()
ERROR 08-21 14:49:19 async_llm_engine.py:65]                     ^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1039, in schedule
ERROR 08-21 14:49:19 async_llm_engine.py:65]     scheduler_outputs = self._schedule()
ERROR 08-21 14:49:19 async_llm_engine.py:65]                         ^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1013, in _schedule
ERROR 08-21 14:49:19 async_llm_engine.py:65]     return self._schedule_default()
ERROR 08-21 14:49:19 async_llm_engine.py:65]            ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 857, in _schedule_default
ERROR 08-21 14:49:19 async_llm_engine.py:65]     prefills = self._schedule_prefills(budget,
ERROR 08-21 14:49:19 async_llm_engine.py:65]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 752, in _schedule_prefills
ERROR 08-21 14:49:19 async_llm_engine.py:65]     num_new_tokens = self._get_num_new_tokens(seq_group,
ERROR 08-21 14:49:19 async_llm_engine.py:65]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65]   File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1349, in _get_num_new_tokens
ERROR 08-21 14:49:19 async_llm_engine.py:65]     assert num_new_tokens > 0
ERROR 08-21 14:49:19 async_llm_engine.py:65]            ^^^^^^^^^^^^^^^^^^
ERROR 08-21 14:49:19 async_llm_engine.py:65] AssertionError
Exception in callback functools.partial(<function _log_task_completion at 0x7f1abf494220>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f1aa38d0050>>)
handle: <Handle functools.partial(<function _log_task_completion at 0x7f1abf494220>, error_callback=<bound method AsyncLLMEngine._error_callback of <vllm.engine.async_llm_engine.AsyncLLMEngine object at 0x7f1aa38d0050>>)>
Traceback (most recent call last):
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 55, in _log_task_completion
    return_value = task.result()
                   ^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 930, in run_engine_loop
    result = task.result()
             ^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 873, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 301, in step_async
    virtual_engine].schedule()
                    ^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1039, in schedule
    scheduler_outputs = self._schedule()
                        ^^^^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1013, in _schedule
    return self._schedule_default()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 857, in _schedule_default
    prefills = self._schedule_prefills(budget,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 752, in _schedule_prefills
    num_new_tokens = self._get_num_new_tokens(seq_group,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/core/scheduler.py", line 1349, in _get_num_new_tokens
    assert num_new_tokens > 0
           ^^^^^^^^^^^^^^^^^^
AssertionError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
  File "/home/mbayser/IBMProjects/FoundationModels/inference/vllm/vllm/engine/async_llm_engine.py", line 67, in _log_task_completion
    raise AsyncEngineDeadError(
```

After the fix, an error 400 is returned:
```
$ curl http://localhost:8000/v1/completions    -H "Content-Type: application/json"    -d '{
     "model": "gpt2",
     "prompt": [""],
     "max_tokens": 20,
     "temperature": 0
}'
{"object":"error","message":"Empty prompt","type":"BadRequestError","param":null,"code":400}```

Problem statement:
[Bug]: assert num_new_tokens > 0 crashes entire worker instead of just failing single API call
### Your current environment

vllm docker 0.5.4

```
docker pull vllm/vllm-openai:latest
docker stop danube3_mig ; docker remove danube3_mig
docker run -d --restart=always \
    --runtime=nvidia \
    --gpus '"device=MIG-a6dbed35-9d05-58da-a0b5-23ae5bf8427e"' \
    --shm-size=10.24gb \
    -p 5004:5004 \
    -e NCCL_IGNORE_DISABLED_P2P=1 \
    -e HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN \
    -e VLLM_NCCL_SO_PATH=/usr/local/lib/python3.10/dist-packages/nvidia/nccl/lib/libnccl.so.2 \
    -v /etc/passwd:/etc/passwd:ro \
    -v /etc/group:/etc/group:ro \
    -u `id -u`:`id -g` \
    -v "${HOME}"/.cache:$HOME/.cache/ -v "${HOME}"/.config:$HOME/.config/   -v "${HOME}"/.triton:$HOME/.triton/  \
    --network host \
    --name danube3_mig \
    vllm/vllm-openai:latest \
        --port=5004 \
        --host=0.0.0.0 \
        --model=h2oai/h2o-danube3-4b-chat \
        --seed 1234 \
        --trust-remote-code \
        --tensor-parallel-size=1 \
        --max-model-len=8192 \
        --gpu-memory-utilization=0.99 \
        --max-num-batched-tokens=131072 --max-log-len=100 \
        --use-v2-block-manager \
        --num-speculative-tokens=5 \
        --ngram-prompt-lookup-max=4 \
        --enable-prefix-caching \
        --speculative-model="[ngram]" \
        --download-dir=$HOME/.cache/huggingface/hub &>> logs.vllm_server.danube3_migb.txt
```

Unsure if has to do with speculative, seems just like prompt='' causes it.

### 🐛 Describe the bug

```
INFO:     172.16.0.199:21756 - "GET /health HTTP/1.1" 200 OK
INFO 08-18 00:51:02 logger.py:36] Received request cmpl-14b87b97d9a8481d8963a0a1652b217b-0: prompt: '', params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.3, top_p=1.>
INFO:     172.16.0.199:21766 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 08-18 00:51:02 async_llm_engine.py:174] Added request cmpl-14b87b97d9a8481d8963a0a1652b217b-0.
ERROR 08-18 00:51:02 async_llm_engine.py:57] Engine background task failed
ERROR 08-18 00:51:02 async_llm_engine.py:57] Traceback (most recent call last):
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 47, in _log_task_completion
ERROR 08-18 00:51:02 async_llm_engine.py:57]     return_value = task.result()
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 642, in run_engine_loop
ERROR 08-18 00:51:02 async_llm_engine.py:57]     result = task.result()
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 585, in engine_step
ERROR 08-18 00:51:02 async_llm_engine.py:57]     request_outputs = await self.engine.step_async(virtual_engine)
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 239, in step_async
ERROR 08-18 00:51:02 async_llm_engine.py:57]     virtual_engine].schedule()
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 950, in schedule
ERROR 08-18 00:51:02 async_llm_engine.py:57]     scheduler_outputs = self._schedule()
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 925, in _schedule
ERROR 08-18 00:51:02 async_llm_engine.py:57]     return self._schedule_default()
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 785, in _schedule_default
ERROR 08-18 00:51:02 async_llm_engine.py:57]     prefills = self._schedule_prefills(budget,
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 683, in _schedule_prefills
ERROR 08-18 00:51:02 async_llm_engine.py:57]     num_new_tokens = self._get_num_new_tokens(seq_group,
ERROR 08-18 00:51:02 async_llm_engine.py:57]   File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 1206, in _get_num_new_tokens
ERROR 08-18 00:51:02 async_llm_engine.py:57]     assert num_new_tokens > 0
ERROR 08-18 00:51:02 async_llm_engine.py:57] AssertionError
Exception in callback _log_task_completion(error_callback=<bound method...72047c646d70>>)(<Task finishe...ertionError()>) at /usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py:37
handle: <Handle _log_task_completion(error_callback=<bound method...72047c646d70>>)(<Task finishe...ertionError()>) at /usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py:37>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 47, in _log_task_completion
    return_value = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 642, in run_engine_loop
    result = task.result()
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 585, in engine_step
    request_outputs = await self.engine.step_async(virtual_engine)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 239, in step_async
    virtual_engine].schedule()
  File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 950, in schedule
    scheduler_outputs = self._schedule()
  File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 925, in _schedule
    return self._schedule_default()
  File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 785, in _schedule_default
    prefills = self._schedule_prefills(budget,
INFO 08-18 00:51:02 async_llm_engine.py:181] Aborted request cmpl-14b87b97d9a8481d8963a0a1652b217b-0.
  File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 683, in _schedule_prefills
    num_new_tokens = self._get_num_new_tokens(seq_group,
  File "/usr/local/lib/python3.10/dist-packages/vllm/core/scheduler.py", line 1206, in _get_num_new_tokens
    assert num_new_tokens > 0
AssertionError


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 59, in _log_task_completion
    raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 265, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 261, in wrap
    await func()
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 238, in listen_for_disconnect
    message = await receive()
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 54, in wrapped_receive
    msg = await self.receive()
  File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 553, in receive
    await self.message_event.wait()
  File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 720537665120

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 192, in __call__
    await response(scope, wrapped_receive, send)
  File "/usr/local/lib/python3.10/dist-packages/starlette/responses.py", line 258, in __call__
    async with anyio.create_task_group() as task_group:
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
    raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/starlette/_utils.py", line 87, in collapse_excgroups
    yield
  File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/base.py", line 190, in __call__
    async with anyio.create_task_group() as task_group:
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 680, in __aexit__
    raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

```

Hints from issue discussion:
To be clear, the bug is at least that the entire vllm engine is taken down by prompt=''

Commit message: move test to another file due to conflicting fixtures

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Patch to review:
diff --git a/tests/entrypoints/llm/test_generate.py b/tests/entrypoints/llm/test_generate.py
index c426e9b4ee8..9df11df98aa 100644
--- a/tests/entrypoints/llm/test_generate.py
+++ b/tests/entrypoints/llm/test_generate.py
@@ -142,6 +142,12 @@ def test_multiple_sampling_params(llm: LLM):
     assert len(PROMPTS) == len(outputs)
 
 
+def test_empty_prompt():
+    llm = LLM(model="gpt2")
+    with pytest.raises(ValueError, match='Empty prompt'):
+        llm.generate([""])
+
+
 def test_chat():
 
     llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
diff --git a/tests/entrypoints/openai/test_prompt_validation.py b/tests/entrypoints/openai/test_prompt_validation.py
new file mode 100644
index 00000000000..a63c693c21c
--- /dev/null
+++ b/tests/entrypoints/openai/test_prompt_validation.py
@@ -0,0 +1,22 @@
+# imports for guided decoding tests
+import re
+
+import openai
+import pytest
+
+from ...utils import RemoteOpenAIServer
+
+
+@pytest.mark.asyncio
+async def test_empty_prompt():
+    model_name = "gpt2"
+    server_args = ["--disable-frontend-multiprocessing", "--enforce-eager"]
+    with RemoteOpenAIServer(model_name, server_args) as remote_server:
+        client = remote_server.get_async_client()
+
+        with pytest.raises(openai.BadRequestError,
+                           match=re.compile('.+Empty prompt.+')):
+            await client.completions.create(model=model_name,
+                                            prompt="",
+                                            max_tokens=5,
+                                            temperature=0.0)
\ No newline at end of file
diff --git a/vllm/engine/async_llm_engine.py b/vllm/engine/async_llm_engine.py
index 8812b853c06..d77fb96b943 100644
--- a/vllm/engine/async_llm_engine.py
+++ b/vllm/engine/async_llm_engine.py
@@ -581,6 +581,7 @@ async def add_request_async(
             lora_request=lora_request,
             prompt_adapter_request=prompt_adapter_request,
         )
+        self._validate_model_inputs(processed_inputs)
 
         self._add_processed_request(
             request_id=request_id,
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
index f72902c3721..fd9445eaff0 100644
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -900,10 +900,8 @@ def _build_decoder_only_llm_inputs(
         prompt_adapter_request: Optional[PromptAdapterRequest],
     ) -> LLMInputs:
         prompt, prompt_token_ids, multi_modal_data = prompt_comps
-
         prompt_token_ids = self._apply_prompt_adapter(
             prompt_token_ids, prompt_adapter_request=prompt_adapter_request)
-
         return LLMInputs(prompt_token_ids=prompt_token_ids,
                          prompt=prompt,
                          multi_modal_data=multi_modal_data)
@@ -1036,6 +1034,7 @@ def add_request(
             lora_request=lora_request,
             prompt_adapter_request=prompt_adapter_request,
         )
+        self._validate_model_inputs(processed_inputs)
 
         self._add_processed_request(
             request_id=request_id,
@@ -1647,3 +1646,14 @@ def is_encoder_decoder_model(self):
 
     def is_embedding_model(self):
         return self.model_config.is_embedding_model
+
+    def _validate_model_inputs(self, inputs: Union[LLMInputs,
+                                                   EncoderDecoderLLMInputs]):
+        if self.is_encoder_decoder_model():
+            if "encoder_prompt_token_ids" not in inputs or\
+                len(inputs["encoder_prompt_token_ids"]) == 0:
+                raise ValueError("Empty prompt")
+        else:
+            if "prompt_token_ids" not in inputs or len(
+                    inputs["prompt_token_ids"]) == 0:
+                raise ValueError("Empty prompt")


Code review comments:
diff_hunk: @@ -0,0 +1,22 @@
+# imports for guided decoding tests
+import re
+
+import openai
+import pytest
+
+from ...utils import RemoteOpenAIServer
+
+
+@pytest.mark.asyncio
+async def test_empty_prompt():
+    model_name = "gpt2"
+    server_args = ["--disable-frontend-multiprocessing", "--enforce-eager"]
path: tests/entrypoints/openai/test_prompt_validation.py
text: @user1:
Do you need to disable the front end multiprocessing? It would be nice to test through that too

@author:
No, it was just to reduce the number of moving pieces. But I've enabled it again.