Repository: vllm-project/vllm

Issue:
AssertionError in LLM.generate from version 0.2.1
I've only tested this using Zephyr (mistral-type model), but when calling LLM.generate in version 0.2.1, I receive an AssertionError in the worker runs.

Code to reproduce:
```py
from vllm import LLM, SamplingParams

llm = LLM("HuggingFaceH4/zephyr-7b-alpha", tensor_parallel_size=4)

sampling_params = SamplingParams(
    temperature=0.1,
    top_p=0.95,
    max_tokens=400
)
output = llm.generate(["Hello, how are you?", "Tell me a joke!"], sampling_params=sampling_params)
```
Same code block works fine in 0.2.0.

Full traceback in version 0.2.1:
```
AssertionError                            Traceback (most recent call last)
File <command-2469377940857761>, line 10
      3 llm = LLM("HuggingFaceH4/zephyr-7b-alpha", tensor_parallel_size=4)
      5 sampling_params = SamplingParams(
      6     temperature=0.1,
      7     top_p=0.95,
      8     max_tokens=400
      9 )
---> 10 output = llm.generate(["Hello, how are you?", "Tell me a joke!"], sampling_params=sampling_params)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-f955471c-8346-4f89-b3b6-d25bfb1d27ae/lib/python3.10/site-packages/vllm/entrypoints/llm.py:157, in LLM.generate(self, prompts, sampling_params, prompt_token_ids, use_tqdm)
    155         token_ids = prompt_token_ids[i]
    156     self._add_request(prompt, sampling_params, token_ids)
--> 157 return self._run_engine(use_tqdm)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-f955471c-8346-4f89-b3b6-d25bfb1d27ae/lib/python3.10/site-packages/vllm/entrypoints/llm.py:177, in LLM._run_engine(self, use_tqdm)
    175 outputs: List[RequestOutput] = []
    176 while self.llm_engine.has_unfinished_requests():
--> 177     step_outputs = self.llm_engine.step()
    178     for output in step_outputs:
    179         if output.finished:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-f955471c-8346-4f89-b3b6-d25bfb1d27ae/lib/python3.10/site-packages/vllm/engine/llm_engine.py:562, in LLMEngine.step(self)
    559     return ignored
    561 # Execute the model.
--> 562 output = self._run_workers(
    563     "execute_model",
    564     seq_group_metadata_list=seq_group_metadata_list,
    565     blocks_to_swap_in=scheduler_outputs.blocks_to_swap_in,
    566     blocks_to_swap_out=scheduler_outputs.blocks_to_swap_out,
    567     blocks_to_copy=scheduler_outputs.blocks_to_copy,
    568 )
    570 return self._process_model_outputs(output, scheduler_outputs) + ignored

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-f955471c-8346-4f89-b3b6-d25bfb1d27ae/lib/python3.10/site-packages/vllm/engine/llm_engine.py:712, in LLMEngine._run_workers(self, method, get_all_outputs, *args, **kwargs)
    710 output = all_outputs[0]
    711 for other_output in all_outputs[1:]:
--> 712     assert output == other_output
    713 return output

AssertionError: 
```
AsyncEngineDeadError: Task finished unexpectedly
Hey! I've been facing the following error with the latest version of vllm. The stacktrace is below: 

> INFO:     127.0.0.1:52238 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 28, in _raise_exception_on_finish
task.result()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 351, in run_engine_loop
has_requests_in_progress = await self.engine_step()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 330, in engine_step
request_outputs = await self.engine.step_async()
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 191, in step_async
output = await self._run_workers_async(
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 228, in _run_workers_async
assert output == other_output
AssertionError

> The above exception was the direct cause of the following exception:

> Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app(  # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 292, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
raise e
File "/usr/local/lib/python3.10/dist-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 273, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 190, in run_endpoint_function
return await dependant.call(**values)
File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/api_server.py", line 52, in generate
async for request_output in results_generator:
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 436, in generate
raise e
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 430, in generate
async for request_output in stream:
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 70, in __anext__
raise result
File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 37, in _raise_exception_on_finish
raise exc
File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 32, in _raise_exception_on_finish
raise AsyncEngineDeadError(
vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for the actual cause. 

I am using vLLM inside Docker, CUDA version is "11.8.0", torch version is 2.0.1+cu118 



Discussion:

I am seeing this too.
same issue
The same
Hi @rinapch thanks for reporting the bug. Could you provide a reproducible script?
same error when running starcoder on 2 A10 cards

Same error when TP was enabled.

Resolving patch:
diff --git a/vllm/sequence.py b/vllm/sequence.py
--- a/vllm/sequence.py
+++ b/vllm/sequence.py
@@ -401,6 +401,12 @@ def __repr__(self) -> str:
         return (f"SequenceGroupOutputs(samples={self.samples}, "
                 f"prompt_logprobs={self.prompt_logprobs})")
 
+    def __eq__(self, other: object) -> bool:
+        if not isinstance(other, SequenceGroupOutputs):
+            raise NotImplementedError()
+        return (self.samples == other.samples
+                and self.prompt_logprobs == other.prompt_logprobs)
+
 
 # For each sequence group, we generate a list of SequenceOutputs object,
 # each of which contains one possible candidate for the next token.


Test patch: