Repository: vllm-project/vllm

Issue:
Batch generation with long prompt generates incorrect number of outputs
When a prompt in a batch generation is too long for the model, `llm.generate` returns an unexpected number of outputs:

```python
In [11]: prompts = ["This is a short prompt", "This is a very long prompt " * 1000]
    ...: print(len(prompts))
2

In [12]: outputs = llm.generate(prompts, sampling_params=sampling_params, use_tqdm=False)
WARNING 11-14 04:11:47 scheduler.py:146] Input prompt (6002 tokens) is too long and exceeds limit of 4096

In [13]: print(len(outputs))
3
```
It appears the too-long prompt gets doubled up in the output:

```python
In [14]: prompts = ["This is a short prompt", "This is a very long prompt " * 1000, "Here's another short
    ...: prompt"]
    ...: print(len(prompts))
3

In [15]: outputs = llm.generate(prompts, sampling_params=sampling_params, use_tqdm=False)
WARNING 11-14 04:15:02 scheduler.py:146] Input prompt (6002 tokens) is too long and exceeds limit of 4096

In [16]: outputs[0].prompt[:100]
Out[16]: 'This is a short prompt'

In [17]: outputs[1].prompt[:100]
Out[17]: 'This is a very long prompt This is a very long prompt This is a very long prompt This is a very long'

In [18]: outputs[2].prompt[:100]
Out[18]: 'This is a very long prompt This is a very long prompt This is a very long prompt This is a very long'

In [19]: outputs[3].prompt[:100]
Out[19]: "Here's another short prompt"
```

We are using `zip` to recombine the `outputs` with input data after the generation, and this causes big problems since the zip is off-by-one after any prompt was encountered over the size limit.

Here's a minimum reproducible script:
```python
from vllm import LLM, SamplingParams
sampling_params = SamplingParams(temperature=0.01, top_p=0.1, max_tokens=256)
llm = LLM(model=f"meta-llama/Llama-2-7b-hf",
          max_num_batched_tokens=4096,
          tensor_parallel_size=1)
prompts = ["This is a short prompt", "This is a very long prompt " * 1000]
print(len(prompts))
outputs = llm.generate(prompts, sampling_params=sampling_params)
print(len(outputs))
```

Environment info:
```
(eb) kwood@kwood-lab:~$ cat /etc/issue
Ubuntu 22.04.3 LTS \n \l
(eb) kwood@kwood-lab:~$ pip freeze | grep vllm
vllm==0.2.1.post1
(eb) kwood@kwood-lab:~$ nvidia-smi
Tue Nov 14 04:22:19 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        On  | 00000000:2D:00.0 Off |                  Off |
|  0%   40C    P8              36W / 450W |      3MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
```


Discussion:
Thanks for the detailed report. I'll take a look. 
I have the same error

Resolving patch:
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -567,7 +567,7 @@ def step(self) -> List[RequestOutput]:
             blocks_to_copy=scheduler_outputs.blocks_to_copy,
         )
 
-        return self._process_model_outputs(output, scheduler_outputs) + ignored
+        return self._process_model_outputs(output, scheduler_outputs)
 
     def _log_system_stats(
         self,


Test patch:
diff --git a/tests/test_regression.py b/tests/test_regression.py
new file mode 100644
--- /dev/null
+++ b/tests/test_regression.py
@@ -0,0 +1,27 @@
+"""Containing tests that check for regressions in vLLM's behavior.
+
+It should include tests that are reported by users and making sure they
+will never happen again.
+
+"""
+from vllm import LLM, SamplingParams
+
+
+def test_duplicated_ignored_sequence_group():
+    """https://github.com/vllm-project/vllm/issues/1655"""
+
+    sampling_params = SamplingParams(temperature=0.01,
+                                     top_p=0.1,
+                                     max_tokens=256)
+    llm = LLM(model="facebook/opt-125m",
+              max_num_batched_tokens=4096,
+              tensor_parallel_size=1)
+    prompts = ["This is a short prompt", "This is a very long prompt " * 1000]
+    outputs = llm.generate(prompts, sampling_params=sampling_params)
+
+    assert len(prompts) == len(outputs)
+
+
+if __name__ == "__main__":
+    import pytest
+    pytest.main([__file__])