Repository: vllm-project/vllm

Issue:
Decode error while inferencing a batch of prompts
I'm trying to benchmark the performance of vLLM OPT. But I find that when I pass a relatively large batch of prompts to vLLM, it will raise decode error **when the sequence length meets a threshold** (which makes the problem look like an OOM).

A minimal reproduction for this issue:
```python
from vllm import LLM, SamplingParams

def make_input(bs):
    return ["Hello!" for _ in range(bs)]

bs = 128
generate_length = 200

# Create a sampling params object.
sampling_params = SamplingParams(
    temperature=0.8, 
    top_p=0.95, 
    max_tokens=generate_length)

# Create an LLM.
llm = LLM(
    model="facebook/opt-125m",
    use_dummy_weights=True,
)
input = make_input(bs)
out = llm.generate(input, sampling_params)
```
When `bs=128`, the error happens in the 108-th token approximately. The error looks like
```
Traceback (most recent call last):
  File "vllm-none-problem-repro.py", line 21, in <module>
    out = llm.generate(input, sampling_params)
  File "/llm-bench/vllm-src/vllm/entrypoints/llm.py", line 127, in generate
    return self._run_engine(use_tqdm)
  File "/llm-bench/vllm-src/vllm/entrypoints/llm.py", line 147, in _run_engine
    step_outputs = self.llm_engine.step()
  File "/llm-bench/vllm-src/vllm/engine/llm_engine.py", line 246, in step
    self._decode_sequences(seq_groups)
  File "/llm-bench/vllm-src/vllm/engine/llm_engine.py", line 263, in _decode_sequences
    new_token, new_output_text = detokenize_incrementally(
  File "/llm-bench/vllm-src/vllm/transformers_utils/tokenizer.py", line 73, in detokenize_incrementally
    output_text = tokenizer.convert_tokens_to_string(output_tokens)
  File "/opt/conda/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 533, in convert_tokens_to_string
    return self.backend_tokenizer.decoder.decode(tokens)
TypeError: argument 'tokens': 'NoneType' object cannot be converted to 'PyString
```
If I use a smaller bs, the "threshold" will also increase (>108). For example, it's around 210 when `bs=64`. Seems that there is a limit for `bs * length`.


Discussion:
I found that the batch size is only indirectly the reason and it doesn't have anything to do with OOM or similar things. For example, if I just change the random seed to the following and keep the sequence length and batch size the same, then the bug doesn't happen anymore for this specific batch size, but it will happen for another larger one:
```python
# Create an LLM.
llm = LLM(
    model="facebook/opt-125m",
    use_dummy_weights=True,
    seed=2,
)
```

The reason is that for some models [there can be a mismatch between the config.vocab_size and the len(tokenizer)](https://github.com/facebookresearch/metaseq/issues/469). The model outputs a distribution over tokens in the range `vocab_size`, but only tokens in the range `len(tokenizer)` should actually be sampled. The remaining tokens are just padding and when sampling these tokens and decoding them, the result will be `None` instead of a string and so the exception will be thrown.

To fix this, if I change the `config.vocab_size` (which is 50272 for `facebook/opt-125m`) in the following line to 50265, i.e. `len(tokenizer)`, then the bug doesn't happen anymore for any seed and batch size.

https://github.com/vllm-project/vllm/blob/58a072be15a4e36bee006d1c9a962e527819cf18/vllm/model_executor/models/opt.py#L276

I have also observed this for LLaMA & LLaMA-2 where it seems like for some models on huggingface the `vocab_size` does correspond to the actual number of tokens that should be sampled while for some others it doesn't. It depends on whether the number of tokens is already a multiple of 16 or if there needs to be padding. There might also be other models than OPT and LLaMA where this happens.

```python3
from transformers import AutoTokenizer, PretrainedConfig

print(len(AutoTokenizer.from_pretrained('meta-llama/Llama-2-13b-hf'))) # 32000
print(PretrainedConfig.from_pretrained('meta-llama/Llama-2-13b-hf').vocab_size) # 32000

print(len(AutoTokenizer.from_pretrained('NousResearch/Nous-Hermes-Llama2-13b'))) # 32001
print(PretrainedConfig.from_pretrained('NousResearch/Nous-Hermes-Llama2-13b').vocab_size) # 32032
```

A fix in vLLM could be to obtain the number of tokens from the tokenizer instead of the `config.json` file.
@tju01 Thank you! Now it looks like larger batch size and sequence length are just to increase the probability this error happens.
怎么替换？
@xxm1668 Modifying the `vocab_size` in `~/.cache/huggingface/hub/<model>/snapshots/<commit>/config.json` would help.

Another solution would be modifying the vLLM. For example, if you are using opt model, you should modify `vocab_size` in `__init__` function of OPTForCausalLM: `self.sampler = Sampler(config.vocab_size)` 
thx
Would there be a fix to this bug?

We should essentially compare len(tokenizer) and config.vocab_size and pick the smaller value
1. I met the same problems. I cannot fix the problem by switching `vocab_size` in config.josn because I am using CodeLLaMA, which use `vocab_parallel_embedding` and there is an assertion in it when loading weights. In order to use vocab parallel, it is also necessary for the model's vocab to be 2^n*k like. 
2. **Hard coding in LLaMAForCausalLM: ```self.sampler = Sampler(1111111) # for example```  does solve the problem**.
3. Simple way to fix this bug: Write another attribute whose value is len(tokenizer) (e.g., write `tokenizer_vocab_size` into model's config when initializing LLM) then use ```self.sampler = Sampler(config.tokenizer_vocab_size)``` instead.

Hope those information will help.
This is still an issue...
Hi all. Since this issue only exists a few models, so it would be better to add the warning log about this (#3500).
Warning log doesn't fix the issue with generation though... this problem is very common in fine-tuned versions of models with added tokens, such as any fine-tuned with ChatML prompts. For example, NousResearch/Hermes-2-Pro-Mistral-7B. These fine-tuned models are unusable with vLLM with this bug.
@simon-mo is there already a way to solve this problem without modifying the vLLM source code? I really would prefer not to have to maintain a fork of vLLM to use Nous-Hermes & other finetuned models...
@esmeetu any suggestion here to have a good default way to support fine-tuned model? 
Looking into this

```python
from transformers import AutoConfig, AutoTokenizer

config = AutoConfig.from_pretrained("NousResearch/Hermes-2-Pro-Mistral-7B")
tokenizer = AutoTokenizer.from_pretrained("NousResearch/Hermes-2-Pro-Mistral-7B")

print(config.vocab_size)  # >> 32032
print(len(tokenizer))     # >> 32002
```

So it seems like 30 tokens are missing from the tokenizer that are included in the model. Not sure how this is possible, but I can see why this would cause issues if the model predicts one of the 30 tokens that are not in the vocabulary of the tokenizer
I think the issue is that the `vocab_size` is expanded to be a nice multiple for the gpu during training. These tokens will not be trained (since there is nothing in the dataset), so they are very unlikely to be sampled.

@simon-mo

I think an approach we could take is to expand the tokenizer to have more pad tokens in this scenario. This will allow
- `vocab_size` to be a nice multiple for the gpus
- gracefully handle a case where one of the "fake" tokens are predicted

Thoughts?
Had the same issue with LeoLM/leo-mistral-hessianai-7b-chat model from huggingface. Vocab size: 32128, tokenizer len: 32002.
I took a look at how `transformers` deals with this problem. Their idea is simple: if we get a token id larger than the length of tokenizer length, the decode step just regard the token as an empty string.

Here is a demo:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-125m")
print(len(tokenizer)) # 50265
tokenizer.decode([340]) # a token id that exists, i.e. ' news'
tokenizer.decode([34000000]) # a token id that does not exist, returns empty string ''
```
@youkaichao I agree with the transformers solution. But now i can't reproduce this issue using large batch sizes. Could someone provide the reproduction code for this? cc @simon-mo @andersonbcdefg
That's not easy to reproduce, but it indeed can happen, especially when we sample for many steps, and sample out a token id that's out of the scope of the tokenizer. Basically an index-out-of-range problem.

Resolving patch:
diff --git a/vllm/engine/llm_engine.py b/vllm/engine/llm_engine.py
--- a/vllm/engine/llm_engine.py
+++ b/vllm/engine/llm_engine.py
@@ -177,14 +177,6 @@ def _init_tokenizer(self, **tokenizer_init_kwargs):
         self.tokenizer: BaseTokenizerGroup = get_tokenizer_group(
             self.parallel_config.tokenizer_pool_config, **init_kwargs)
 
-        if len(self.get_tokenizer()) != self.model_config.get_vocab_size():
-            logger.warning(
-                f"The tokenizer's vocabulary size {len(self.get_tokenizer())}"
-                f" does not match the model's vocabulary size "
-                f"{self.model_config.get_vocab_size()}. This might "
-                f"cause an error in decoding. Please change config.json "
-                "to match the tokenizer's vocabulary size.")
-
     def _verify_args(self) -> None:
         self.model_config.verify_with_parallel_config(self.parallel_config)
         self.cache_config.verify_with_parallel_config(self.parallel_config)
diff --git a/vllm/entrypoints/openai/serving_engine.py b/vllm/entrypoints/openai/serving_engine.py
--- a/vllm/entrypoints/openai/serving_engine.py
+++ b/vllm/entrypoints/openai/serving_engine.py
@@ -68,14 +68,6 @@ async def _post_init(self):
             tokenizer_mode=engine_model_config.tokenizer_mode,
             trust_remote_code=engine_model_config.trust_remote_code)
 
-        if len(self.tokenizer) != engine_model_config.get_vocab_size():
-            logger.warning(
-                f"The tokenizer's vocabulary size {len(self.tokenizer)}"
-                f" does not match the model's vocabulary size "
-                f"{engine_model_config.get_vocab_size()}. This might "
-                f"cause an error in decoding. Please change config.json "
-                "to match the tokenizer's vocabulary size.")
-
     async def show_available_models(self) -> ModelList:
         """Show available models. Right now we only have one model."""
         model_cards = [
diff --git a/vllm/transformers_utils/tokenizer.py b/vllm/transformers_utils/tokenizer.py
--- a/vllm/transformers_utils/tokenizer.py
+++ b/vllm/transformers_utils/tokenizer.py
@@ -232,9 +232,13 @@ def detokenize_incrementally(
              all_input_ids[:-1],
              skip_special_tokens=skip_special_tokens)
 
-    # Put new_token_id in a list so skip_special_tokens is respected
-    new_tokens = tokenizer.convert_ids_to_tokens(
-        [new_token_id], skip_special_tokens=skip_special_tokens)
+    # If the new token id is out of bounds, return an empty string.
+    if new_token_id >= len(tokenizer):
+        new_tokens = [""]
+    else:
+        # Put new_token_id in a list so skip_special_tokens is respected
+        new_tokens = tokenizer.convert_ids_to_tokens(
+            [new_token_id], skip_special_tokens=skip_special_tokens)
     output_tokens = prev_tokens + new_tokens
 
     # If this is the first iteration, return all tokens.


Test patch:
diff --git a/tests/tokenization/test_detokenize.py b/tests/tokenization/test_detokenize.py
--- a/tests/tokenization/test_detokenize.py
+++ b/tests/tokenization/test_detokenize.py
@@ -83,6 +83,13 @@ def test_decode_streaming(tokenizer_id, truth, with_prompt,
 
     assert decoded_text == generated
 
+    decoded_text = _run_incremental_decode(
+        tokenizer, [len(tokenizer)],
+        skip_special_tokens=skip_special_tokens,
+        starting_index=starting_index)
+
+    assert decoded_text == ''
+
 
 @pytest.fixture
 def detokenizer(tokenizer_name: str) -> Detokenizer: