Repository: vllm-project/vllm

Issue:
LlaMA 2: Input prompt (2664 tokens) is too long and exceeds limit of 2048/2560
As we all know, LlaMA 2 can support a maximum context length of 4096 tokens, but the current code will report an warning then return empty string:
```
CompletionOutput(index=0, text='', token_ids=[], cumulative_logprob=0.0, logprobs=[], finish_reason=length)
```


Discussion:
Could you double check you are using the latest version of vllm? We fixed a bug about sequence length recently #489.
Also, we haven't finished the support for LLaMA2, please keep an eye on #505.
Hi @foamliu. I think you may need to set `max_num_batched_tokens` to at least 4096 to support llama2 model max_seq_length， because the default value of `max_num_batched_tokens` is 2560 in EngineArgs.
> Could you double check you are using the latest version of vllm? We fixed a bug about sequence length recently #489. Also, we haven't finished the support for LLaMA2, please keep an eye on #505.

Big thanks to the super awesome vllm project team, I'll try the latest version and update here.
The "too long and exceeds limit of 2048" warning is still active using the latest vllm code (as of July 26 2023).

The Llama-2 config.json contains:
```
  "max_length": 4096,
  "max_position_embeddings": 2048,
```

[THIS IS A BUG in the original Llama-2 config.json and has been updated in newer revisions.]

In vllm/config.py, the ModelConfig method get_max_model_len() has a list of possible keys.  It searches these keys in order and returns the minimum of all attribute values:
```
    def get_max_model_len(self) -> int:
        max_model_len = float("inf")
        possible_keys = [
            # OPT
            "max_position_embeddings",
            # GPT-2
            "n_positions",
            # MPT
            "max_seq_len",
            # Others
            "max_sequence_length",
            "max_seq_length",
            "seq_len",
        ]
        for key in possible_keys:
            max_len_key = getattr(self.hf_config, key, None)
            if max_len_key is not None:
                max_model_len = min(max_model_len, max_len_key)
        return max_model_len
```
Thus, Llama-2 models will be constrained to a maximum context length of 2048 (max_position_embeddings) since that is the possible_key with the minimum value.
It turns out this is a bug in the Llama-2 config.json.  The updated config.json eliminates `max_length` and sets `max_position_embeddings` to 4096.
WARNING 07-28 03:23:18 scheduler.py:196] Input prompt (2716 tokens) is too long and exceeds limit of 4096

> 

There is a logging error over here: https://github.com/vllm-project/vllm/blob/64f23c29000ca832f50d3470a83d7a9bdbe95a0c/vllm/core/scheduler.py#L196-L198

We will fix it!

Resolving patch:
diff --git a/vllm/core/scheduler.py b/vllm/core/scheduler.py
--- a/vllm/core/scheduler.py
+++ b/vllm/core/scheduler.py
@@ -190,13 +190,13 @@ def _schedule(
                     break
 
                 num_prompt_tokens = seq_group.get_seqs()[0].get_len()
-                if num_prompt_tokens > min(
-                        self.scheduler_config.max_model_len,
-                        self.scheduler_config.max_num_batched_tokens):
+                prompt_limit = min(
+                    self.scheduler_config.max_model_len,
+                    self.scheduler_config.max_num_batched_tokens)
+                if num_prompt_tokens > prompt_limit:
                     logger.warning(
                         f"Input prompt ({num_prompt_tokens} tokens) is too long"
-                        " and exceeds limit of "
-                        f"{self.scheduler_config.max_model_len}")
+                        f" and exceeds limit of {prompt_limit}")
                     for seq in seq_group.get_seqs():
                         seq.status = SequenceStatus.FINISHED_IGNORED
                     ignored_seq_groups.append(seq_group)


Test patch: