Repository: vllm-project/vllm

Issue:
Support for  Smaug-72B-v0.1 on vLLM



Discussion:
@cdxzyc This model uses the same architecture as `Llama`, which vLLM already supports.
I only get garbage results out of vLLM for this model.  Tried both llama2 prompt and qwen prompt as suggested: https://huggingface.co/abacusai/Smaug-72B-v0.1/discussions/6

https://huggingface.co/abacusai/Smaug-72B-v0.1

Command:

Using vlm 0.3.2
```
python -m vllm.entrypoints.openai.api_server \
        --port=5001 \
        --host=0.0.0.0 \
        --tokenizer=hf-internal-testing/llama-tokenizer \
        --model=abacusai/Smaug-72B-v0.1 \
        --seed 1234 \
        --tensor-parallel-size=4 \
        --trust-remote-code \
        --download-dir=/workspace/.cache/huggingface/hub
```
and without `        --tokenizer=hf-internal-testing/llama-tokenizer \`  as well.

or:
```
docker run -d \
   --runtime nvidia \
   --gpus '"device=4,5,6,7"' \
    -v "${HOME}"/.cache:/workspace/.cache \
    -p 5001:5001 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --port=5001 \
    --model abacusai/Smaug-72B-v0.1 \
        --host=0.0.0.0 \
        --seed 1234 \
        --tensor-parallel-size=4 \
        --trust-remote-code \
        --download-dir=/workspace/.cache/huggingface/hub &>> logs.vllm_server.smag.5chat.txt
```
both have this issue.

E.g. for llama2 prompt:

![image](https://github.com/vllm-project/vllm/assets/2249614/bb08faff-d5d1-40d7-bdc4-613489f49242)

![image](https://github.com/vllm-project/vllm/assets/2249614/896ae965-7145-49cc-b34e-d8d52b3e8e4e)

@pseudotensor Could you try to change `attention_bias` key name to `bias` in `config.json`?
Edited the file to go from true to "bias" for, and still same issues.

![image](https://github.com/vllm-project/vllm/assets/2249614/f6e64209-7676-422f-aca2-8404b6d90785)


![image](https://github.com/vllm-project/vllm/assets/2249614/14718055-00a9-4bf8-bd63-2dd0a7df0a27)

Oh, maybe you mean the key name's value not value
Still no help:

![image](https://github.com/vllm-project/vllm/assets/2249614/05bbaf13-8a4f-43b6-a6a2-c08a14239a81)

![image](https://github.com/vllm-project/vllm/assets/2249614/e66ff5df-ba27-43dc-b3f7-1248edf88724)

The change should be

```
diff --git a/vllm/model_executor/models/llama.py b/vllm/model_executor/models/llama.py
index b7f6b8f..af5b018 100644
--- a/vllm/model_executor/models/llama.py
+++ b/vllm/model_executor/models/llama.py
@@ -184,7 +184,7 @@ class LlamaDecoderLayer(nn.Module):
             rope_scaling=rope_scaling,
             max_position_embeddings=max_position_embeddings,
             linear_method=linear_method,
-            bias=getattr(config, "bias", False),
+            bias=getattr(config, "attention_bias", False),
             sliding_window=sliding_window,
         )
```
> Still no help:
> 
> 
> 
> ![image](https://github.com/vllm-project/vllm/assets/2249614/05bbaf13-8a4f-43b6-a6a2-c08a14239a81)
> 
> 
> 
> ![image](https://github.com/vllm-project/vllm/assets/2249614/e66ff5df-ba27-43dc-b3f7-1248edf88724)
> 
> 


Seems right now. Is that model a chat model? Try without chat template.
Yes, it's chat model with DPO etc. and they mention both llama2 and other chat templates will work, like is common:

https://huggingface.co/abacusai/Smaug-72B-v0.1/discussions/6#65c660810831983147b3b78f
What I find confusing is the model card says based upon qwen, but that has own model class, not llama

https://huggingface.co/abacusai/Smaug-72B-v0.1
@pseudotensor That's so weird. This is hard to debug since that model lack of modeling file. But for the config file, i think it's correct when running it as a llama model.
the model is converted from qwen to llama using conversion scripts like this https://github.com/hiyouga/LLaMA-Factory/blob/main/scripts/llamafy_qwen.py
one difference is this model uses MHA, but Llama2 has MQA, that might be causing the issue


Resolving patch:
diff --git a/vllm/model_executor/models/llama.py b/vllm/model_executor/models/llama.py
--- a/vllm/model_executor/models/llama.py
+++ b/vllm/model_executor/models/llama.py
@@ -172,6 +172,10 @@ def __init__(
         max_position_embeddings = getattr(config, "max_position_embeddings",
                                           8192)
         sliding_window = getattr(config, "sliding_window", None)
+        # Support abacusai/Smaug-72B-v0.1 with attention_bias
+        # Support internlm/internlm-7b with bias
+        attention_bias = getattr(config, "attention_bias", False) or getattr(
+            config, "bias", False)
         self.self_attn = LlamaAttention(
             hidden_size=self.hidden_size,
             num_heads=config.num_attention_heads,
@@ -181,7 +185,7 @@ def __init__(
             rope_scaling=rope_scaling,
             max_position_embeddings=max_position_embeddings,
             linear_method=linear_method,
-            bias=getattr(config, "bias", False),
+            bias=attention_bias,
             sliding_window=sliding_window,
         )
         self.mlp = LlamaMLP(


Test patch: