Repository: vllm-project/vllm

Issue:
CUDA error: an illegal memory acces with Falcon 40B
Hi,
I am testing different models with vllm. I see 
```CUDA error: an illegal memory access``` when I use falcon 40 b. The code I use is 
```
llm = LLM(model=ckpt_dir,tensor_parallel_size=4,trust_remote_code=True,gpu_memory_utilization=0.8)
sampling_params = SamplingParams(temperature=0, top_p=1.0, max_tokens=300)
results = llm.generate(prompts, sampling_params)
```
I am using an A100 with 4 GPUs. Please let me know if you have any questions


Discussion:
Hi @LopezGG, thanks for trying our vLLM and reporting the error. Could you share what your `prompts` is?
Hi @WoosukKwon ,
Here is a sample prompt

> "prompt": "Here are the answers for the problems in the exam.\nProblem 1.    A cycled from P to Q at 10 kmph and returned at the rate of 9 kmph. B cycled both ways at 12 kmph. In the whole journey B took 10 minutes less than A. Find the distance between P and Q.\nChoose from the following options:    (A)1.75km (B)2.75km (C)3.75km (D)4.75km (E)5.75km\nThe answer is therefore C\n<END>\nProblem 2.    In a poll of 45,000 dentists, only 30 percent responded; of these, 20 percent claimed to prefer Toothpaste A. How many of the dentists who responded did not claim a preference for Toothpaste A?\nChoose from the following options:    (A)2,200 (B)2,640 (C)6,160 (D)8,800 (E)10,800\nThe answer is therefore E\n<END>\nProblem 3.    In a shipment of 120 machine parts, 5 percent were defective. In a shipment of 80 machine parts, 10 percent were defective. For the two shipments combined, what percent of the machine parts were defective?\nChoose from the following options:    (A)6.5% (B)7.0% (C)7.5% (D)8.0% (E)8.5%\nThe answer is therefore C\n<END>\nProblem 4.    In recent Malta elections, in a particular constituency, 100,000 votes were cast and each vote was cast for\neither Candidate A or Candidate B. If candidate A has won by 500 votes, what percent of the 100,000 votes\nwere cast for Candidate A?\nChoose from the following options:    (A)50.05% (B)50.25% (C)50.5% (D)51% (E)52.5%\nThe answer is therefore B\n<END>\nProblem 5.    In 4 years, Raj's father will be double Raj's age then. Two years ago, while his mother was twice his age that time. If Raj is going to be 32 years old 8 years from now, then what is the sum of his parents age now?\nChoose from the following options:    (A)97 (B)98 (C)99 (D)100 (E)101\nThe answer is therefore B\n<END>\nProblem 6.    It takes a worker 9 minutes to drive from home to work at an average rate of 20 kilometers per hour. How many minutes will it take the worker to cycle from home to work along the same route at an average rate of 6 kilometers per hour?\nChoose from the following options:    (A)30 (B)32 (C)35 (D)36 (E)40\n",

It comes from https://github.com/microsoft/AGIEval
Hi, 

I am facing the same issue with 4 GPU A10G.
Using aa84c92ef636e689b506b9842c712e5c615cc73a makes it work for me while 621980bdc0d5a41e224febf962a6e0474e2b14ef gives me the same error. So it seems like the error was introduced in 621980bdc0d5a41e224febf962a6e0474e2b14ef. I am using [`OpenAssistant/falcon-40b-sft-top1-560`](https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560) on 4xA6000 and tried it both with `tensor_parallel_size=4` and `tensor_parallel_size=2`.
I used the following code for finding out where the issue is:

```python
import vllm
vllm.LLM('OpenAssistant/falcon-40b-sft-mix-1226', trust_remote_code=True, tensor_parallel_size=2)
```

The logic of the commit 621980bdc0d5a41e224febf962a6e0474e2b14ef itself seems to be correct. However, there seems to be a previous issue with the check for whether a model is of type `falcon` that just got relevant through this commit. The check is in the following line:

https://github.com/vllm-project/vllm/blob/d2b2eed67c49cdda3c1d6fa09ee2ec128b318138/vllm/config.py#L102

If I add the following code before this check:

```python
print('self.hf_config.model_type =', self.hf_config.model_type)
```

Then I get the following result:

```
(RayWorker pid=186681) self.hf_config.model_type = RefinedWeb
```

However, if I add the following:

```python
print('self.hf_config =', self.hf_config)
```

I get this result:

```
(RayWorker pid=186681) self.hf_config = RWConfig {
[...]
(RayWorker pid=186681)   "model_type": "falcon",
(RayWorker pid=186681)   "multi_query": true,
[...]
(RayWorker pid=186681) }
```

This clearly states `"model_type": "falcon"` and not `RefinedWeb`.

I don't understand this. I also get the warning `You are using a model of type RefinedWeb to instantiate a model of type falcon. This is not supported for all configurations of models and can yield errors.` at the beginning which is probably significant here.
> I used the following code for finding out where the issue is:
> 
> ```python
> import vllm
> vllm.LLM('OpenAssistant/falcon-40b-sft-mix-1226', trust_remote_code=True, tensor_parallel_size=2)
> ```
> 
> The logic of the commit [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef) itself seems to be correct. However, there seems to be a previous issue with the check for whether a model is of type `falcon` that just got relevant through this commit. The check is in the following line:
> 
> https://github.com/vllm-project/vllm/blob/d2b2eed67c49cdda3c1d6fa09ee2ec128b318138/vllm/config.py#L102
> 
> If I add the following code before this check:
> 
> ```python
> print('self.hf_config.model_type =', self.hf_config.model_type)
> ```
> 
> Then I get the following result:
> 
> ```
> (RayWorker pid=186681) self.hf_config.model_type = RefinedWeb
> ```
> 
> However, if I add the following:
> 
> ```python
> print('self.hf_config =', self.hf_config)
> ```
> 
> I get this result:
> 
> ```
> (RayWorker pid=186681) self.hf_config = RWConfig {
> [...]
> (RayWorker pid=186681)   "model_type": "falcon",
> (RayWorker pid=186681)   "multi_query": true,
> [...]
> (RayWorker pid=186681) }
> ```
> 
> This clearly states `"model_type": "falcon"` and not `RefinedWeb`.
> 
> I don't understand this. I also get the warning `You are using a model of type RefinedWeb to instantiate a model of type falcon. This is not supported for all configurations of models and can yield errors.` at the beginning which is probably significant here.

That's interesting! I followed the bug and found some additional information:

1. The bug occurs in the `from_pretrained` method, where the parent class `PretrainedConfig` initializer is called. This initializer overwrites the `model_type` class variable with the value from `config.json`. This explains why `hf_config.model_type` equals `RefinedWeb`.
2. However, when printing out the config, the `PretrainedConfig.to_dict()` method is used. Inside this method, the `model_type` is set back to `__class__.model_type`. This can be seen [here](https://github.com/huggingface/transformers/blob/52574026b6740a3882d6dd1cbf1e1663d4cea27b/src/transformers/configuration_utils.py#L869C41-L869C61). In this case, `__class__` refers to `RWConfig`, whose model_type is falcon.
Regardless of the error message (`You are using a model ~~~`) I am able to load Falcon-7B models on the vLLM but not the 40B one. Perhaps the issue is related to the parameter size?
> Regardless of the error message (`You are using a model ~~~`) I am able to load Falcon-7B models on the vLLM but not the 40B one. Perhaps the issue is related to the parameter size?

Same situation as you. Do you find any solution to run falcon 40b with tp=2 so far? @kyujin-cho 
> Using [aa84c92](https://github.com/vllm-project/vllm/commit/aa84c92ef636e689b506b9842c712e5c615cc73a) makes it work for me while [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef) gives me the same error. So it seems like the error was introduced in [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef). I am using [`OpenAssistant/falcon-40b-sft-top1-560`](https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560) on 4xA6000 and tried it both with `tensor_parallel_size=4` and `tensor_parallel_size=2`.

hi, do you mean current vllm version can run OpenAssistant/falcon-40b-sft-top1-560 this repo OK? @tju01 
@lw921014 I've looked into it a little bit but couldn't go further. gave up at the moment. 
> 

Same as you. I see this has been set as a bug. Do we have some solution? @WoosukKwon 
> > Using [aa84c92](https://github.com/vllm-project/vllm/commit/aa84c92ef636e689b506b9842c712e5c615cc73a) makes it work for me while [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef) gives me the same error. So it seems like the error was introduced in [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef). I am using [`OpenAssistant/falcon-40b-sft-top1-560`](https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560) on 4xA6000 and tried it both with `tensor_parallel_size=4` and `tensor_parallel_size=2`.
> 
> hi, do you mean current vllm version can run OpenAssistant/falcon-40b-sft-top1-560 this repo OK? @tju01

No, it can't. You have to revert to the commit aa84c92ef636e689b506b9842c712e5c615cc73a
> > > Using [aa84c92](https://github.com/vllm-project/vllm/commit/aa84c92ef636e689b506b9842c712e5c615cc73a) makes it work for me while [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef) gives me the same error. So it seems like the error was introduced in [621980b](https://github.com/vllm-project/vllm/commit/621980bdc0d5a41e224febf962a6e0474e2b14ef). I am using [`OpenAssistant/falcon-40b-sft-top1-560`](https://huggingface.co/OpenAssistant/falcon-40b-sft-top1-560) on 4xA6000 and tried it both with `tensor_parallel_size=4` and `tensor_parallel_size=2`.
> > 
> > 
> > hi, do you mean current vllm version can run OpenAssistant/falcon-40b-sft-top1-560 this repo OK? @tju01
> 
> No, it can't. You have to revert to the commit [aa84c92](https://github.com/vllm-project/vllm/commit/aa84c92ef636e689b506b9842c712e5c615cc73a)

Thank you. I checkout commit [aa84c92](https://github.com/vllm-project/vllm/commit/aa84c92ef636e689b506b9842c712e5c615cc73a), and it really works.

Resolving patch:
diff --git a/vllm/config.py b/vllm/config.py
--- a/vllm/config.py
+++ b/vllm/config.py
@@ -114,8 +114,9 @@ def get_num_heads(self, parallel_config: "ParallelConfig") -> int:
         # Note: for falcon, when new_decoder_architecture is True, the
         # multi_query flag is ignored and we use n_head_kv for the number of
         # KV heads.
+        falcon_model_types = ["falcon", "RefinedWeb", "RefinedWebModel"]
         new_decoder_arch_falcon = (
-            self.hf_config.model_type == "falcon"
+            self.hf_config.model_type in falcon_model_types
             and getattr(self.hf_config, "new_decoder_architecture", False))
         if not new_decoder_arch_falcon and getattr(self.hf_config,
                                                    "multi_query", False):


Test patch: