Weird behaviour running AWQ code on RTX 4000 Ada that worked on Tesla T4
Author: samrickman
Pull request: no

### System Info

#### RTX 4000 Ada 20gb

```
- `transformers` version: 4.41.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.1
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```

#### AWS T4 16gb

```
- `transformers` version: 4.38.2
- Platform: Linux-5.15.0-1055-aws-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```

### Who can help?

@SunMarc I hope it's OK to tag you on this - the messages below indicate you are the quantization expert.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am running the same code on a PC with an Nvidia RTX 4000 Ada with that I ran on an EC2 instance with a Tesla T4. The Ada is a [better](https://technical.city/en/video/Tesla-T4-vs-RTX-4000-Ada-Generation) GPU with more RAM (20gb vs 16gb). But it's an order of magnitude slower. This doesn't seem to be the case with all models - it's slightly faster with T5 - but it's much slower with a quantized version of Llama3.

In both cases, `torch.cuda.is_available() == True`. During the `model.generate()` phase, I can see with `nvidia-smi` that the GPU usage is close to 100% on the RTX but only about 60-65% on the T4.

#### Some weird things:

1. This happens whether I use `AutoModelForCausalLM.from_pretrained(device_map="auto")` or `AutoModelForCausalLM.from_pretrained(device_map="auto").to("cuda")`. However, in the second case, on both systems I get a warning: 

```
You have loaded an AWQ model on CPU and have a CUDA device available, make sure to set your model on a GPU device in order to run your model.
```

I think this warning is a false positive because in both cases I can see with `nvidia-smi` that the GPU is being used. This occurs whether I use `.to()` in any form - whether the argument provided is `"cuda"`, `0` or `torch.device("cuda")`. Both systems only have one GPU.

2. The model itself is cached on both and loads in a couple of seconds. Then the RTX machine takes about 10 minutes to generate text, while the T4 takes about 1 minute to produce the same text verbatim (setting `do_sample = False`).

#### Minimal reproducible example:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from torch import cuda, float16

def write_text(tokenizer, model, max_new_tokens=2048):
    cuda.empty_cache()
    messages = [
    {"role": "system", "content": "You are a helpful, respectful and honest assistant."},
    {"role": "user", "content": "Can you explain why an Nvidia RTX Ada 4000 with 20gb GPU RAM might be slower running a quantized model than a Tesla T4 with 16GB GPU RAM."}
    ]
    tokens = tokenizer.apply_chat_template(
        messages,
        add_generation_prompt=True,
        return_tensors="pt"
    ).to(model.device)
    
    terminators = [
        tokenizer.eos_token_id,
        tokenizer.convert_tokens_to_ids("
")
    ]

    generation_output = model.generate(
        tokens,
        do_sample=False,
        max_new_tokens=max_new_tokens,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=terminators
    )

    # Decode but chop off input - we don't need to regurgitate it
    return tokenizer.decode(generation_output[0])

assert(cuda.is_available())

# Load model
model_id = "solidrust/Meta-Llama-3-8B-Instruct-hf-AWQ"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,torch_dtype=float16).to("cuda")

print(write_text(tokenizer,model))
```


### Expected behavior

I wouldn't expect the warning when the GPU is apparently being used in both cases. I also am confused about why it's so much slower on the apparently faster GPU. It feels as if there might be something crucial about the architecture of the RTX 4000 that I'm missing. I tried changing the `torch_dtype` to other sizes of float or bfloat but that didn't help and also generates a warning. 

It looks to me that what is happening is that for some reason on the Ada, the same model is using the entire GPU, so some is offloaded to CPU, which is obviously much slower. But I don't see why this should be the case.

Comments:
Hi @samrickman, thanks for the detailed report and the minimal reproducer. I really appreciate that. 
- For the warning, when you load the model like that `model = AutoModelForCausalLM.from_pretrained(model_id,torch_dtype=float16)`, it will put the model on the cpu. This is why you get a warning to remind you to move the model to the GPU. When we use device_map="auto" or device_map="cuda", it will put the model directly on the cuda.Hence you don't get the warning. 
- As for the difference of speed, I don't really understand why it happens. 10min seems like a really long time. Could you try to debug which ops is taking the longest in the RTX4000 and compare it with the T4. CPU offloading is not supported with AWQ, so I don't think the issue is there (you would get a warning otherwise). You can just check if the GPU RAM allocated is around the same value. 

Hi @SunMarc thanks so much for the quick response. OK great that makes sense re the warning.

In terms of debugging, the operation that takes longest is `model.generate()`. Everything else takes a couple of seconds at most.

The output I get from [`gpustat -i`](https://github.com/wookayin/gpustat) on the RTX is:

```
sam                                Wed May 22 14:39:43 2024  551.61
[0] NVIDIA RTX 4000 Ada Generation | 36°C,  93 % |  6381 / 20475 MB | root(?M)
```

On the RTX4000 the CPU fluctuates around the 92-96% range.

Whereas on the T4 it is:

```
sam               Wed May 22 13:48:25 2024  535.104.12
[0] Tesla T4 | 38°C,  63 % |  6187 / 15360 MB | jupyter-sam(6182M)
```

The CPU usage on the T4 stays around 60-65%.

On both this remains until `model.generate()` is finished. The RAM doesn't change by more than a few mb.

I didn't realise there was no CPU-offloading. I have no idea what the issue is. Could it be something to do with the architecture used when the AWQ model was created? I can't see on the model card page which device it was trained on - if it happened to be a T4 would that mean it might be optimised for a T4 compared with other GPUs?

This doesn't explain or exactly solve the problem, but I have realised that I can use the original meta Llama 3 8b on the RTX4000 as it uses just over 16GB of GPU RAM. On the T4 it is (I assume) either being offloaded to CPU or swapped in and out of GPU RAM - not sure which?

When I try the [example](https://github.com/huggingface/blog/blob/main/llama3.md#using-transformers) from the Hugging Face docs, once the model is loaded it takes about 2 mins on the T4 and about about 3 seconds on the RTX4000.

This means that full Llama 3 on the RTX is 
about the same speed as
 about 25% faster than the quantized Llama 3 on the T4. So I can happily use the full version on the RTX for now.

As it happens, though, the AWQ Llama 3 seems excellent. In an ideal world I'd be able to use it on the RTX and generate even faster. I'm not quite sure what the next steps will be with regard to that. I suppose I should try more AWQ models and see if I can isolate whether the issue is only with the AWQ model I've tried, or whether it applies to other AWQ models.

Another update - it does seem to be do with AWQ. I tried the minimal example in my original post with another AWQ Llama3 model, [`PrunaAI/Meta-Llama-3-8b-instruct-AWQ-smashed`](https://huggingface.co/PrunaAI/Meta-Llama-3-8b-instruct-AWQ-smashed)

The `model.generate()` step took just over 9 minutes.


Whereas with [`meta-llama/Meta-Llama-3-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) the same step takes 36 seconds with `torch_dtype=float16` and 33 seconds with `torch_dtype=bfloat16`.

This is what `gpustat` looks like with the full Meta model:

```
sam                                Wed May 22 21:01:52 2024  551.61
[0] NVIDIA RTX 4000 Ada Generation | 46°C,  96 % | 15889 / 20475 MB | root(?M)
```

I think that's it for me today. I suppose the next step is to try some other models with both the full and AWQ versions, and try to work out whether this is specific to Llama3 or to do with AWQ models more generally. Very open to try other debugging suggestions though?

Thanks for sharing your findings ! 

> This doesn't explain or exactly solve the problem, but I have realised that I can use the original meta Llama 3 8b on the RTX4000 as it uses just over 16GB of GPU RAM. On the T4 it is (I assume) either being offloaded to CPU or swapped in and out of GPU RAM - not sure which?

On T4, if you are using device_map="auto", it is most probably using CPU offload. 

> . In an ideal world I'd be able to use it on the RTX and generate even faster. I'm not quite sure what the next steps will be with regard to that. I suppose I should try more AWQ models and see if I can isolate whether the issue is only with the AWQ model I've tried, or whether it applies to other AWQ models.

You can try to do fusing. This will make the model faster ! See the docs here : https://huggingface.co/docs/transformers/main/en/quantization#fused-modules

> I think that's it for me today. I suppose the next step is to try some other models with both the full and AWQ versions, and try to work out whether this is specific to Llama3 or to do with AWQ models more generally. Very open to try other debugging suggestions though?

This is indeed very strange. If you have access to other RTX hardware, it can be good to give a try there. 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.