NaNs when running bitsandbytes quantized Chameleon 
Author: nreHieW
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using distributed or parallel set-up in script? No
- Using GPU in script?: Yes
- GPU type: Tesla T4

### Who can help?

@amyeroberts @SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I used the following code snippet from the Official Chameleon Meta Docs [here](https://huggingface.co/docs/transformers/main/en/model_doc/chameleon)

```python
from transformers import ChameleonProcessor, ChameleonForConditionalGeneration, BitsAndBytesConfig
from PIL import Image
import requests
import torch

bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.float16
)

model_id = "facebook/chameleon-7b"
processor = ChameleonProcessor.from_pretrained(model_id)
model = ChameleonForConditionalGeneration.from_pretrained(model_id, quantization_config=bnb_config, device_map={"":0})
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
prompt = "What do you see in this image?<image>"

inputs = processor(prompt, image, return_tensors="pt").to(model.device, model.dtype)

output = model(**inputs)
torch.isnan(output.logits).sum().item()/output.logits.numel() # 0.65625
```

### Expected behavior

There shouldnt be any NaNs at the very least. Now more than 60% of the output logits are NaNs

Comments:
Hmm, not sure what might be the exact issue with Chameleon, but NaNs are present only for BnB quantization. I tried HqqQuantization with 4bits and got 0 NaNs, while BnB with different parameters still is over 60%

@SunMarc , do you know why this might happen? Maybe we should change docs example for another quantization type, if the issue is about choosing the right quantization type

```python
from transformers import ChameleonProcessor, ChameleonForConditionalGeneration, BitsAndBytesConfig, HqqConfig
from PIL import Image
import requests
import torch

hqq_config = HqqConfig(nbits=4, skip_modules=['model.vqmodel'])

model_id = "facebook/chameleon-7b"
processor = ChameleonProcessor.from_pretrained(model_id)
model = ChameleonForConditionalGeneration.from_pretrained(model_id, quantization_config=bnb_config, device_map={"":0})

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
prompt = "What do you see in this image?<image>"

inputs = processor(prompt, image, return_tensors="pt").to(model.device)

output = model(**inputs, output_hidden_states=True, output_attentions=True)
print(torch.isnan(output.logits).sum().item()/output.logits.numel()) # 0.0
```

@zucchini-nlp  Thanks for the reply! I am not too familiar with Hqq Quantization but I am assuming that this doesn't support qlora? 

Here is a cool guide on how to choose a quantization methods that suits your use-case (https://huggingface.co/docs/transformers/v4.43.0/quantization/overview)

From what I see, HQQ supports PEFT tuning, but I can;t say for sure. You can try out different methods and see which one works for you

Hi @nreHieW, I think that you are hitting nan values because you are setting `bnb_4bit_compute_dtype=torch.float16`. The model was trained on torch.bfloat16, so it's best to set `bnb_4bit_compute_dtype=torch.bfloat16` and set the `torch_dtype="torch.bfloat16"` in `from_pretrained`. Otherwise, you might get overflow / get nan values. LMK how it goes, I can't access the model yet since I just asked for access. 

Hi @SunMarc, thanks! it works!

Leaving the code snippet here for any future reference.
```python
from transformers import ChameleonProcessor, ChameleonForConditionalGeneration, BitsAndBytesConfig
from PIL import Image
import requests
import torch

bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
)

model_id = "facebook/chameleon-7b"
processor = ChameleonProcessor.from_pretrained(model_id)
model = ChameleonForConditionalGeneration.from_pretrained(model_id, quantization_config=bnb_config, device_map={"":0}, torch_dtype=torch.bfloat16)
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)
prompt = "What do you see in this image?<image>"

inputs = processor(prompt, image, return_tensors="pt").to(model.device, model.dtype)

output = model(**inputs, output_hidden_states=True)
torch.isnan(output.logits).sum().item()/output.logits.numel() #0.0
```

I am surprised that this works given that Colab's T4 doesn't have native bf16 support?



As a side node, i checked the hidden states when using fp16 , only the final one has nan values, not sure why that is the case but might be useful as a reference. 
```python
for x in output.hidden_states:
    print(torch.isnan(x).sum().item()/x.numel())
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.4178398456352558
```

Thanks so much for your help!