"Qwen2-VL FP16 inference results in errors or gibberish output."
Author: GeLee-Q
Pull request: no
Labels: name: bug

### System Info

base this pull request :https://github.com/huggingface/transformers/pull/33211
python: Python 3.10.12

### infer code:
```
from PIL import Image
import requests
import torch
from torchvision import io
from typing import Dict
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from torch.cuda.amp import autocast


model_dir = "/workspace/qwenvl-dev/Qwen2-VL-2B-Instruct"
# Load the model in half-precision on the available device(s)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_dir,
    torch_dtype=torch.float16,  # Explicitly set to float16 for half-precision
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_dir)

# Image
image_path = "/workspace/qwenvl-dev/demo.jpeg"
image = Image.open(image_path)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]


# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '
system\nYou are a helpful assistant.
\n
user\n


Describe this image.
\n
assistant\n'

inputs = processor(
    text=[text_prompt], images=[image], padding=True, return_tensors="pt"
)
inputs = inputs.to("cuda")

output_ids = model.generate(
    **inputs,
    max_new_tokens=128,
)


generated_ids = [
    output_ids[len(input_ids) :]
    for input_ids, output_ids in zip(inputs.input_ids, output_ids)
]
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
print(output_text)

```

#### the error information
```
Traceback (most recent call last):
  File "/workspace/qwenvl-dev/test_infer.py", line 61, in <module>
    output_ids = model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2015, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2998, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0
```

### another infer to set do_sample false
```
output_ids = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
    top_k=None,
    top_p=None,
    temperature=None,
)
```

#### infer res 
```
['!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!']
```


### the float32 res is 
```
['The image depicts a serene beach scene with a woman and her dog. The woman is sitting on the sand, wearing a plaid shirt and black pants, and appears to be smiling. She is giving a high-five to the dog, which is sitting on the sand next to her. The dog has a harness around its neck and is looking up at the woman with its front paws raised in a gesture of affection or playfulness. The background shows the ocean with gentle waves, and the sky is clear with a soft glow from the setting or rising sun, casting a warm light over the entire scene. The overall atmosphere is peaceful and joyful']
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from PIL import Image
import requests
import torch
from torchvision import io
from typing import Dict
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from torch.cuda.amp import autocast


model_dir = "/workspace/qwenvl-dev/Qwen2-VL-2B-Instruct"
# Load the model in half-precision on the available device(s)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_dir,
    torch_dtype=torch.float16,  # Explicitly set to float16 for half-precision
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_dir)

# Image
image_path = "/workspace/qwenvl-dev/demo.jpeg"
image = Image.open(image_path)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]


# Preprocess the inputs
text_prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '
system\nYou are a helpful assistant.
\n
user\n


Describe this image.
\n
assistant\n'

inputs = processor(
    text=[text_prompt], images=[image], padding=True, return_tensors="pt"
)
inputs = inputs.to("cuda")

output_ids = model.generate(
    **inputs,
    max_new_tokens=128,
)


generated_ids = [
    output_ids[len(input_ids) :]
    for input_ids, output_ids in zip(inputs.input_ids, output_ids)
]
output_text = processor.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
)
print(output_text)
```


### Expected behavior

I hope this error gets fixed and merged in.

Comments:
The language model of the Qwen2 series does not support float16 reasoning. The official document mentioned this. Please refer to https://qwen.readthedocs.io/en/latest/inference/chat.html  and check Troubleshooting section

> Qwen2系列的语言模型不支持float16推理，官方文档提到了这一点，请参考https://qwen.readthedocs.io/en/latest/inference/chat.html 并查看Troubleshooting部分

For NVIDIA GPUs, being able to perform inference with float32 and bfloat16 but not with float16 is highly unusual and goes against common sense.
I have resolved this issue and will submit a PR for everyone's reference after [this PR ](https://github.com/huggingface/transformers/pull/33211)is merged.

> Qwen2系列的语言模型不支持float16推理，官方文档提到了这一点，请参考https://qwen.readthedocs.io/en/latest/inference/chat.html 并查看Troubleshooting部分

Thank you. I've taken a look at the [Qwen team's solution](https://github.com/jklj077/transformers/commit/53dba62c9c35c4a632c81f0406d844ea3679d4dd), and while it's different from mine, it's quite valuable for reference. I'll test it out tomorrow.