IDEFICS-9B output NAN with transformer 4.33.2
Author: ForJadeForest
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.27
- Python version: 3.10.14
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts @SunMarc 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I run the official code in my machine, but get a invalid ouput.

```python
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/idefics-9b"
model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint)

# We feed to the model an arbitrary sequence of text strings and images. Images can be either URLs or PIL Images.
prompts = [
    [
        "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",
        "In this picture from Asterix and Obelix, we can see"
    ],
]

# --batched mode
inputs = processor(prompts, return_tensors="pt").to(device)
# --single sample mode
# inputs = processor(prompts[0], return_tensors="pt").to(device)

# Generation args
bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

generated_ids = model.generate(**inputs, bad_words_ids=bad_words_ids, max_length=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
for i, t in enumerate(generated_text):
    print(f"{i}:\n{t}\n")
```

output is below: 
```
(['In this picture from Asterix and Obelix, we can see'],
 tensor([[    1, 32000, 32001, 32000,   512,   445,  7623,   515,   319,  2475,
            861,   322,  4250,   295,   861, 29892,   591,   508,  1074,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
              0,     0,     0,     0,     0,     0,     0,     0,     0,     0]],
        device='cuda:6'))
```

I try to do forward and find the logits are all NAN.

![image](https://github.com/user-attachments/assets/a336f550-f316-4970-8795-870910759a33)

But, when I upgrade my transformers package to 4.38.2 (Only upgrade the transformers). The result will be correct. 
![image](https://github.com/user-attachments/assets/dcf4edaa-c4f6-4a9b-9470-2d34875aeeba)


### Expected behavior

Different version of Transformers should generate the same output.

Comments:
Hi @ForJadeForest, 

This is the desired behaviour: we don't want to experience nans in our outputs and the latest version doesn't have this. It's true that we generally want consistency of behaviour across versions, however in this case it appears there's a fix which we want

> Hi @ForJadeForest,
> 
> This is the desired behaviour: we don't want to experience nans in our outputs and the latest version doesn't have this. It's true that we generally want consistency of behaviour across versions, however in this case it appears there's a fix which we want

Yeah, but I try to my old conda env with 4.33.2 transformers and I get the correct output. The old env's sys info is 
```
- `transformers` version: 4.33.2
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.27
- Python version: 3.10.14
- Huggingface_hub version: 0.24.3
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
**The correct output with 4.33.2**
![image](https://github.com/user-attachments/assets/e16fc5fb-558f-4457-a469-1efc390509a0)

**The error output with 4.33.2**
![image](https://github.com/user-attachments/assets/a42b291e-1888-44bd-9a35-b5c063af5c88)

**The output with 4.38.2**
![image](https://github.com/user-attachments/assets/266ff727-7e75-4d6c-b4bc-769c344a1a8e)

Btw, I find the first 4.33.2 output is different with 4.38.2 (I set do_sample to False). This phenomenon is very strange, it might affect my performance testing.


@ForJadeForest I'm not sure I understand the issue being reported. In the issue description, you said:

> transformers version: 4.33.2
> I run the official code in my machine, but get a invalid ouput.
> I try to do forward and find the logits are all NAN.
> But, when I upgrade my transformers package to 4.38.2 (Only upgrade the transformers). The result will be correct.

And then in the reply you said 

> Yeah, but I try to my old conda env with 4.33.2 transformers and I get the correct output. 

So do you get the expected output with 4.33.2 or not? 

Why are you comparing specifically with 4.38.2? Could you compare with the output on the most stable release -- 4.43.2? If there's a bug we can add commits which fix things for future releases - but we can't change the behaviour of older released versions

Sorry for my mistake. I meant there are three environments:

1. The first environment (env_1) has transformers version 4.33.2 (see figure 1).
2. The second environment (env_2) also has transformers version 4.33.2 (see figure 2). Env_2 has different dependency compared with env_1. 
3. The third environment (env_3) is an upgraded version of env_2 with transformers version 4.38.2 (see figure 3).

The other dependency packages are not identical between env_1 and env_2. However, env_2 and env_3 have the same dependency packages.

I have two questions:

1. I tested the result of transformers version 4.38.2 (env_3). However, I don't understand why transformers version 4.33.2 (env_2) fails. Maybe it can fix other dependency package version to solve.  Besides, I upgrade Transformers of env_3 to 4.43.2, which output the same with 4.38.2
2. I found that using env_1 and env_3 produces different outputs when `do_sample = False`. This is problematic because I am trying to reproduce a paper, and the performance on all task (Image Captioning, VQA) of env_3 is lower compared to env_1. (Compared with figure1, the output of figure 3 are simple and less information. )

Hi @ForJadeForest, apologies for the delay in response. 

With regards to the difference in env_1 and env_2, you'll need to compare the installed packages in the two environments to try and figure out what's causing the discrepancies. 

Thanks for checking with a more recent version - I can confirm I get the same output with 4.48.2 and the most recent release. 

If I install 4.33.2, 4.33.1 or 4.33.0 I also get an output of nans. As 4.33 is the version idefics was released in, I means I can't get a good reference baseline for what the "correct" generation should be.

>  found that using env_1 and env_3 produces different outputs when do_sample = False

To make sure I've understood, do you mean that when `do_sample=False` there are difference between the outputs of env 1 and env 3? 

I'm going to be off for a while. @andimarafioti Could you help in verifying what the outputs of the release model from 4.33 should be? 

> If I install 4.33.2, 4.33.1 or 4.33.0 I also get an output of nans. As 4.33 is the version idefics was released in, I means I can't get a good reference baseline for what the "correct" generation should be.

Thanks for you testing!  I know fix bug in old version is not necessary. We can only upgrade the transformers version to solve it. But I think it's better to give other people who have the same issue (a hard-to-notice bug I think) a solution. 

> To make sure I've understood, do you mean that when `do_sample=False` there are difference between the outputs of env 1 and env 3?

Yes, I use different env to evaluate the VQA performance, but the performance gap between the two versions is significant. I think it's important to know why because I can not reproduce some papers' results with the latest transformers.  

Thank you once again for your patient response.


cc @andimarafioti if you have the time to check what might be happening here

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.