IDEFICS Cross Attention: Text tokens appearing before images still attend to image embeddings
### System Info

- `transformers` version: 4.33.1
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.17.1
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1: Run the following code snippet altered from `examples/idefics/inference.py` in the notebooks repo. 

```                                                                                                                                                                                                           
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/idefics-9b"
model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint, use_auth_token=False)
model.eval()

url = "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg"
image = processor.image_processor.fetch_images(url)
prompts = [
    [
        "User:",
        image,
        "Describe this image.\nAssistant: An image of two kittens in grass.",
    ],
]


inputs = processor(prompts, return_tensors="pt").to(device)
logits = model(**inputs)['logits']
```

2: During the model forward pass, inspect hidden states in Line 912 of `models/idefics/modeling_idefics.py` 

### Expected behavior

Hello! I believe there is a bug in how cross attention is performed within `IdeficsGatedCrossAttentionLayer` in `models/idefics/modeling_idefics.py` for text tokens appearing before any images are given to the model. As IDEFICS is autoregressive, the hidden state for a text token appearing before any image is observed should not be changed after cross attention. During the forward pass in the code snippet I provided, I expect the following behavior immediately after Line 911 of `models/idefics/modeling_idefics.py`:

Expected behavior:
`torch.all(residual[0, 0:4] == hidden_states[0, 0:4])` evaluates to `True`

Observed behavior:
`torch.all(residual[0, 0:4] == hidden_states[0, 0:4])` evaluates to `False`

I believe this is due to how the attention mask is applied. For the first 4 tokens which appear before any image, all values of `image_attention_mask` are set to the smallest possible value. This results in the attention weights during the call to `nn.functional.scaled_dot_product_attention` in Line 692 to each be equal to each other. This in turn means that these four text tokens appearing before any image each attend to the image embeddings. 

Is my understanding correct here? I would greatly appreciate it if you could look into this.