Idefics: Fix information leak with cross attention gate in modeling
Author: leot13
Pull request: yes

# What does this PR do?


Fixes #26428 reported by @momergul
This PR fixes the cross-attention hidden states of tokens attending no images.
This issue had no detrimental effect for inference, however this should be useful for the few tokens attending no images during training.
This PR should also prevent the cross-attentions from generating NaNs when using torch>=2.1
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @amyeroberts @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Let me run the slow tests to see if this PR also fix them.

When I ran `TF_FORCE_GPU_ALLOW_GROWTH=true RUN_SLOW=1 python3 -m pytest -v tests/models/idefics/` on a single T4 GPU machine, there are following errors. The `xxx_offload`  could be ignored as it is likely due to the `safetensors` PRs.

`test_cross_attention_gates` and `test_feed_forward_chunking` need a look: I am not sure about `test_feed_forward_chunking` as I don't know if it was already failing before this PR.


```bash
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_cross_attention_gates - RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_disk_offload_bin - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_disk_offload_safetensors - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsModelTest::test_feed_forward_chunking - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_cpu_offload - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_cross_attention_gates - RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_determinism - ValueError: zero-size array to reduction operation maximum which has no identity
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_disk_offload_bin - AssertionError: False is not true
FAILED tests/models/idefics/test_modeling_idefics.py::IdeficsForVisionText2TextTest::test_disk_offload_safetensors - AssertionError: False is not true

```

Thanks for running this! Last update should fix `test_cross_attention_gate`. I am not sure about `test_feed_forward_chunking` though

test `test_feed_forward_chunking though` starts to fail  (on GPU) after we switch to torch 2.1, so I think it's the same issue of SDPA.
Could you check with torch 2.0.1 (on GPU) and see if it pass? If so, this implies either there is more to do in this PR, or the test has to be updated if that makes more sense.

For offload tests, it's unclear as there are also the recent changes in `safetensors`.

Sorry for the delay @ydshieh, I did the slow tests with both torch 2.1 and torch 2.0.1, and double checked the attention values. 
With torch 2.1 I do get `nans` for both `test_feed_forward_chunking` and `cross_attention_gate`, this is why they both fail in slow tests currently.
With torch 2.0.1 both tests pass now

Hi @leot13 Thank you for double checking.

@leot13 I think this PR tries to zero-out some hidden states, right? So intuitively, I expect there should be no more `nan` value (as this happens when a token has no image to attend, if I still remember correctly). Do I miss any detail here - I mean maybe I misunderstand the goal of this PR.

This PR zeros-out a given `hidden_state` coming out of the cross-attention if the `image_attention_mask` is filled with zeros for this particular `hidden_state` . However, it won't solve the nan issue, because the gate is multiplying the `hidden_states` by 1 or 0. So if a hidden_state has `nans`, they will still stay `nans`.

I see. But maybe we can use the gate as a kind of  mask, and set the corresponding places in `hidden_state` to be `zero` if the mask (i.e. the gate) says a places is `image_attention_mask is filled with zeros`. So no multiplication, but instead  `simply replace values`. This should avoid the `nan`  IMO. Does this sound make sense ?



The last update should have fixed the slow tests as well. Now the hidden_states are filled with zeros instead of being multiplied by zero when they should attend to no images

Félicitations, @leot13 

<img width="212" alt="Screenshot 2023-11-16 165134" src="https://github.com/huggingface/transformers/assets/2521628/16728ce7-e7e3-4a29-b80e-242389d1e4c1">

🎉