FLAVA ITM output changes due to padding
Author: hughmee
Pull request: no

TLDR; batch processing (padding) changes the output of `itm_scores` in FLAVA.

Hi,

I opened a GitHub [issue](https://github.com/apsdehal/flava-tutorials/issues/3#issue-2140965074) with the original FLAVA authors a while ago, but it appears that that repository is currently not being maintained. Since the issue arises from using the model from the `transformers` library, I think you might know what is happening here.

When calculating the ITM score with FLAVA, the results vary depending on the batch size. More specifically, padding seems to drastically change the model output. I ran this [demo notebook](https://github.com/apsdehal/flava-tutorials/blob/main/winoground-flava-example.ipynb) that tests FLAVA (the HF transformers implementation) on the Winoground dataset. With that, I get the following behaviour (copied from [my original GitHub issue](https://github.com/apsdehal/flava-tutorials/issues/3)).

Under the header 'Look at an example from Winoground and get the image-caption scores from FLAVA', the text and image of Winograd sample 155 are processed with `max_length=77, padding=True`: 

```
inputs_c0_i0 = flava_processor(text=[winoground[155]["caption_0"]], images=[winoground[155]["image_0"].convert("RGB")], return_tensors="pt", max_length=77, padding=True, return_codebook_pixels=True, return_image_mask=True).to("cuda")
```

According to the [HuggingFace documentation](https://huggingface.co/docs/transformers/pad_truncation), `padding=True` pads to the max sequence in the batch. That means that for this sample with a batch size of 1, no padding is applied. Inspecting the output of the processor, we find that indeed, no padding is applied.

Now in the next section of the notebook ('Get FLAVA image-caption scores from the whole dataset'), padding is set as `padding="max_length", max_length=77`, causing all tokenized inputs to have length 77. This yields the following scores:

```
contrastive text score: 0.2525
contrastive image score: 0.135
contrastive group score: 0.09
itm text score: 0.3225
itm image score: 0.1975
itm group score: 0.14
```

Changing the settings here to `padding=True` (removing padding and leaving everything else as it was), the scores now become:

```
contrastive text score: 0.2525
contrastive image score: 0.135
contrastive group score: 0.09
itm text score: 0.2125
itm image score: 0.1025
itm group score: 0.0475
```

We observe that the contrastive scores remain the same, but the itm scores have changed. Going back to sample 155, we find that going from `padding=True` (no padding) to `padding="max_length"` drastically changes the itm-score for each (image, text) pair.

```
FLAVA itm image-text match scores (no padding):
image_0, caption_0: 0.5821857452392578
image_0, caption_1: 0.6948502063751221
image_1, caption_0: 0.5699254274368286
image_1, caption_1: 0.7856388092041016

FLAVA itm image-text match scores (padding to max_length=77):
image_0, caption_0: 0.9999473094940186
image_0, caption_1: 0.9999871253967285
image_1, caption_0: 0.9997100234031677
image_1, caption_1: 0.9999109506607056
```

What could be the cause of this, and would this mean that we cannot use FLAVA with HuggingFace for batch processing (since that will add padding)?

### Who can help?
@ArthurZucker @younesbelkada @amyeroberts

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/apsdehal/flava-tutorials/blob/main/winoground-flava-example.ipynb

### Expected behavior

Padding does not affect the ITM output.

Comments:
FYI @zucchini-nlp . 
Thanks for reporting, this issue should bring a few answerers here: https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535 should shed some light. I don't know which padding side is used but padding is never completely ignored. 

@hughmee , after exploring the flava model a bit, I found that indeed there was a bug with attention masks for the multimodal encoder. The PR is linked to this issue, after it is merged you can install `transformers` from `main` to get the correct `itm_scores` 😄