Error Attention Mask Size for Customized Encoder-Decoder Architecture Using Multimodal Encoder
Author: allanj
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.1
- Platform: Linux-5.4.56.bsk.10-amd64-x86_64-with-debian-10.11
- Python version: 3.7.3
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.10.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:  yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

EncoderDecoder @patrickvonplaten  and Multimodal @NielsRogge 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
I was following the content (https://huggingface.co/blog/warm-starting-encoder-decoder) here to leverage the encoder-decoder model.
```python3
## code below is standard
from transformers import EncoderDecoderModel
import torch
model = EncoderDecoderModel.from_encoder_decoder_pretrained("microsoft/layoutlmv3-base", "roberta-base")
tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.eos_token_id = tokenizer.sep_token_id
model.config.bos_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id
model.config.vocab_size = model.config.encoder.vocab_size

model.config.max_length = 100
model.config.no_repeat_ngram_size = 3
model.config.early_stopping = True
model.config.num_beams = 1

### starting to test
batch_size = 2
input_len = 20
output_len = 30
vocab_size = 100 # just example number
input_ids = torch.randint(0, vocab_size, (batch_size, input_len))
bbox = torch.tensor([[[0,0,0,0] for _ in range(input_len)] for _ in range(batch_size)])
attention_mask = torch.ones((batch_size, input_len))

labels = torch.randint(0, vocab_size, (batch_size, output_len))
pixel_values = torch.randn((batch_size, 3,  224, 224))

model(input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, pixel_values=pixel_values,labels=labels)

```

__Error Message__:
```bash
Traceback (most recent call last):
  File "/Users/allanjie/projects/LayoutLMv3-DocVQA/demo.py", line 73, in <module>
    model(input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, pixel_values=pixel_values,labels=labels)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 516, in forward
    decoder_outputs = self.decoder(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 971, in forward
    outputs = self.roberta(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 848, in forward
    encoder_outputs = self.encoder(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 524, in forward
    layer_outputs = layer_module(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 435, in forward
    cross_attention_outputs = self.crossattention(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 336, in forward
    self_outputs = self.self(
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/allanjie/anaconda3/envs/general/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 259, in forward
    attention_scores = attention_scores + attention_mask
RuntimeError: The size of tensor a (217) must match the size of tensor b (20) at non-singleton dimension 3
```

### Expected behavior

Thus, I think the reason for the error is actually the mismatch between the `attention_mask` and the actual attention_mask size.

The attention mask we provide for multimodal model such as LayoutLMv3 usually only considers the textual component, 
```
attention_mask: batch_size x input_text_length
```
While the actual encoder_hidden_states are 
```
encoder_hidden_states: batch_size x (input_text_length + visual token length) x hidden_size
```

Usually, `visual token length` is around 144. 

__My Solution__:
What I did by myself is make a customized `EncoderDecoder` Class and modify the `forward` function to become something like this 
```python3
visual_attention_mask = torch.ones(
            (batch_size, seq_len - attention_mask.size(1)), dtype=torch.long, device=encoder_hidden_states.device
        )
updated_attention_mask = torch.cat([attention_mask, visual_attention_mask], dim=1)

# Decode
decoder_outputs = self.decoder(
    input_ids=decoder_input_ids,
    attention_mask=decoder_attention_mask,
    encoder_hidden_states=encoder_hidden_states,
   encoder_attention_mask=updated_attention_mask, ## originally, it should be just attention mask.
```

Similar here
```python3
def prepare_inputs_for_generation(
    self, input_ids, past=None, attention_mask=None, use_cache=None, encoder_outputs=None, **kwargs
):
    decoder_inputs = self.decoder.prepare_inputs_for_generation(input_ids, past=past)
    decoder_attention_mask = decoder_inputs["attention_mask"] if "attention_mask" in decoder_inputs else None
    batch_size, seq_len, _ = encoder_outputs[0].size()
    visual_attention_mask = torch.ones(
        (batch_size, seq_len - attention_mask.size(1)), dtype=torch.long, device=encoder_outputs[0].device
    )
    updated_attention_mask = torch.cat([attention_mask, visual_attention_mask], dim=1)
    input_dict = {
        "attention_mask": updated_attention_mask,
        "decoder_attention_mask": decoder_attention_mask,
        "decoder_input_ids": decoder_inputs["input_ids"],
        "encoder_outputs": encoder_outputs,
        "past_key_values": decoder_inputs["past_key_values"],
        "use_cache": use_cache,
    }
    return input_dict
```

__Suggested Modification__:
Because different models have a different design, we cannot think that everything is compatible with the above modification. 
But I do suggest that we can add an `assertion` statement in between the encoder and decoder for users to check that.
```
assert encoder_outputs[0].size(1) == attention_mask.size(1), you should have the same length ......
```
This better help user understand what's wrong.


Comments:
I see what you mean. However, if we add this error message to `modeling_encoder_decoder.py`, then users will update the attention mask of the encoder to make sure it has the same sequence length as the encoder final hidden states.

However, for a model like LayoutLMv3, the `attention_mask` should just contain values for the text sequence length. It will result in an error in case you're providing an attention mask that has the same sequence length as the final hidden states.

Hence, I don't think adding this error message is possible, and it might be better to just do as you did - forking the library and tweak the code.

I see. I get your concern. Yeah. maybe that's the only way for now.