Dimension error while finetuning longformer with roberta-large EncoderDecoderModel 
Author: plutonium-guy
Pull request: no

## Environment info
- `transformers` version: 4.2.2
- Platform: Linux-3.10.0-1160.6.1.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.11
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
- maintained examples (not research project or legacy): @patrickvonplaten


Models:
- allenai/longformer-base-4096 with roberta-large
- allenai/longformer-base-4096 with xlm-roberta-large

## To reproduce

Steps to reproduce the behavior:
I have follow the steps on https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16
but switched model roberta-base to roberta-large

**CODE:-**
```
model = EncoderDecoderModel.from_encoder_decoder_pretrained("allenai/longformer-base-4096", "roberta-large")
tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-base-4096")
```
model params
```
# enable gradient checkpointing for longformer encoder
model.encoder.config.gradient_checkpointing = True

# set decoding params
model.config.decoder_start_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
model.config.max_length = 142
model.config.min_length = 56
model.config.no_repeat_ngram_size = 3
model.early_stopping = True
model.length_penalty = 2.0
model.num_beams = 4

encoder_length = 2048
decoder_length = 128
batch_size = 16
```
training parms
```
training_args = TrainingArguments(
    output_dir="./",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    #predict_from_generate=True,
    #evaluate_during_training=True,
    do_train=True,
    do_eval=True,
    logging_steps=1000,
    save_steps=1000,
    eval_steps=1000,
    overwrite_output_dir=True,
    warmup_steps=2000,
    save_total_limit=3,
    fp16=True,
)

# instantiate trainer
trainer = Trainer(
    model=model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
)

# start training
trainer.train()
```

**ERROR:-**
```
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
output = module(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 430, in forward
**kwargs_decoder,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 928, in forward
return_dict=return_dict,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 808, in forward
return_dict=return_dict,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 505, in forward
output_attentions,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 424, in forward
output_attentions,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 328, in forward
output_attentions,
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 198, in forward
key_layer = self.transpose_for_scores(self.key(encoder_hidden_states))
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/home/jovyan/.local/lib/python3.6/site-packages/torch/nn/functional.py", line 1692, in linear
output = input.matmul(weight.t())
RuntimeError: mat1 dim 1 must match mat2 dim 0
```


Comments:
Hey @amiyamandal-dev, you can only combine `longformer-base-4096` with `roberta-base` since those two models have the same `hidden_size`. Combining `longformer-base-4096` with `roberta-large` will necessarily lead to errors.

Thank you for your reply @patrickvonplaten,
Able to understand why it's not working. If I want to run train a model with Encoder `longformer` and Decoder some big model like `roberta-large` , so what steps to follow. That would be a great help.

I think you should be able to use longformer-large, e.g. https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa for your case

Thanks @patrickvonplaten  it worked