[EncoderDecoder] Add encoder-decoder for roberta/ vanilla longformer
Author: patrickvonplaten
Pull request: yes

This PR adds Roberta to the Encoder Decoder framework. Thus, it automatically makes it possible to use both `Roberta2Roberta` models and `Longformer2Roberta` model: 

```python
from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_pretrained("roberta-base", "roberta-base")
input_ids = torch.tensor([10 * [0]])
model(input_ids=input_ids, decoder_input_ids=input_ids)
```

and 

```python
from transformers import EncoderDecoderModel
model = EncoderDecoderModel.from_pretrained("allenai/longformer-base-4096", "roberta-base")
input_ids = torch.tensor([10 * [0]])
model(input_ids=input_ids, decoder_input_ids=input_ids)
```

Also pinging @ibeltagy and @patil-suraj 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=h1) Report
> Merging [#6411](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/404782912ad1324592c2d5bb2e88d1ee99a040b6&el=desc) will **decrease** coverage by `1.93%`.
> The diff coverage is `92.85%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6411/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6411      +/-   ##
==========================================
- Coverage   79.77%   77.84%   -1.94%     
==========================================
  Files         150      150              
  Lines       27789    27826      +37     
==========================================
- Hits        22170    21660     -510     
- Misses       5619     6166     +547     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.25% <ø> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `63.95% <ø> (-14.54%)` | :arrow_down: |
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `91.02% <ø> (ø)` | |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.42% <50.00%> (ø)` | |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.22% <50.00%> (-0.36%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.98% <97.36%> (+0.20%)` | :arrow_up: |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `28.94% <0.00%> (-67.11%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.20%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `26.98% <0.00%> (-52.81%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/huggingface/transformers/pull/6411/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=footer). Last update [4047829...ed8414a](https://codecov.io/gh/huggingface/transformers/pull/6411?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

Hmm, I think it's very new so not sure if many people have tried out the framework yet. @patil-suraj - do you know if people work a lot with EncoderDecoder by chance? 

> do you know if people work a lot with EncoderDecoder by chance?

Seems like it, seen quite a few issues and questions (on forum as well) regarding EncoderDecoder, but no one has reported any good results yet

Looks great. Thanks, @patrickvonplaten. 

> LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?

@sshleifer, was thinking about the same thing. My guess is that numbers won't be great because cross-attention is randomly initialized? 


> Looks great. Thanks, @patrickvonplaten.
> 
> > LGTM. Have people had good ROUGE with the compose two pretrained glue models and finetune for summarization approach?
> 
> @sshleifer, was thinking about the same thing. My guess is that numbers won't be great because cross-attention is randomly initialized?

Btw, this paper does some great analysis on reusing checkpoints for Seq2Seq models: https://arxiv.org/pdf/1907.12461.pdf