[EncoderDecoder] Fix initialization and save/load bug
Author: patrickvonplaten
Pull request: yes

This PR fixes two bugs:

- Cross attention layers were not initialized when initiating the `EncoderDecoderModel` via `from_encoder_decoder_pretrained()`. Thanks goes to https://github.com/huggingface/transformers/issues/4293 for finding this bug! A slow test is including in this PR to prevent future bugs of this  kind.

- Saving / loading of pretrained models didn't work because the weights were initialized from the `model_to_load` weights because of a missing `base_model_prefix` in the `EncoderDecoderModel` class. Another slow test is included in this PR to prevent future bugs. I think this problem was mentioned in this issue: https://github.com/huggingface/transformers/issues/4517. I didn't rerun the code attached here: https://github.com/huggingface/transformers/issues/4517#issuecomment-636189365 but I'm quite positive that this is the bug that will be fixed in this PR.

## IMPORTANT
*To everybody who has been training Bert2Bert using the EncoderDecoder framework: Training with the EncoderDecoderModel before this PR did not work because there were no cross attention layers to be trained if you initialized your `EncoderDecoderModel` using `.from_encoder_decoder_pretrained(...)`.  - I'm very sorry for the wasted compute and energy! Training should work now. I will add an encoder decoder notebook in the next 1-2 weeks showing in-detail how Bert2Bert can be used with `EncoderDecoder`*

This regards the Issues: #4445, #4647, #4517, #4443, #4293, #4640

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=h1) Report
> Merging [#4680](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a801c7fd74f56a651ba43bfc93eba93c63e84766&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `75.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4680/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4680      +/-   ##
==========================================
- Coverage   78.02%   78.01%   -0.02%     
==========================================
  Files         124      124              
  Lines       20626    20634       +8     
==========================================
+ Hits        16094    16098       +4     
- Misses       4532     4536       +4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/4680/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `92.20% <75.00%> (-3.45%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4680/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.41% <0.00%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=footer). Last update [a801c7f...a61742a](https://codecov.io/gh/huggingface/transformers/pull/4680?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@thomwolf @LysandreJik @sshleifer  - merging for now to solve a bunch of issues. On a side-note, we have not really released the `EncoderDecoderModel` feature of `transformers` yet, or? Are we planning on doing something semi-official for this?

>I will add an encoder decoder notebook in the next 1-2 weeks showing in-detail how Bert2Bert can be used with EncoderDecoder

@patrickvonplaten 

I'm already working on a EncoderDecoder notebook for summarization task using kaggle news summery dataset. Hope to finish it in a week :)

> > I will add an encoder decoder notebook in the next 1-2 weeks showing in-detail how Bert2Bert can be used with EncoderDecoder
> 
> @patrickvonplaten
> 
> I'm already working on a EncoderDecoder notebook for summarization task using kaggle news summery dataset. Hope to finish it in a week :)

That's great news! Do you use a Bert2Bert implementation? 

Yes, I am using Bert2Bert

@patrickvonplaten I noticed that since this change was merged this issue happens #5826, is it possible I'm using the config api incorrectly or may be a real issue? 

Sorry for tagging you, thanks in advance!

Hey @afcruzs - will answer on the issue :-) 

Hey, @patrickvonplaten ,  I wonder if there is any way to get the cross attention weights in the decoder from `EncoderDecoderModel` model. I looked at the document, it only will return the self-attention weights (`decoder_attentions `) in the decoder from `EncoderDecoderModel`? Thanks very much, it bothers me for a while.

Hey @kimmo1019 - could you please open a new issue about this? :-)