[EncoderDecoder] Add functionality to tie encoder decoder weights
Author: patrickvonplaten
Pull request: yes

This PR adds the functionality to tie encoder / decoder weights for encoder decoder models, by adding a new configuration parameters flag:
`tie_encoder_decoder`
to the config.

If this parameter is set to `True` and the model is an `encoder_decoder` model, then all weights of the encoder layer are set to their respective weights of the decoder layer. This can save a lot of memory for big models, such as shared `t5-3b` models. 

Tests are added to both `T5Model, T5ForConditionalGeneration` and the `EncoderDecoderModel`. Because BART uses different parameter names for the encoder and decoder, the generic weight tying function as defined in modeling_utils.py cannot be used and a customized function should be implemented if needed in a further PR.

The API looks as follows:

```python 
# T5
from transformers import T5ForConditionalGeneration, T5Config
share_t5 = T5ForConditionalGeneration(T5Config(tie_encoder_decoder=True))
t5 = T5ForConditionalGeneration(T5Config())
assert sum(p.numel() for p in share_t5.parameters()) < sum(p.numel() for p in t5.parameters())

# Encoder Decoder
from transformers import EncoderDecoderModel
share_bert2bert = EncoderDecoderModel.from_pretrained_encoder_decoder("bert-base-cased", "bert-base-cased", tie_encoder_decoder=True)
bert2bert = EncoderDecoderModel.from_pretrained_encoder_decoder("bert-base-cased", "bert-base-cased")
assert sum(p.numel() for p in share_bert2bert.parameters()) < sum(p.numel() for p in bert2bert.parameters())
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=h1) Report
> Merging [#6538](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/37709b59099bd984858ca1884c6c70403420347d&el=desc) will **increase** coverage by `0.81%`.
> The diff coverage is `98.18%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6538/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6538      +/-   ##
==========================================
+ Coverage   79.26%   80.07%   +0.81%     
==========================================
  Files         156      156              
  Lines       28073    28124      +51     
==========================================
+ Hits        22252    22521     +269     
+ Misses       5821     5603     -218     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `88.05% <97.43%> (+0.69%)` | :arrow_up: |
| [src/transformers/configuration\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2VuY29kZXJfZGVjb2Rlci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.62% <100.00%> (+0.70%)` | :arrow_up: |
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `92.39% <100.00%> (+0.72%)` | :arrow_up: |
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `83.83% <100.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.81%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+0.25%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.97% <0.00%> (+0.97%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.25% <0.00%> (+1.29%)` | :arrow_up: |
| ... and [5 more](https://codecov.io/gh/huggingface/transformers/pull/6538/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=footer). Last update [37709b5...1c77a34](https://codecov.io/gh/huggingface/transformers/pull/6538?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Are the bart differences just layernorms?
[`embed_positions`, 'embed_tokens`, `self_attn`] all the same afaict.

@sshleifer - great review, thanks a lot!
Regarding Bart, yeah the naming is different for some layers "final_layer_norm" for example and the objects are not of the same type, so I think the best idea is to add a `_tie_encoder_decoder_weights` function to `BartPretrainedModel` that overwrites the default function - what do you think? 

great idea

> Look good to me and a great addition!
> 
> Should the `tie_encoder_to_decoder_recursively` method be also used in the general encoder-decoder models?

Yes, it's added for the encoder-decoder models as well :-) 

See: 
```python
# Encoder Decoder
from transformers import EncoderDecoderModel
share_bert2bert = EncoderDecoderModel.from_pretrained_encoder_decoder("bert-base-cased", "bert-base-cased", tie_encoder_decoder=True)
bert2bert = EncoderDecoderModel.from_pretrained_encoder_decoder("bert-base-cased", "bert-base-cased")
assert sum(p.numel() for p in share_bert2bert.parameters()) < sum(p.numel() for p in bert2bert.parameters())
```