Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models.
Author: patrickvonplaten
Pull request: yes

This PR adds the models from the following paper:

Paper: https://arxiv.org/pdf/1907.12461.pdf

The paper does a great job at showing how pretrained BERT & RoBERTa model can be leveraged for Seq2Seq tasks and yields good results on many seq2seq tasks. It's fits very well with the current implementation of the EncoderDecoder framework.

This PR adds code to port all pretrained encoder decoder models that can be found here: https://tfhub.dev/s?module-type=text-generation&subtype=module,placeholder, 

which can be found here: https://huggingface.co/models?search=google%2Froberta
and here: https://huggingface.co/models?search=google%2Fbert2

An example of how a model can be used is here:
https://huggingface.co/google/roberta2roberta_L-24_bbc

Big thanks to @shashiongithub for providing me with the tokenizer files and giving valuable insights on setting the correct generation parameters!

Comments:
If I understand correctly, the BERT model used here is slightly different because:
- It doesn't use token type IDs
- It's tying its word embedding layer to its LM head
- No pooling layer

Doesn't that just mean we could use an additional architecture instead of an entire model class? Something like the following, in `modeling_bert.py`:

```py
@add_start_docstrings(
    """Bert Model with a `language modeling` head on top that acts as a decoder in a seq2seq setting.""", BERT_START_DOCSTRING
)
class CausalBertModel(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)

        if not config.is_decoder:
            logger.warning("If you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`")

        self.bert = BertModel(config)
        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

        self.init_weights()
```

And this module wouldn't accept token type IDs as an input.

I don't know what to do regarding the tokenizer though. This ^ approach could probably leverage @julien-c's https://github.com/huggingface/transformers/pull/6995

Naming ideas: `BertFor{Conditional}Generation`, `BertEncoder`, `BertDecoder`. 

Reasoning:

+ `CausalBert` doesn't make sense if the encoder wasn't trained with a causal mask. 
+ I think in class naming it's more important to give someone a sense of how to use something than how that thing was trained, but that's not an opinion I hold strongly.

Anyways, your signatures look super clean, easy and consistent!
Excited to try these out+happy to help check metrics.

@LysandreJik, 

There are a couple of problems with that: 

1) I also need a different `BertEmbeddings` or manually set `self.token_type_embeddings` to a zero matrix. Even if `token_type_ids` is set to `None` in Bert, the `self.token_type_embeddings` is always used. This model just does not have the embeddings (and should not have them IMO). I could set the `self.token_type_embedding` matrix just to 0, but then people using this class for training would not realize that a `self.token_type_embedding` matrix is trained which it shouldn't. So, here I think either way, I will need a separete `BertEmbeddings` class.

2) A bigger problem is the config class. Because I need both the new `CausalBertForCausalLM` and `BertLMHeadModel` in the `AUTO_MODELS_FOR_CAUSAL_LM` class (to leverage both models with the `EncoderDecoder` framework), the two models have to have different config classes. I guess we could also create a separate config class and overwrite the inherited config class from `BertPretrainedModel`, but then IMO, it's cleaner to just create a new `PretrainedModelClass` and in this case we can directly create a completely new model class

So overall, it seems to me that a separate model class is the cleaner way to go - what do you think?

@sshleifer - very much agree here! Think the naming should be different...`BertEncoder` is already taken though. I could go for `BertForGenerationEncoder` and `BertForGenerationDecoder` and `BertForGenerationConfig` - No need for `BertForConditionalGeneration` as the `EncoderDecoderModel will be used for this

BertForGenerationEncoder and BertForGenerationDecoder and BertForGenerationConfig  👍 

I do see lysandre's point though and would be fine with you setting `token_type` matrix to 0 if it's small (which I think it is).


summarization models seem to function and are uploaded here: 

https://huggingface.co/models?search=google%2Froberta2roberta

**UPDATE**: PR is ready for review @sshleifer @LysandreJik @sgugger . 

Would be awesome if you could take a look

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=h1) Report
> Merging [#6594](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/15478c1287a4e7b52c01730ffb0718243d153600?el=desc) will **increase** coverage by `2.11%`.
> The diff coverage is `75.76%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6594/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6594      +/-   ##
==========================================
+ Coverage   78.37%   80.49%   +2.11%     
==========================================
  Files         164      167       +3     
  Lines       31026    31314     +288     
==========================================
+ Hits        24318    25207     +889     
+ Misses       6708     6107     -601     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdDUucHk=) | `95.23% <ø> (-0.05%)` | :arrow_down: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `91.52% <40.00%> (-4.78%)` | :arrow_down: |
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `88.78% <50.00%> (-3.22%)` | :arrow_down: |
| [src/transformers/modeling\_bert\_generation.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0X2dlbmVyYXRpb24ucHk=) | `69.19% <69.19%> (ø)` | |
| [src/transformers/tokenization\_bert\_generation.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydF9nZW5lcmF0aW9uLnB5) | `94.64% <94.64%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.33% <100.00%> (+0.01%)` | :arrow_up: |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `93.61% <100.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/configuration\_bert\_generation.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2JlcnRfZ2VuZXJhdGlvbi5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `82.41% <100.00%> (-0.26%)` | :arrow_down: |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `96.92% <100.00%> (-0.28%)` | :arrow_down: |
| ... and [22 more](https://codecov.io/gh/huggingface/transformers/pull/6594/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=footer). Last update [15478c1...aa953cb](https://codecov.io/gh/huggingface/transformers/pull/6594?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> Looks great to me! I mostly have annoying nits about the docs, cause I'm an annoying person.

Haha, no you are 100% right - sorry for  being so sloppy with the docs! I should have learnt it by now ....

@sshleifer @sgugger - thanks a lot for your suggestions. I went for the name `BertGenerationEncoder` and `BertGenerationDecoder` now. I think it's the best trade-off between short and concise name that is not confusing.

Have the "share" models been implemented? In the paper, in many tasks they achieve the best results.

Yes you can find then under google/roberta2roberta

Thank you. How to tie weights in the code for training own model?

`tie_encoder_decoder=True`  -> The code in this model card should show you how to do it :-) https://huggingface.co/patrickvonplaten/roberta2roberta-share-cnn_dailymail-fp16