[WIP] Add PretrainedEncoderDecoder class
Author: rlouf
Pull request: yes

In this PR we add the possibility to define encoder-decoder architectures. We:

- Added a `PreTrainedEncoderDecoder` class that can be initialized from pre-trained models;
- Modified the BERT model so it can behave as a decoder;
- Added a `Model2Model`class that simplifies the definition of an encoder-decoder when both encoder and decoder are based on the same model;
- Added relevant tests and updated the documentation;
- We also include a script to fine-tune an encoder-decoder model on the CNN/DailyMail dataset;
- We added a draft for a beam search.

Only the BERT model is available as a decoder right now.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=h1) Report
> Merging [#1455](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ae1d03fc51bb22ed59517ee6f92c560417fdb049?src=pr&el=desc) will **decrease** coverage by `1.92%`.
> The diff coverage is `53.35%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1455/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1455      +/-   ##
==========================================
- Coverage    85.9%   83.97%   -1.93%     
==========================================
  Files          91       87       -4     
  Lines       13653    12866     -787     
==========================================
- Hits        11728    10804     -924     
- Misses       1925     2062     +137
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `92.44% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_beam\_search.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlYW1fc2VhcmNoLnB5) | `0% <0%> (ø)` | |
| [transformers/modeling\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbS5weQ==) | `88.42% <100%> (ø)` | :arrow_up: |
| [transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fdXRpbHMucHk=) | `97.33% <100%> (-1.34%)` | :arrow_down: |
| [transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2NvbW1vbl90ZXN0LnB5) | `74.68% <100%> (-1.34%)` | :arrow_down: |
| [transformers/tests/modeling\_bert\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2JlcnRfdGVzdC5weQ==) | `96.92% <100%> (+0.53%)` | :arrow_up: |
| [transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2VuY29kZXJfZGVjb2Rlci5weQ==) | `67.69% <67.69%> (ø)` | |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.59% <86.79%> (-0.59%)` | :arrow_down: |
| [...ransformers/tests/modeling\_encoder\_decoder\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2VuY29kZXJfZGVjb2Rlcl90ZXN0LnB5) | `96.29% <96.29%> (ø)` | |
| [transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3B5dG9yY2hfdXRpbHMucHk=) | `76.92% <0%> (-16.04%)` | :arrow_down: |
| ... and [45 more](https://codecov.io/gh/huggingface/transformers/pull/1455/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=footer). Last update [ae1d03f...a88a0e4](https://codecov.io/gh/huggingface/transformers/pull/1455?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Regarding the initialization of `Bert2Rnd` with pretrained-weights for the encoder and random initialization for the decoder (as per the name), I see two potential solutions:

1. Patch `from_pretrained` in `modeling_utils.py` by not attempting to load weights if `Decoder` is in the name. I don't like this solution at all: the burden of initialization should be borne by the instantiating class, and adding model-specific logic in this function will bite us in the  🍑 with almost 100% certainty at some point in the future.

2. Override `from_pretrained` in `Bert2Rnd`. This would be fairly simple if there was a way to fetch the config created by `from_pretrained` in the Base class. I imagined the following:

```python
@classmethod
def from_pretrained(cls, pretrained_model_or_path, *args, **kwargs):
    pretrained_encoder, config = BertEncoder.from_pretrained(pretrained_model_or_path, *args, **kwargs)
    model = cls(config)
    model.encoder = pretrained_encoder
    return model
```

Changing `PretrainedModel`'s `from_pretrained` to output the config as well as the model is a non-breaking change, thanks to python's magic ✨. The following code runs without problem:

```python
def magic_function(number):
    return number, number+1

a = magic_function(10)
```

I'd appreciate your opinion since this is a central part of the library. In the meantime I'll dive into the way parameter loading works and see if I can find another solution.

Not sure I get the point of your `magic_function` stuff but yes solution 2 is the way to go.
You'll have to write a specific `from_pretrained` function for the seq2seq models.

Now I realize there was strictly zero point :smile: 

I hope to have a functioning version by noon :crossed_fingers: 

Here is something that "works" in the sense that:

1. All tests pass (with a new one that tests the initialization)
2. I can add an LM head at the top of the decoder and have a working `text -> Bert2Rand -> text` pipeline; the output is rubbish since the decoder is initialized randomly.

*Edit:* I just re-read the paper and it turns out they initialized the decoder with pretrained embeddings and not random embeddings. I’ll make the change.

To be able to generate meaningful text we would need to fine-tune the model. From here I can either:

- fine-tune the model for text generation (create `run_seq2seq_finetuning.py` and use `run_generation.py`)
- fine-tune for abstractive summarization (and create `run_abstractive_summarization.py`);

I’d vote for text generation for now as it is narrower in scope and won’t add yet another concept in the PR. Then we can finalize the API of the model and ship it + examples.

Tentative plan:
1. `Bert2Rnd` finetuning + text generation;
2. `UniLM`+ finetuning + text generation in a separate PR;
3. Abstractive summarization using `Bert2Rnd`and `UniLM`.

I am also strangely fascinated by the `BertShare`architecture (decoder sharing weights with encoder, asymmetry between them due to encoder-decoder attention only & outperforming everything else), but we can keep this one for later.

I implemented all elements necessary to reproduce the results from Lapata & Liu:

* Separate sentences in the document by `[SEP] [CLS]`. I currently did this in the `run_seq2seq_finetuning.py` file, but I could instead add a `add_special_tokens_sentence_representation` function in `tokenizer_bert.py` if you think it is cleaner.
* Add alternating `token_type_ids` for each sentence. Same remark as the previous point.
* Add a custom Optimizer class: they use separate optimizers for encoder & decoder + different learning schedules.
* Add the beloved beam-search in `modeling_beam_search.py.` It is a bit awkward, and I would like to have it well tested.

Things I need input on:
- [ ] Any mistake
- [ ] First and second point: is it worth adding two functions in `bert_tokenizer`?
- [ ] What do we do about beam search?

Ok LGTM, let's merge this and continue the work on summarization and T5 on separate PRs.

@rlouf This is a really great addition! Any plan to complete the run_summarization_finetuning.py end-to-end soon? Or any psuedo code to point me to the right direction would be great too. 

> @rlouf This is a really great addition! Any plan to complete the run_summarization_finetuning.py end-to-end soon? Or any psuedo code to point me to the right direction would be great too.

Would something like this work? 
1. Initialize a `TransformerBeamSearch` from a `PreTrainedEncoderDecoder`
2. Call the `forward` method of `TransformerBeamSearch` with encoder_input_ids and other necessary arguments
3. Use the tokenizer to convert results from step 2 back to text. 

You mentioned `TransformerBeamSearch` is a draft version. Not sure how much more work is needed on it. Looks OK to me, but I'm new to seq2seq models. :)

@hlums Thanks! You can follow the `example-summarization` branch where we are currently completing the example (and solidifying the Beam Search).

The answers to your questions are in the `evaluate` function of the `run_summarization.py` example, and you are essentially right :) We will soon release the example with a short example of how to you use BeamSearch.

@rlouf , great addition. Is that possible to initialize Model2Model class with both the encoder/ decoder are going to be XLMRoberta model and pre-train it with my own data?