[Bart] Refactor - fix issues, consistency with the library, naming
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR refactors the Bart model. The goal is to fix a couple of bugs related to Bart, make Bart more consistent with other models in the library and make Bart the "default" Seq2Seq template model for other models. The PR may be a bit difficult to review, so the following sections lists the main changes and the reasons why they are taken.

## In-detail explanation of main changes

1. Fix a bug related to `past_key_values`, `use_cache` and `decoder_input_ids`. Previously it was assumed that if `use_cache=True`, then `decoder_input_ids` have to be of length 1. This is not always the case! E.g. If the first decoder_input_ids prompt is longer than 1 and `use_cache=True` this would have led to errors previously - see #7814, #6353. This is fixed now so that any length of `past_key_values` can be combined with any length of `decoder_input_ids`, just as it can be done for GPT2, T5, CTRL, ... In order to make the pt_tf_equivalence tests pass, some hotfixes are applied for TFBart. TFBart will be refactored in a later PR. A test `create_and_check_decoder_model_past_large_inputs` is added to ensure that this functionality works.

2. Allow to use `BartEncoder` and `BartDecoder` separately from the `BartModel`. Because Bart is the default seq2seq model it's a great opportunity to combine just the `BartDecoder` with other "encoder-only" models. E.g. if someone wants to run experiments on long-range summarization `Longformer-Bart` could be an interesting combination (@ibeltagy). This PR lays the groundwork to easily combine these models by making `BartEncoder` and `BartDecoder` fully functional models on their own. One should probably also add a `BartForCausalLM` class analogs to https://github.com/huggingface/transformers/blob/df311a5ccf50be3031474e289b43b1be43111144/src/transformers/models/prophetnet/modeling_prophetnet.py#L1882 (could be a good first issue). further improves how to handle an issue like #5282

3. Simplify query, key, value projections in attention layer. A rather difficult if-else cascade with a complex follow-up function to concat past_key_values is simplified to a single if-elif-elif-else clause. IMO, the code in `BartAttention.forward()` is much clearer now.

4. Change the cache from dict to tuple and make it stateless. The general design in the library is to have a stateless design for the cache. Bart previously used a dict -> this PR changes the cache to a tuple for consistency. It should also be a bit more efficient, more consistent and easier to use with torchscript and onnx.

5. Bart did a lot of dimensions transposing from time -> batch and batch -> time. This is not at all necessary IMO. We can just have the batch dimension in the first spot the whole time just like the other models do too. Therefore, I deleted a bunch of `transpose(0, 1)` operations.

6. Add inputs_embeds. Just like other models Bart can make use of inputs_embeds.

7. Rename all classes from `...Model` to `Bart...Model`. Public class names that needed to be renamed were depreciated for backwards compatibility. This is better for look-up and consistency with other models.

8. Simpler handling of attention_masks. Previously Bart moved many different masks with many different names through-out the model. This PR aligns the functionality with other models, but creating the full attention mask for each model in the beginning of `BartEncoder` and `BartDecoder` instead of doing it in the attention function. This simplifies the code and is more consistent with other models.

9. Re-structure order in `modeling_bart.py`. Usually, modeling files have helper functions in the beginning, followed by submodules, followed by docstring, followed by the pre-trained models. This PR re-orders `modeling_bart.py` accordingly.

10. Replace functionality to make lm head embeddings on-the-fly by the usual `_init_weights` tying mechanism that we have in PyTorch. This is a) much more consistent with other models and b) cleaner because we don't have to instantiate a new class each time `get_output_embeddings()` is called. Solves #5282.

11. (subjectively) better naming. Replace x -> hidden_states, etc...

## Breaking changes

- There are no breaking changes to the "public" API IMO (except if it corrects a bug). `BartModel`, `BartForConditionalGeneration` and all other `BartPretrainedModel`s have exactly the same as before except for the following case which was a bug:
Previously, the following code:

```python
from transformers import BartForConditionalGeneration, BartTokenizer
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")

input_ids = tokenizer("the encoded sequence in all its beauty", return_tensors="pt").input_ids
decoder_input_ids = tokenizer("the decoder sequence", return_tensors="pt").input_ids

print(model(input_ids, decoder_input_ids=decoder_input_ids).logits.shape)
```

would have printed out only a single output because `use_cache` is enabled which was wrong because no causal-mask was used. This PR corrects the behavior so that the output seq length matches the decoder_input_ids seq lengths.

- BartEncoder and BartDecoder now have a rather different API. This is OK for me since it was not possible to import the models directly and there were only model components. 

- sub modules of Bart are named differently, *e.g.* LayerNorm is now called BartLayerNorm. Since these modules are also not public I don't think we have to depreciate the names.

- The API of `BartModel`, ... is extended by `inputs_embeds` and `decoder_inputs_embeds`.

## Review:

Because Bart is the most important Seq2Seq model in the library (5 other models classes depend on it), I would be very happy for a couple of thorough reviews. Also all kinds of comments, improvements, discussions, questions are welcome! I ran all slow tests and tried to be careful with the changes. In case @sshleifer is interested I'd also be more than happy about some feedback from you ;-) 

## TODO-List

- [x] Keep dims consistent within the model -> no switching around between time x batch_size and batch_size x time. We can just stick to batch_size x time throughout the whole forward pass just like other models do too.
- [x] Add same `lm_head` logic, other models have as well. Bart should make use of the `tie_weight_embeddings` function instead of doing weird `"on-the-fly"` output embeddings, #5282
- [x] Clean the Attention layer: Replace dict cache by past_key_values tuple (consistency with other models and stateless which is better IMO). Break up complicated if-else cascade and remove unnecessary parameters.
- [x] Make Encoder/Decoder stand-alone models to be used on their own: #7127, this way pretrained weights can be used in the Encoder-Decoder framework as well. If I remember correctly @ibeltagy was interested in this as well
- [x] Correct error with past_key_values/decoder_input_ids/use_cache: #7814, #6353, 
- [x] Make Bart torchscriptable: #6348
- [x] Add input_embeds to Bart
- [x] (very subjectively) better naming
- [x] Check that all slow tests are passing - ran the following slow tests:
```
[
# assumes USE_CUDA is exported, rather than passed
RUN_SLOW=1 pytest tests/test_modeling_pegasus.py
RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_marian.py
RUN_SLOW=1 pytest tests/test_modeling_mbart.py
RUN_SLOW=1 pytest tests/test_modeling_fsmt.py
RUN_SLOW=1 pytest tests/test_modeling_blenderbot.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_conversational.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_text2text_generation.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_summarization.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_translation.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_dialog.py
]
```
=> MBartEnroIntegrationTest.test_enro_generate_batch fails on PR, but also on master with the same message, so that's ok for me!

- [x] Update docstring and final design change check
- [x] Refactor Bart tests
- [x] Check no speed regression
- [x] Check no training performance regression (Is there a good fine-tuning script I can run for this @patil-suraj, @sshleifer)?

Comments:
Try the various Marian fine-tuning scripts. You should easily be able to get 22+ BLEU on wmt-en-ro with both `finetune_trainer.py` and `finetune.py` in < 30 minutes on brutasse.

Speed / Memory benchmark of master vs. this PR is ok for me:

![Screenshot from 2020-12-08 10-12-43](https://user-images.githubusercontent.com/23423619/101463805-2d8e7b00-393e-11eb-832f-114982569829.png)


@patrickvonplaten I would measure impact of running cnn summarization (which uses seq_len 1024) to fully acknowledge the trade off you are making:
```bash
cd examples/seq2seq
python run_eval.py facebook/bart-large-cnn cnn_dm/test.source cnn_gens.txt \
	--reference_path cnn_dm/test.target \
	--score_path cnn_rouge.json --task summarization \
	--n_obs 500 --fp16
```
This should take ~5 mins per branch.

Otherwise, LGTM! Thanks for cleaning up after me :)

> @patrickvonplaten I would measure impact of running cnn summarization (which uses seq_len 1024) to fully acknowledge the trade off you are making:
> 
> ```shell
> cd examples/seq2seq
> python run_eval.py facebook/bart-large-cnn cnn_dm/test.source cnn_gens.txt \
> 	--reference_path cnn_dm/test.target \
> 	--score_path cnn_rouge.json --task summarization \
> 	--n_obs 500 --fp16
> ```
> 
> This should take ~5 mins per branch.
> 
> Otherwise, LGTM! Thanks for cleaning up after me :)



> @patrickvonplaten I would measure impact of running cnn summarization (which uses seq_len 1024) to fully acknowledge the trade off you are making:
> 
> ```shell
> cd examples/seq2seq
> python run_eval.py facebook/bart-large-cnn cnn_dm/test.source cnn_gens.txt \
> 	--reference_path cnn_dm/test.target \
> 	--score_path cnn_rouge.json --task summarization \
> 	--n_obs 500 --fp16
> ```
> 
> This should take ~5 mins per branch.
> 
> Otherwise, LGTM! Thanks for cleaning up after me :)

Thanks for the command! What do you mean by "per branch"? 

Also @sshleifer I didn't really manage to find a good marian command for fine-tuning. Can you by chance copy-paste a command that fine-tunes a marian model in ~30min to verify that fine-tuning works as expected?

> @patrickvonplaten I would measure impact of running cnn summarization (which uses seq_len 1024) to fully acknowledge the trade off you are making:
> 
> ```shell
> cd examples/seq2seq
> python run_eval.py facebook/bart-large-cnn cnn_dm/test.source cnn_gens.txt \
> 	--reference_path cnn_dm/test.target \
> 	--score_path cnn_rouge.json --task summarization \
> 	--n_obs 500 --fp16
> ```
> 
> This should take ~5 mins per branch.
> 
> Otherwise, LGTM! Thanks for cleaning up after me :)

I got this result:

![Screenshot from 2020-12-08 18-56-50](https://user-images.githubusercontent.com/23423619/101522361-61da5980-3987-11eb-9531-cfcfd85000cf.png)


on brutasse - does this look reasonable to you? took 2min30 

What I meant by "per branch" was to also run that command on master to facilitate comparison. Your `refactor-bart` output looks completely reasonable.

#### Train Command
replace num_train_epochs=1 in this [./examples/seq2seq/builtin_trainer/train_distil_marian_enro.sh](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/builtin_trainer/train_distil_marian_enro.sh). 

+ It should take 12-25 minutes on 1 GPU.
+ I don't know the BLEU/timing to expect, you should again run on `master` and `bart-refactor` to compare.






> @patrickvonplaten I would measure impact of running cnn summarization (which uses seq_len 1024) to fully acknowledge the trade off you are making:
> 
> ```shell
> cd examples/seq2seq
> python run_eval.py facebook/bart-large-cnn cnn_dm/test.source cnn_gens.txt \
> 	--reference_path cnn_dm/test.target \
> 	--score_path cnn_rouge.json --task summarization \
> 	--n_obs 500 --fp16
> ```
> 
> This should take ~5 mins per branch.
> 
> Otherwise, LGTM! Thanks for cleaning up after me :)


Applied as much perf improvement as possible -> Time from master to this PR for the above command is reduced from ~2min10s to ~2min05s (ran three times) = 2.5% speed-up. Removed as many `contiguous()` operations as possible

Training gives good/equal results to master. However, I see a 5% slow-down in training. `generation()` is as fast or faster than master, but training yields a slow-down of around 5% => so still investigating. Could be the masks

Reran the fine-tuning script a couple of times on a gcp instance so that no other tasks can interfere and float masks are actually faster than boolean masks and and give more or less same results than previous bart model on master. Here the results: 


Refactor (no boolean masks)

![bart_refactor_branch](https://user-images.githubusercontent.com/23423619/101674871-548d9f80-3a59-11eb-9f6c-0cd33668be22.png)

master

![bart_master](https://user-images.githubusercontent.com/23423619/101674915-62dbbb80-3a59-11eb-835f-5b84c2e8150d.png)


good to merge for me