Support T5 Generation
Author: patrickvonplaten
Pull request: yes

In this PR some first commits are added to make T5 work for generation. 

the `T5WithLMHeadModel.forward()` has a special API due to its encoder-decoder nature. 
This is why we need to add a `prepare_inputs_for_generation()` in `t5_modeling_utils.py` to 
correctly prepare t5's inputs for generation. 

Some easy translation seems to give alright results (same results for TF): 

```
model = T5WithLMHeadModel.from_pretrained('t5-base')
tok = T5Tokenizer.from_pretrained('t5-base')

text = "translate English to German: How old are you?"

input_ids = tok.encode(text, return_tensors='pt')
outputs = model.generate(input_ids, bos_token_id=tok.pad_token_id, max_length=22, num_beams=4, do_sample=False, early_stopping=True)

print(tok.decode(outputs[0], skip_special_tokens=True))
# prints:
# Wie alt bist du?st du?st du?st
```

UPDATE:

Updated generate() in both TF and PT to compute the `encoder_outputs` only once for `encoder-decoder` models as discussed below.

Tests `RUN_SLOW=1` for `test_modeling_bart.py`, `test_modeling_gpt2.py` and `test_modeling_tf_gpt2.py` all pass.

#### **FUTURE PR**:
- [ ] add generation integration test for T5 in PT and TF (could be similar to what is done in 
e.g.  OR better compare numbers to original T5 model numbers @craffel.

Good for me to merge! 
@thomwolf @sshleifer @craffel

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=h1) Report
> Merging [#3228](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/68ef0a111f8740f06ca4e5a00374ec4e2adb0a6d&el=desc) will **increase** coverage by `0.12%`.
> The diff coverage is `97.56%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3228/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3228      +/-   ##
==========================================
+ Coverage   77.48%   77.60%   +0.12%     
==========================================
  Files          99       99              
  Lines       16799    16828      +29     
==========================================
+ Hits        13017    13060      +43     
+ Misses       3782     3768      -14     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.91% <ø> (ø)` | |
| [src/transformers/configuration\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3Q1LnB5) | `96.55% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `93.28% <ø> (+3.35%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `96.17% <94.20%> (-0.37%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `75.47% <100.00%> (ø)` | |
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `96.26% <100.00%> (+<0.01%)` | :arrow_up: |
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `81.20% <100.00%> (+0.10%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `68.62% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.82% <100.00%> (+1.60%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `93.99% <100.00%> (+0.28%)` | :arrow_up: |
| ... and [2 more](https://codecov.io/gh/huggingface/transformers/pull/3228/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=footer). Last update [68ef0a1...62cf76f](https://codecov.io/gh/huggingface/transformers/pull/3228?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> I noticed that the T5 tokenizer does not have a BOS token and since we require at the moment to use a BOS token for encoder-decoder generation, I set the bos_token_id to the pad_token_id which is probably not the best way to do it.

This is actually the correct thing to do, see e.g. https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/transformer.py#L1744

What is currently happening in `model.generation()` for encoder-decoder models (Bart and T5) is the following: 

The `input_ids` variable of the generate() is given to the variable `encoder_input_ids`, which is then **always** put into the forward() of `BartLMHeadModel` and `T5LMHeadModel`. The `input_ids` variable is then initialized with the `BOS` token and auto-regressively updated. 

After the first step the `encoder_output_ids` are calculated and handed to the `past` variable, which from then on is also **always** put into forward() of `BartLMHeadModel` and `T5LMHeadModel`. 

At the moment, the `encoder_input_ids` are always put in the forward() of Bart and T5 and then ignored there. This is probably not the cleanest way to do it. Other possibilities might be:
1. calculate the encoder_outputs one time before going into the auto-regressive loop and setting them to the `past` variable already on the first step. 
2. leave it as it is now, but set `encoder_input_ids` to None in `prepare_inputs_for_generation()`

Or other ideas? 

I think option 1 is clean - I think Bart and T5 let's you calculate only the encoder_outputs @sshleifer , @craffel  no?

@craffel  @thomwolf @sshleifer

> calculate the encoder_outputs one time before going into the auto-regressive loop and setting them to the past variable already on the first step.

This definitely seems best - the model should compute the encoder outputs and then treat them as fixed (to be attended to) as the decoder generates.

UPDATE 2: 

I'm pretty happy with the current version now. 

To summarize:
This PR allows `generate()` for TF & PT T5Model.

Three important changes to mention:

1. remove the if `decoder_start_token_id` != `bos_token_id` statement in `generate()` to keep generate() generic (for more explanation, see comments above)
2.  remove `encoder()` abstraction method in `Bart` and `T5` and replace by `get_encoder()`. 
3. **IMPORTANT**: Move responsablitiy to transform `input_ids` to `inputs_embeds` from `T5ForConditionalGeneration.call()` to `encoder.call()` and `decoder.call()` -> Reasons:
           a) This way, the encoder is a complete model which can transform `input_ids` to `input_embeds`
           b) cleaner code in `T5ForConditionalGeneration.call()`
           c) Bart had this behavior already implemented - make API more similar
    NOTE: this led to some problems with TF scopes, but thanks to @mfuntowicz and  @craffel is solved now by injecting the correct absolute scope to the call method and wrapping the Embedding layer (see comments above). This will issue will also be important when translating Bart to T5Bart @sshleifer 

4. `T5Models.call()` arguments are renamed to `BartModel` argument names.

T5 produces same good translation results (same as results mentioned on the top) and Bart tests all pass. 

@craffel @thomwolf @sshleifer 

when i use decoder of bart to call generate(), there's mistake of 'has no attribute 'get_encoder',and the decoder is a tensorRT engine Inherited from GenerationMixin.
![截屏2022-01-13 下午5 12 00](https://user-images.githubusercontent.com/13781668/149300420-6c3373dc-528b-412c-849d-863e01e1bc70.png)

Is any one knows how to fix it? very many thanks!
@patrickvonplaten @craffel @codecov-io @jplu 

> when i use decoder of bart to call generate(), there's mistake of 'has no attribute 'get_encoder',and the decoder is a tensorRT engine Inherited from GenerationMixin. ![截屏2022-01-13 下午5 12 00](https://user-images.githubusercontent.com/13781668/149300420-6c3373dc-528b-412c-849d-863e01e1bc70.png)
> 
> Is any one knows how to fix it? very many thanks! @patrickvonplaten @craffel @codecov-io @jplu

@yuanhuachao - could you please open a new issue for this?