Improve special_token_id logic in run_generation.py and add tests
Author: patrickvonplaten
Pull request: yes

This PR finally implements the following `bos_token_id, pad_token_id, eos_token_ids` logic for lm model generation.

1. If `bos_token_id` is None, then the input_ids must be defined otherwise, the model cannot generate text, which is checked by the asserts in the beginning. The `bos_token_id` is only relevant for starting a new sentence.

2. If `eos_token_id` is None, then the length of the generated text will always equal max_length,
no matter how the pad_token_id is defined. Since there is no `eos_token_id` the text will also not "end".

3. If `pad_token_id` is None and `eos_token_ids` is defined (as it is the case for gpt2), then the pad_token_id will be set to the `eos_token_ids[0]` tensor `batches_len` is used to keep track of the first time the sequence generated an eos_token and will later set all tokens following this token to the `pad_token_id`, which is `eos_token_ids[0]` and can thus be handled by the tokenizer (whereas the -1 cannot be handled by the tokenizer).

4. **No** eos_token_id is appended to sentences that finish due to `max_length`. Instead those sentences are returned with the last token being the last token produced by the model until `max_length` was hit.

As an overview, here a table showing which LMModel Tokenizer have which of the tokens `bos_token_id`, `pad_token_id` and `eos_token_ids` is defined:


| LM Model  | bos_token_id | pad_token_id | eos_token_ids |
| ------------- | ------------- | ------------- | ------------- | 
| XLNet  | x | x | x |
| OpenAIGPT  | o | o | o |
| CTRL  | o | o | o |
| GPT2  | x | o | x |
| Transfo-XL  | o | o | x |
| XLM  | x | x | o |


Testing times are increased by the following times (measured on a local machine):

| LM Model  | Increase in test time |
| ------------- | ------------- |
| XLNet  | 8.0s -> 9.7s |
| OpenAIGPT  | 7.1s -> 8.3s |
| CTRL  | 2.5s -> 4.3s |
| GPT2  | 7.3s -> 8.0s |
| Transfo-XL  | 7.5s -> 8.0s |
| XLM  | 7.4s -> 7.7s |

-> So overall mostly around 10% increase in testing time

## Future PRs:

- [x] [WIP] adding hard-coded slow tests for pretrained lms in PR #2909
- [x] [WIP] adapting the `generate` function for Seq-2-Seq and DoubleHeads or other special LM models in PR #2888 
- [x] checking and possibly adapting behavior of `generate_beam_search`
- [x] treat Issues: #2482 and #2415

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=h1) Report
> Merging [#2885](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d490b5d5003654f104af3abd0556e598335b5650?src=pr&el=desc) will **increase** coverage by `1.75%`.
> The diff coverage is `86.22%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2885/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #2885      +/-   ##
=========================================
+ Coverage   75.35%   77.1%   +1.75%     
=========================================
  Files          94      98       +4     
  Lines       15444   15971     +527     
=========================================
+ Hits        11638   12315     +677     
+ Misses       3806    3656     -150
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19kaXN0aWxiZXJ0LnB5) | `97.59% <ø> (-0.04%)` | :arrow_down: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.99% <ø> (ø)` | :arrow_up: |
| [src/transformers/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fZ3B0Mi5weQ==) | `96.82% <ø> (-0.03%)` | :arrow_down: |
| [src/transformers/modeling\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19lbmNvZGVyX2RlY29kZXIucHk=) | `26.66% <ø> (+1.36%)` | :arrow_up: |
| [src/transformers/data/processors/utils.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvdXRpbHMucHk=) | `21.73% <ø> (ø)` | :arrow_up: |
| [src/transformers/utils\_encoder\_decoder.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy91dGlsc19lbmNvZGVyX2RlY29kZXIucHk=) | `0% <0%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `75.47% <100%> (+0.23%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.2% <100%> (+30.87%)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.45% <100%> (-0.14%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.71% <100%> (-0.07%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=footer). Last update [d490b5d...80ca73d](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> Love the tests!
> 
> Do we ever test cases:
> 
> * eos_token_ids are None, pad_token_id present
> * pad_token_id=None, eos_ids present
> * pad_token_id present, eos_ids None
> 
> We should also figure out if those are realistic scenarios. Because if they are not we can delete a lot of code!

eos_token_ids present and pad_token_id = None -> GPT2, so this scenario is tested. This case is the hardest to handle for batch_size > 1, therefore quite a lot of assert statements and a warning in modeling_utils.py

eos_token_ids = None and pad_token_id present -> If eos_token_ids = None, the pad_token_id is somewhat irrelevant for generation because all batches will always be of max. length and therefore never will have to be padded (they can't finish because no eos_token can be generated)



This PR finally implements the following `bos_token_id, pad_token_id, eos_token_ids` logic for lm model generation.

1. If `bos_token_id` is None, then the input_ids must be defined otherwise, the model cannot generate text, which is checked by the asserts in the beginning. The `bos_token_id` is only relevant for starting a new sentence.

2. If `eos_token_id` is None, then the length of the generated text will always equal max_length,
no matter how the pad_token_id is defined. Since there is no `eos_token_id` the text will also not "end".

3. If `pad_token_id` is None and `eos_token_ids` is defined (as it is the case for gpt2), then the pad_token_id will be set to the `eos_token_ids[0]` tensor `batches_len` is used to keep track of the first time the sequence generated an eos_token and will later set all tokens following this token to the `pad_token_id`, which is `eos_token_ids[0]` and can thus be handled by the tokenizer (whereas the -1 cannot be handled by the tokenizer).

4. **No** eos_token_id is appended to sentences that finish due to `max_length`. Instead those sentences are returned with the last token being the last token produced by the model until `max_length` was hit.

As an overview, here a table showing which LMModel Tokenizer have which of the tokens `bos_token_id`, `pad_token_id` and `eos_token_ids` is defined:


| LM Model  | bos_token_id | pad_token_id | eos_token_ids |
| ------------- | ------------- | ------------- | ------------- | 
| XLNet  | x | x | x |
| OpenAIGPT  | o | o | o |
| CTRL  | o | o | o |
| GPT2  | x | o | x |
| Transfo-XL  | o | o | x |
| XLM  | x | x | o |

## Future PRs:

- [x] [WIP] adding hard-coded slow tests for pretrained lms in PR #2909
- [ ] [WIP] adapting the `generate` function for Seq-2-Seq and DoubleHeads or other special LM models in PR #2888 
- [x] checking and possibly adapting behavior of `generate_beam_search`
- [x] treat Issues: #2482 and #2415

> # [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=h1) Report
> > Merging [#2885](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/59c23ad9c931ac4fe719abeb3c3851df046ef3a6?src=pr&el=desc) will **increase** coverage by `1.49%`.
> > The diff coverage is `100%`.
> 
> [![Impacted file tree graph](https://camo.githubusercontent.com/0d9954474a38f8b8a437b85ef46677aec2ec2f8a/68747470733a2f2f636f6465636f762e696f2f67682f68756767696e67666163652f7472616e73666f726d6572732f70756c6c2f323838352f6772617068732f747265652e7376673f77696474683d36353026746f6b656e3d39714f6c4e3648623163266865696768743d313530267372633d7072)](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=tree)
> 
> ```diff
> @@            Coverage Diff             @@
> ##           master    #2885      +/-   ##
> ==========================================
> + Coverage    75.3%   76.79%   +1.49%     
> ==========================================
>   Files          94       94              
>   Lines       15424    15448      +24     
> ==========================================
> + Hits        11615    11864     +249     
> + Misses       3809     3584     -225
> ```
> 
> [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=tree)	Coverage Δ	
> [src/transformers/modeling_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==)	`92.14% <100%> (+30.81%)`	
> [src/transformers/configuration_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5)	`96.46% <100%> (ø)`	
> [src/transformers/modeling_transfo_xl.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5)	`75.63% <0%> (+0.84%)`	
> [src/transformers/modeling_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG0ucHk=)	`88.43% <0%> (+2.05%)`	
> [src/transformers/modeling_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==)	`75.77% <0%> (+2.61%)`	
> [src/transformers/modeling_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19ncHQyLnB5)	`86.11% <0%> (+2.83%)`	
> [src/transformers/modeling_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2885/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5)	`98.23% <0%> (+3.96%)`	
> [Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=continue).
> 
> > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> > Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=footer). Last update [59c23ad...ac2e172](https://codecov.io/gh/huggingface/transformers/pull/2885?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I'm a bit confused that the coverage for the file modeling_openai.py did not change. Tests for the openai lm were added but they had seemingly no effect - do you know why? @LysandreJik 

No need to do all my `torch.Tensor` vs .new comments, TIL that it lets you copy the device and dtype of the first tensor.