Add support for past states
Author: sgugger
Pull request: yes

This adds support in the `Trainer` for models that can use past states. Since there is no way to know in advance where the output will be in the model (we can guess it's going to be the second, but that may not always be the case for all models or user-defined models), I added an argument to the `TrainingArguments` for the index of where to look at mems in the outputs.

If it's left to -1, nothing happens, and if it's set, the `Trainer` will save the past mems in its state at each training step and add them to the inputs on the next step.

If this looks good to you, I'll add the same thing on the TF side before merging.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=h1) Report
> Merging [#5399](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/87716a6d072b2b66415ce43086c73b04e63fe0fe&el=desc) will **decrease** coverage by `0.09%`.
> The diff coverage is `13.33%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5399/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5399      +/-   ##
==========================================
- Coverage   77.69%   77.60%   -0.10%     
==========================================
  Files         140      140              
  Lines       24334    24362      +28     
==========================================
- Hits        18906    18905       -1     
- Misses       5428     5457      +29     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `16.98% <0.00%> (-0.88%)` | :arrow_down: |
| [src/transformers/training\_args\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzX3RmLnB5) | `51.16% <ø> (ø)` | |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `38.81% <21.42%> (-0.58%)` | :arrow_down: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `77.08% <100.00%> (+0.24%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `20.78% <0.00%> (-74.20%)` | :arrow_down: |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `95.55% <0.00%> (-2.23%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.72% <0.00%> (-1.38%)` | :arrow_down: |
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `95.68% <0.00%> (-0.72%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.43% <0.00%> (ø)` | |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.22% <0.00%> (+0.31%)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/5399/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=footer). Last update [87716a6...7cce295](https://codecov.io/gh/huggingface/transformers/pull/5399?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I like it! This solution seems to be the cleanest one we have. We have to make sure that all models name their inner state `mems`. 
GPT2 has a `past` parameter, but it should not be used for training as far as I know. 

@jplu adding yo as a reviewer for the TF side of things :-)