[T5, generation] Add decoder caching for T5
Author: patrickvonplaten
Pull request: yes

This PR greatly speeds up the autoregressive decoding for T5 by storing past key / value states.
The summarization test: https://github.com/huggingface/transformers/blob/500aa12318ce5acd289d5edb6cb8266b3c3b162e/tests/test_modeling_t5.py#L260 now takes only 44s whereas before it took 311s -> 7.5x Speed up

This will also significantly speed up the translation and summarization pipelines when using T5.

- [x] Add key value state caching
- [x] Test for equal output on hard-coded tests
- [x] Add simple past tests including using an attention mask
- [x] update the docstring
- [x] clean up code

The caching design was already somewhat outcommented in place. It was cleaned, made functional and implemented very similar to GPT2's one.

### IMPORTANT: 
This PR has a breaking change, in that it increases the default output length of T5Model and T5ForConditionalGeneration from 4 to 5 (including the `past_key_value_states`).

### Future PR:
- [ ] Do the same for TF if this PR is accepted.

Would be nice if you could take a look @craffel @thomwolf @LysandreJik @sshleifer 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=h1) Report
> Merging [#3682](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a594ee9c84dde933a3d0b4e07ff2994a1960574c&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `89.76%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3682/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3682      +/-   ##
==========================================
+ Coverage   78.02%   78.09%   +0.07%     
==========================================
  Files         104      104              
  Lines       17710    17786      +76     
==========================================
+ Hits        13818    13890      +72     
- Misses       3892     3896       +4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3682/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `83.21% <89.68%> (+1.72%)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3682/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.03% <100.00%> (+0.18%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=footer). Last update [a594ee9...67ae81f](https://codecov.io/gh/huggingface/transformers/pull/3682?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).