Fixed use of memories in XLNet (caching for language generation + warning when loading improper memoryless model)
Author: TevenLeScao
Pull request: yes

The default XLNet model is loaded with 0 memory length, which is an issue both at training time (improper performance) and inference time (as there's no caching speed-up since it doesn't return former attentions). As discussed with @LysandreJik , this PR introduces a warning that in the future, the default XLNet model will have 1024 memory length, in accordance with [the original paper](https://arxiv.org/abs/1906.08237). It also fixes the re-use of cached memory, which was broken similarly to TransfoXL (#4752).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=h1) Report
> Merging [#5632](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa5423b1695cd24856bcff47214172e0f540d924&el=desc) will **decrease** coverage by `0.07%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5632/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5632      +/-   ##
==========================================
- Coverage   77.79%   77.72%   -0.08%     
==========================================
  Files         145      145              
  Lines       25355    25364       +9     
==========================================
- Hits        19726    19715      -11     
- Misses       5629     5649      +20     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3hsbmV0LnB5) | `94.33% <100.00%> (+0.33%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG5ldC5weQ==) | `91.39% <100.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.96% <100.00%> (+0.10%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9kaXN0aWxiZXJ0LnB5) | `64.90% <0.00%> (-33.90%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.46% <0.00%> (+0.50%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/5632/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9yb2JlcnRhLnB5) | `93.36% <0.00%> (+49.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=footer). Last update [fa5423b...25fae1b](https://codecov.io/gh/huggingface/transformers/pull/5632?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Also the slow test of XLNet will have to be adapted when doing this change no?

So in order:

a) b) I would really like to to have proper GPT-2 style caching in XLNet. This would require changing the outputs of the XLNet `forward` to add `presents` outputs that contain the K/V pairs, like in GPT-2. That probably counts as BC-breaking, right @LysandreJik ?

c) `offset = 2` is definitely a bit random. I was mistaken at first, as I thought it would be proper autoregressive generation, but `offset=1` is what should be used in this case. After manually checking outputs I had the impression that `offset = 2` was slightly better (mostly it goes less into repetitive generation loops) at a negligible computation time cost; but I agree that `offset = 1` is more principled and I don't have a strong opinion on that choice.

d) The slow tests run with the default model, which still has `mem_length = 0` and no caching, so it doesn't make a difference yet. 

> So in order:
> 
> a) b) I would really like to to have proper GPT-2 style caching in XLNet. This would require changing the outputs of the XLNet `forward` to add `presents` outputs that contain the K/V pairs, like in GPT-2. That probably counts as BC-breaking, right @LysandreJik ?
> 
> c) `offset = 2` is definitely a bit random. I was mistaken at first, as I thought it would be proper autoregressive generation, but `offset=1` is what should be used in this case. After manually checking outputs I had the impression that `offset = 2` was slightly better (mostly it goes less into repetitive generation loops) at a negligible computation time cost; but I agree that `offset = 1` is more principled and I don't have a strong opinion on that choice.
> 
> d) The slow tests run with the default model, which still has `mem_length = 0` and no caching, so it doesn't make a difference yet.

a)b) I think would count as a feature enhancement. We would make the `past` variable optional so no backward breaking here IMO. But I agree it would definitely be better to this in a new PR.

I've added a comment. I'm preparing another PR for proper caching and merging this one.