XLNet `use_cache` refactor
Author: TevenLeScao
Pull request: yes

As discussed with @joeddav and @sgugger, this PR lightly refactors the `use_cache` argument in XLNet.

- In line with #5438, in the model methods `use_cache` defaults to None, which redirects to the model config value if no value is passed.
- `use_cache` is independent of `mem_len`: if `use_cache` is True and `mem_len` is 0 or None (which is the case in the base model config), the model behaves like GPT-2 and returns `mems` to be used as `past` in generation.
- This changes functionality and will enable the default model to use caching; for instance, it should speed up the [inference widget](https://huggingface.co/xlnet-base-cased?text=My+name+is+Clara+and+I+am+) significantly (x3 speed-up on my CPU)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=h1) Report
> Merging [#5770](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/3653d01f2af0389207f2239875a8ceae41bf0598&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5770/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5770      +/-   ##
==========================================
- Coverage   77.26%   77.25%   -0.02%     
==========================================
  Files         146      146              
  Lines       25948    25958      +10     
==========================================
+ Hits        20048    20053       +5     
- Misses       5900     5905       +5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3V0aWxzLnB5) | `96.52% <ø> (ø)` | |
| [src/transformers/configuration\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX3hsbmV0LnB5) | `94.33% <ø> (ø)` | |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `81.75% <100.00%> (+0.26%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5770/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `85.21% <0.00%> (-1.26%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=footer). Last update [3653d01...cfd7c66](https://codecov.io/gh/huggingface/transformers/pull/5770?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Not sure what is going on with Cicle CI here - might be related to the problems with github yesterday...

Yeah it doesn't want to let me re-run, I think I'll just add a small change to refresh it

After our discussion yesterday @TevenLeScao, I think I am more or less fine with this PR. A couple of things we should maybe think about:

1) ***Consistency with `use_cache` of other models***
As it is implemented at the moment (correct me if I'm wrong), if `use_cache=False`, `mem_len` has no effect as `mems` will always be `None`. This is not ideal IMO because this means that one has to enable `use_cache=True` to correctly train the model => this is not really consistent with the way `use_cache` is used in the library in general. In all other models GPT2, CTRL, T5, BART, Reformer `use_cache` should only be used for inference and disabled for training. So as a solution I think, there are two options:
    A.) All `use_cache` statements should be updated with `use_cache or self.training` => this way I think it would be consistent with the logic / design of other models (Let me know if this is not clear)
    B.) Add an explicit warning that `use_cache` should be `True` for training to enable the `mems` => I don't like this very much because this is confusing for people that are used to GPT2 where `use_cache` is only for inference. 

2)  ***Equality `use_cache=True/False`*** 
After our discussion yesterday, it seems that since `XLNet` uses a mixture of CLM and bi-directional self-attention (defined in the `perm_mask` input), it seems that it is **not** possible to have a (mathematically) identical output between `use_cache=True` and `use_cache=False`. I guess that's just how it is in `XLNet` and we can't really do anything against it. 

   - A.) I guess we should change (...could be done in a 2nd PR though), is to add the parameter `offset` to `XLNet` config and add it to all `forward(...)` functions and maybe give it a better name... As I understood it yesterday, `offset` defines the trade-off between how many tokens should be used from `mems` (and are therefore not in the *query projection*) and how many are in the query projection and have a bi-directional mask on them. At the moment `offset` is hard-coded to `2` in the `prepare_generate` function which does not allow for a lot of flexibility and is somewhat arbitrary IMO. I think, this parameter should be handled in a similar way as `num_hashes` is handled in `Reformer`

- B.) This is one is more important to me. We should add a test to `XLNet` that verifies that `use_cache` does give the **same** outputs if the `perm_mask` is set to a causal mask. We have tests for this in `GPT2, T5, Bart` I think. Here is the one for GPT2 e.g.: https://github.com/huggingface/transformers/blob/1f75f9317e381425ee56f7108e5ec8d3f3d6b6ad/tests/test_modeling_gpt2.py#L165 . The test here can be very similar - only that we need to define the `perm_mask` correctly. This test would be extremely useful a) to make sure we fully understand what's going on in `XLNet` and b) make sure that there is no bug.

3) ***Benchmark performance improvement***

Soon we will have benchmarking for the `generate` function, see PR here: https://github.com/huggingface/transformers/pull/5802 . So I took this case here as a good trial to test the benchmark scripts for generation. I started two benchmarks 1 on CPU 1 on GPU and will post the results here as soon as the benchmarking is done. It be great to add benchmark results in general to PRs like this.


I think this is quite an important Model and an important PR, so I'd love to loop in @sshleifer and @thomwolf here as well to hear their opinions. 

@TevenLeScao - sorry for the long message! It's mostly because I want to better understand `XLNet` in detail. Let me know if some of my points are not clear or not 100% correct.

I used the following code for benchmarking:

```python

#!/usr/bin/env python3
from transformers import XLNetConfig, PyTorchBenchmark, PyTorchBenchmarkArguments

config_with_cache = XLNetConfig.from_pretrained("xlnet-base-cased", use_cache=True, mem_len=1024)
config_without_cache = XLNetConfig.from_pretrained("xlnet-base-cased", use_cache=False, mem_len=1024)
config_without_cache_no_mems = XLNetConfig.from_pretrained("xlnet-base-cased", use_cache=False)

assert config_without_cache_no_mems.mem_len is None, "Configs are wrong"
assert config_with_cache.mem_len == config_without_cache.mem_len == 1024, "Configs are wrong"
assert config_with_cache.use_cache is (not config_without_cache.use_cache) is True and (not config_without_cache_no_mems.use_cache) is True, "Configs are wrong"

args = PyTorchBenchmarkArguments(models=["xlnet-cache", "xlnet-no-cache", "xlnet-no-mems"], sequence_lengths=[300], batch_sizes=[1], generate=True, no_inference=True)

benchmark = PyTorchBenchmark(args=args, configs=[config_with_cache, config_without_cache, config_without_cache_no_mems])
benchmark.run()


```
and the benchmark code from #5802 


=> I could not see a real speed-up on GPU for the following case:
Start with 1 token and generate up to `Seq Length` tokens:

 INFERENCE - SPEED - RESULT
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
         xlnet-cache                 1              500             11.8     
        xlnet-no-cache               1              500            11.925    
        xlnet-no-mems                1              500            11.911    
--------------------------------------------------------------------------------

  INFERENCE - MEMORY - RESULT 
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
         xlnet-cache                 1              500             1351     
        xlnet-no-cache               1              500             1397     
        xlnet-no-mems                1              500             1397     
--------------------------------------------------------------------------------

  ENVIRONMENT INFORMATION 
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 11:28:43.644495
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False

=> For CPU, there is a small speed up and the speed up is probably more significant for longer sequence length (so I run it again for longer sequences):

INFERENCE - SPEED - RESULT
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
         xlnet-cache                 1              50             5.032     
        xlnet-no-cache               1              50            6.149    
        xlnet-no-mems                1              50            6.175    
--------------------------------------------------------------------------------

INFERENCE - MEMORY - RESULT 
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------
         xlnet-cache                 1              50             691     
        xlnet-no-cache               1              50             695     
        xlnet-no-mems                1              50             696     
--------------------------------------------------------------------------------

ENVIRONMENT INFORMATION
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 11:28:43.644495
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: False
- use_tpu: False


But overall, the speed-up is not really significant... @TevenLeScao can you post the code you used to benchmark the CPU speed-up here? Maybe I am doing something wrong.

I think if we manage to cache the key, value projections instead of recalculating them for every token we would see a bigger speed-up, but this has to checked.

- on GPU I also haven't observed a difference, and doing inference/text generation on 1 sentence isn't going to be very affected by the caching trick since the GPU is going to parallelize everything anyway
- on CPU I've run from the text generation pipeline:

```
#!/usr/bin/env python3
from transformers import pipeline, XLNetLMHeadModel
import time

# xlnet = XLNetLMHeadModel.from_pretrained("xlnet-base-cased")
xlnet = XLNetLMHeadModel.from_pretrained("xlnet-base-cased", use_cache=False)
generator = pipeline("text-generation", model=xlnet, tokenizer="xlnet-base-cased")
start = time.time()
number_gen = 100
for i in range(number_gen):
    output_to_check = generator("Today is a beautiful day and I, ")
    print(output_to_check)
    print((time.time() - start) / (i+1))
    print()
```

which makes I think a big (33.2 vs 8.6 seconds on my CPU) difference since XLNet has the 170-long padding text at the start! Which we should probably also take a look at since it heavily influences the generated text...

I can confirm a much bigger speed-up on CPU for longer sequences. Here the results:

====================       INFERENCE - SPEED - RESULT       ====================             
--------------------------------------------------------------------------------                         
          Model Name             Batch Size     Seq Length     Time in s                            
--------------------------------------------------------------------------------
         xlnet-cache                 1              300            37.733                  
        xlnet-no-cache               1              300            98.142                                
        xlnet-no-mems                1              300            94.338                 
--------------------------------------------------------------------------------                         
                                                                                                         
====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB 
--------------------------------------------------------------------------------                         
         xlnet-cache                 1              300             728                                  
        xlnet-no-cache               1              300             752                             
        xlnet-no-mems                1              300             750                              
--------------------------------------------------------------------------------                         
                                                                                                         
====================        ENVIRONMENT INFORMATION         ====================                         
- transformers_version: 3.0.2                                                                            
- framework: PyTorch                                                                                     
- use_torchscript: False                            
- framework_version: 1.5.0 
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit                                                                                    
- date: 2020-07-16                                                                                       
- time: 13:49:43.409581                                                                                  
- fp16: False                                                                                            
- use_multiprocessing: True                                                                              
- only_pretrain_model: False                                                                             
- cpu_ram_mb: 32089                                                                                      
- use_gpu: False                                                                                         
- use_tpu: False


@patrickvonplaten I added the `self.training` check and the test for `use_cache` in autoregressive mode.

So weird, it passed the tests earlier but the CI just told me the push had been refused, I'll revert it and look at it again tomorrow