[Reformer] Improved memory if input is shorter than chunk length
Author: patrickvonplaten
Pull request: yes

This PR improves memory and speed of Reformer for language generatinon. 

Reformer is based on chunked self attention. This means that for an input length which is not a multiple of the chunk length, the input has to be padded to be a multiple of the chunk length. This is not the case though when the input length is less than the chunk length (happens in language generation). In this case, normal self attention should be applied to save memory. 

Code is updated for both LSH and Local self attention and a test is added.



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=h1) Report
> Merging [#4720](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/47a551d17b6ed2eaf03301f049006d559fca5cf3&el=desc) will **decrease** coverage by `0.56%`.
> The diff coverage is `96.92%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4720/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4720      +/-   ##
==========================================
- Coverage   77.14%   76.57%   -0.57%     
==========================================
  Files         128      128              
  Lines       21073    21089      +16     
==========================================
- Hits        16256    16149     -107     
- Misses       4817     4940     +123     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4720/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `89.94% <ø> (ø)` | |
| [src/transformers/modeling\_reformer.py](https://codecov.io/gh/huggingface/transformers/pull/4720/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yZWZvcm1lci5weQ==) | `88.21% <96.92%> (+0.26%)` | :arrow_up: |
| [src/transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/4720/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19ncHQyLnB5) | `31.73% <0.00%> (-40.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4720/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.50% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4720/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.78% <0.00%> (+1.37%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=footer). Last update [47a551d...6fe9553](https://codecov.io/gh/huggingface/transformers/pull/4720?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).