[Longformer] fix longformer slow-down
Author: patrickvonplaten
Pull request: yes

A drastic slow-down of Longformer after the PR: https://github.com/huggingface/transformers/pull/5219 was deteced by @HHousen (thanks a lot!) here: https://github.com/huggingface/transformers/issues/4406#issuecomment-658483050 .

After digging a bit into the code it can be seen that the line:
`is_global_attn = any(is_index_global_attn.flatten())` is the reason of the drastic slow-down.

Running the following benchmark on master:
```
python examples/benchmarking/run_benchmark.py --models allenai/longformer-base-4096 --no_memory --sequence_length 512 1024
```

yields the following result:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
 allenai/longformer-base-4096        8              512            0.647     
 allenai/longformer-base-4096        8              1024           1.284     
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 14:07:02.719531
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False
```

Now on this branch the results are as follows:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
 allenai/longformer-base-4096        8              512            0.141     
 allenai/longformer-base-4096        8              1024           0.271     
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 14:07:57.623378
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False
```

So this simple line is responsible for a slow-down of factor 4.

Moral of the story: Never use `any()` on a PyTorch tensor => always use tensor.any(). I wonder if this is actually a known problem of PyTorch/Python. It might be a good to check our code if we have more statements like this. 

Another lesson for me is that one should always run the benchmarking before and after doing such a big refactoring as in https://github.com/huggingface/transformers/pull/5219 . 
It's very simple to run the benchmark script for a model and takes usually only a couple of seconds. Ideally we should have performance regression tests to automatically detect such slow downs.

Pinging @thomwolf @mfuntowicz @sshleifer @LysandreJik @ibeltagy - think this is quite interesting.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=h1) Report
> Merging [#5811](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/89a78be51f1c2afd263da66ec76d3297432f0c2a&el=desc) will **decrease** coverage by `0.80%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5811/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5811      +/-   ##
==========================================
- Coverage   78.19%   77.38%   -0.81%     
==========================================
  Files         146      146              
  Lines       26047    26047              
==========================================
- Hits        20367    20157     -210     
- Misses       5680     5890     +210     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `89.21% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_mobilebert.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9tb2JpbGViZXJ0LnB5) | `23.38% <0.00%> (-73.39%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `86.21% <0.00%> (ø)` | |
| [src/transformers/generation\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3V0aWxzLnB5) | `97.11% <0.00%> (+0.28%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `82.31% <0.00%> (+1.28%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5811/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `95.18% <0.00%> (+74.91%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=footer). Last update [89a78be...25c6419](https://codecov.io/gh/huggingface/transformers/pull/5811?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great find. +1 to tests!