TF Longformer
Author: patrickvonplaten
Pull request: yes

This PR adds Longformer

In a first step, it is made sure that code is clean and that all tests pass. Todo:

### ToDo List:

- [x] same output for local attention only
- [x] same output for local + global attention only
- [x] same output for aggressive test
- [x] add all other tests
- [x] add longformer QA
- [x] refactor code and run benchmark
- [x] adds weights to all QA models and check performance via notebook

### ToDo after PR is merged:
- [ ] Add Longformer for SeqClass, MC, ...  ("good first issue")
- [ ] Speed up performance and make GPU XLA work -> use Benchmark tools and possible TF Profiler

### For Review

This PR adds `TFLongformer` and the two most important parent classes `TFLongformerForMaskedLM` and `TFLongformerForQuestionAnswering`.  Many tests are added to verify that TFLongformer gives identical results to PT Longformer and a colab notebook (see below) is attached to show performance on a real task.

Below you can find a Benchmark showing that TFLongformer is about 1.5x slower than PT on GPU. For now this is acceptable IMO, but in a future PR I want to take a deeper look at how TF code can be optimized and also solve a problem there is currently with TF XLA.

I spent a lot of time, trying to solve this issue: https://github.com/huggingface/transformers/issues/5815 for TFLongformer and didn't manage to find a good solution. The corresponding tests are in `SLOW` mode so they won't fail on this PR. Since we are currently thinking about a better solution than using `cast_bool_to_primitive` to solve the know TF graph boolean error, I think I will leave this small bug in TFLongformer for now (it's quite an edge IMO anymays). 

Docs are added and checked, comments are added, performance on TriviaQA is verified in TF colab: https://colab.research.google.com/drive/1UmU3T1nPmJ2LgXQtPcaEVtXUnoBJ1onF?usp=sharing and TF weights were added to all longformer models here: https://huggingface.co/models?search=longformer. 

Would be happy about a review @jplu @ibeltagy @LysandreJik @sshleifer @sgugger @julien-c @thomwolf 

Comments:
In `RUN_SLOW=1` the new tests: `test_saved_model_with_attentions_outputs` and `test_saved_model_with_hidden_states_output` fail @jplu from https://github.com/huggingface/transformers/pull/5468. The problem is that I have to use `tf.cond(...)` and it seems like this forces me to also use `cast_bool_...`. Not sure if you have any ideas on how to fix this @jplu .

Yes, It is still an issue with the AutoGraph thing :cry: I suggest to comment them for now.

I'm currently thinking on how to properly rework all the booleans handling in TF. As this is the main issue.

Ok...will leave it for now since the test are only in `RUN_SLOW` mode, so they won't show up in Circle CI

Confirmed that this PR will not slow down the Longformer PyTorch version.

Running the following benchmark on master:
```
python examples/benchmarking/run_benchmark.py --models allenai/longformer-base-4096 --no_memory --sequence_length 512 1024
```
gives same performance is in as in https://github.com/huggingface/transformers/pull/5811.

Benchmarking the model in TF leads to a slow-down vs. PyTorch of ca. 1.5, which is reasonable IMO:

Running: 
```
python examples/benchmarking/run_benchmark_tf.py --models allenai/longformer-base-4096 --no_memory --sequence_length 512 1024
```

gives:

```
                                                                                                                                                                                                                   
====================       INFERENCE - SPEED - RESULT       ====================                                                                                                                                   
--------------------------------------------------------------------------------                                                                                                                                   
          Model Name             Batch Size     Seq Length     Time in s                                                                                                                                           
--------------------------------------------------------------------------------                                                                                                                                   
 allenai/longformer-base-4096        8              512            0.226                                                                                                                                           
 allenai/longformer-base-4096        8              1024           0.446                                                                                                                                           
--------------------------------------------------------------------------------                                                                                                                                   

====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.2
- framework: TensorFlow
- eager_mode: False
- use_xla: False
- framework_version: 2.3.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-08-06
- time: 15:55:55.754100
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False
```

At the moment running the model in XLA on GPU fails...=> should take a closer look in a next PR.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=h1) Report
> Merging [#5764](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1b8a7ffcfdfe37f5440ac0eafb58089ff5aef00a&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `24.52%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5764/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5764      +/-   ##
==========================================
+ Coverage   79.33%   79.38%   +0.05%     
==========================================
  Files         148      149       +1     
  Lines       27196    27670     +474     
==========================================
+ Hits        21577    21967     +390     
- Misses       5619     5703      +84     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9sb25nZm9ybWVyLnB5) | `15.76% <15.76%> (ø)` | |
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `89.76% <92.98%> (+0.54%)` | :arrow_up: |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.25% <100.00%> (+<0.01%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `66.86% <100.00%> (+0.20%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `24.53% <0.00%> (-63.20%)` | :arrow_down: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `92.41% <0.00%> (+1.33%)` | :arrow_up: |
| [src/transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fb3BlbmFpLnB5) | `84.09% <0.00%> (+1.51%)` | :arrow_up: |
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `91.60% <0.00%> (+1.59%)` | :arrow_up: |
| ... and [7 more](https://codecov.io/gh/huggingface/transformers/pull/5764/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=footer). Last update [1b8a7ff...41cb64f](https://codecov.io/gh/huggingface/transformers/pull/5764?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Where do I send the pizza & beer to get this merged?

I'm curious if you think the longformer should be added to the language_generation_model? 

@sgugger - I added the `return_dict` functionality and adapted the doc strings -> the new docstring functions are awesome!

> I'm curious if you think the longformer should be added to the language_generation_model?

We will add it to the EncoderDecoderModel framework, where it can be used with `generate()`