Longformer
Author: ibeltagy
Pull request: yes

## Adding the Longformer model
Paper: https://arxiv.org/abs/2004.05150
The code is mostly copied from https://github.com/allenai/longformer
Addressing issue: https://github.com/huggingface/transformers/issues/3783

**TODOs**
- [x] Add unit tests
- [x] Address PR reviews
- [x] Upload model to HF instead of AllenAI s3 buckets. 

**TODOs for other PRs**
- Add `LongformerForSequenceClassification`, `LongformerForMultipleChoice`, `LongformerForTokenClassification`, `LongformerForQuestionAnswering` with automatic setting of global attention based on the task.
- Add `LongformerEncoderDecoder`
- Add an example to show how to convert existing pretrained models into their long version.

Comments:
This is awesome @ibeltagy . I´m very excited about longformer.

In the original implementation a good speedup was thanks to the personalized kernel in tvm. In this pull, this speedup is lost, or it was adapted somehow?

Thanx @ibeltagy! 

Is it possible for you or someone else who is familiar with TF2 to submit also a PR for TF2? I tried to migrate your code in the past but I couldn’t migrate the new attention mechanism written for TVM, so I moved on. Your work alongside Reformer has special interest for people like me and my colleagues, who are usually doing research with long documents (>=1000words). 

Maybe @thomwolf and @julien-c know the right person...


@ibeltagy - Looks awesome the PR :-) I added some minor comments. 

I think we can merge this very soon :-) 

To pass the check_code_qualtiy test, you can run `make style` from the root folder. 

Regarding the unit test, I think you can copy-paste a lot of the code from `test_roberta_modeling.py`. In order to skip tests for `torch_script, pruning, ..` (features we can add in another PR) you can do the same as is done in the reformer tests here: https://github.com/huggingface/transformers/blob/94cb73c2d2efeb188b522ff352f98b15124ba9f8/tests/test_modeling_reformer.py#L510 . 

If possible it would also be great to add some integration tests comparing the outputs (and maybe even gradients) to those of your original model code. I think you could use the same integration test structure as was done for Roberta or Reformer.

@bratao , @iliaschalkidis, about the custom CUDA kernel. We [recently](https://github.com/allenai/longformer/pull/27/files#diff-04c6e90faac2675aa89e2176d2eec7d8R4) added an efficient PyTorch implementation of our sliding window attention (without dilation), which is faster and uses less memory (with fp16) and should be easy to port to TF. The TVM code is still needed for the dilated sliding window attention, but dilation is more important for char-lm than fine-tuning on downstream tasks (QA, classification .. etc). This PR only uses the PyTorch code.

Thanks, @patrickvonplaten. I will address your comments and let you know. 

I also want to add `LongformerForSequenceClassification`, `LongformerForMultipleChoice`, `LongformerForTokenClassification`, `LongformerForQuestionAnswering` with automatic setting of global attention based on the task, so that the user doesn't need to worry about it. I will keep this to a different PR unless you think it is better to add it here. 

> Thanks, @patrickvonplaten. I will address your comments and let you know.
> 
> I also want to add `LongformerForSequenceClassification`, `LongformerForMultipleChoice`, `LongformerForTokenClassification`, `LongformerForQuestionAnswering` with automatic setting of global attention based on the task, so that the user doesn't need to worry about it. I will keep this to a different PR unless you think it is better to add it here.

It'd be great to add those as well :-) I think doing it in a second PR would be better as well

@patrickvonplaten, @sshleifer, I am done addressing your reviews. I still need to upload model weights but waiting to get access to the AllenAI organization account. Also, can you check the failed tests? they seem like issues with CircleCI rather than the code. 


@sshleifer, addressed your comments. Now `ci/circleci: run_tests_torch_and_tf` is timing out.

I'm sorry.

Rebase may help the timeout, but I agree that it does not appear to be caused by this code.

I'd just verify that there aren't new tests that should be decorated `@slow` (append `--durations 0` to your local pytest command) and it should be fine.

The slowest one is 0.10s. 
```
0.10s call     tests/test_modeling_longformer.py::LongformerModelTest::test_save_load
0.10s call     tests/test_modeling_longformer.py::LongformerModelTest::test_attention_outputs
0.07s call     tests/test_modeling_longformer.py::LongformerModelTest::test_resize_tokens_embeddings
0.06s call     tests/test_modeling_longformer.py::LongformerModelTest::test_determinism
0.05s call     tests/test_modeling_longformer.py::LongformerModelTest::test_inputs_embeds
0.04s call     tests/test_modeling_longformer.py::LongformerModelTest::test_correct_missing_keys
0.04s call     tests/test_modeling_longformer.py::LongformerModelTest::test_hidden_states_output
0.04s call     tests/test_modeling_longformer.py::LongformerModelTest::test_longformer_model
0.03s call     tests/test_modeling_longformer.py::LongformerModelTest::test_initialization
0.03s call     tests/test_modeling_longformer.py::LongformerModelTest::test_longformer_for_masked_lm
0.02s call     tests/test_modeling_longformer.py::LongformerModelTest::test_model_common_attributes
```

Maybe the integration tests are slower than average because of the long document? 

@patrickvonplaten, @sshleifer, looks like rebasing fixed the problem. Thanks.

> ## Adding the Longformer model
> Paper: https://arxiv.org/abs/2004.05150
> The code is mostly copied from https://github.com/allenai/longformer
> Addressing issue: #3783
> 
> **TODOs**
> 
> * [x]  Add unit tests
> * [x]  Address PR reviews
> * [ ]  Upload model to HF instead of AllenAI s3 buckets.
> 
> **TODOs for other PRs**
> 
> * Add `LongformerForSequenceClassification`, `LongformerForMultipleChoice`, `LongformerForTokenClassification`, `LongformerForQuestionAnswering` with automatic setting of global attention based on the task.
> * Add `LongformerEncoderDecoder`
> * A few small TODOs in the code
> * Add an example to show how to convert existing pretrained models into their long version.

Is there any particular timeframe by when we can have LongformerForQuestionAnswering? Would love to use it in a small project of mine.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@18d233d`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `84.48%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4352/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master    #4352   +/-   ##
=========================================
  Coverage          ?   78.30%           
=========================================
  Files             ?      123           
  Lines             ?    20373           
  Branches          ?        0           
=========================================
  Hits              ?    15953           
  Misses            ?     4420           
  Partials          ?        0           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19sb25nZm9ybWVyLnB5) | `82.75% <82.75%> (ø)` | |
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `99.13% <100.00%> (ø)` | |
| [src/transformers/configuration\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2F1dG8ucHk=) | `92.68% <100.00%> (ø)` | |
| [src/transformers/configuration\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2xvbmdmb3JtZXIucHk=) | `100.00% <100.00%> (ø)` | |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `78.57% <100.00%> (ø)` | |
| [src/transformers/tokenization\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYXV0by5weQ==) | `97.61% <100.00%> (ø)` | |
| [src/transformers/tokenization\_longformer.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbG9uZ2Zvcm1lci5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/transformers/tokenization\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fb3BlbmFpLnB5) | `83.72% <0.00%> (ø)` | |
| [src/transformers/configuration\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2JhcnQucHk=) | `93.47% <0.00%> (ø)` | |
| [src/transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9iZXJ0LnB5) | `96.54% <0.00%> (ø)` | |
| ... and [120 more](https://codecov.io/gh/huggingface/transformers/pull/4352/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=footer). Last update [18d233d...d1c4bbc](https://codecov.io/gh/huggingface/transformers/pull/4352?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).