very poor performance of Longformer on SQuAD-like question-answering tasks 
Author: xixiaoyao
Pull request: no
Labels: name: wontfix

## Environment info
any environment can easily re-produce the awful results     

- `transformers` version: newest
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.6
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: parallel

### Who can help
Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Longformer):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run examples/run_squad.py with Longformer-base-4096 Model (with fp16 and gradient_checkpointing enabled)

This results in very awful results than BERT, RoBERTa and any pre-trained models...

E.g., I run it on MARCO data, nothing modify the script. Then the BERT and RoBERTa can easily achieved 50+ F1 score on dev set, while Longformer can only reach **18** points! I believe there are severe bugs for the support of Longformer on SQuAD-like QA tasks. Please confirm that.

## Expected behavior

At least superior than RoBERTa, and marginal superior than BERT models

Comments:
Hey @xixiaoyao ,

We have a couple of Longformer models that have been fine-tuned on Squad and that yield good results, see here: https://huggingface.co/models?search=longformer .

Also, it might be helpful to take a look at this notebook: https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb . The notebook shows how to train Longformer for QA tasks.

I mean, when I train longformer based on the script `example/run_squad.py` you provided, the poor performance can be re-produced. And I have located that `squad_convert_example_to_features` function in `transformers/data/processors/squad.py` will split the document into word-level tokens, and then the longformer tokenizer will tokenize for each token rather than the whole document text, which behavior is not consistent with the pre-trained process of Longformer. 

As a general purpose script, I think this is a bug in `examples/run_squad.py`, and users are easy to be bothered with the poor performance of Longformer on `run_squad.py`, except for the professional users thar are quite familiar with the implementation details of `transformers/data/processors/squad.py`.

Is there any plan to compact with Longformer on `examples/run_squad.py`? Or can I find any instructions to finetune Longformer Models on my own SQuAD-like dataset?

Hey @xixiaoyao , feel free to open a PR to better integrate Longformer into `run_squad.py` :-) Otherwise I think @patil-suraj's notebook here: https://github.com/patil-suraj/Notebooks/blob/master/longformer_qa_training.ipynb  should be very helpful :-) 

The bug is in `squad_convert_examples_to_features` not in `run_squad` see #4615

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.