Using LongformerForQuestionAnswering on large documents (40K+ characters)
Author: zirlman
Pull request: no

 # ❓ Questions & Help

<!-- The GitHub issue tracker is primarily intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiasts can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you can always open a question on GitHub. You should then link to the SO question that you posted.
     -->

## Details
<!-- Description of your issue -->
So I decided to try the new LongformerForQuestionAnswering model on a larger input. From the [paper](https://arxiv.org/pdf/2004.05150.pdf) & the [source code](https://huggingface.co/transformers/_modules/transformers/modeling_longformer.html#LongformerForQuestionAnswering) I've understood that large document processing can be achieved by batching the input sequence in a certain format. For QA task the format goes like this: **question<\/sep><\/sep>context_block** where **<\/sep>** represents the separator token. This pattern allows the model to place global attention on the question. Feel free to correct me if I'm wrong. 

To test my assumption I've made a [notebook](https://colab.research.google.com/drive/1bEglkGcTXM_ZvdqbUT2_csp_TsRFPJwx?usp=sharing), but after running it I receive a `CUDA run out of memory error`. To get more insight, I've decided to process each batch record separately (`answer = get_answer(text, question, False)` ) but none of the records gave a reasonable answer.

So my questions are as follows:
1. Is my assumption for processing longer documents correct?
2. If so, what are the possible solutions for the memory error? I was thinking about a sliding window approach on the batch records 🤔
3. What could be the reason for such poor results when processing a single batch record?
4. This is more a PyTorch question, but shouldn't this code snippet empty the allocated GPU storage? Am I missing something here? As far as I know, CUDA storage is cleared when detaching the variable back to the CPU. Snippet: 
```python
if torch.cuda.is_available():
      input_ids = input_ids.to("cpu")
      attention_mask = attention_mask.to("cpu")
```
 
Thank you 🤗



Comments:
_What model of GPU are you using ?_

I didn't open your notebook, that's what I was looking for :
```
CUDA out of memory. Tried to allocate 4.92 GiB (GPU 0; 11.17 GiB total capacity; 4.64 GiB already allocated; 3.69 GiB free; 7.12 GiB reserved in total by PyTorch)
Detached cpu cpu
```

> What model of GPU are you using ?

@tuanardouin I was running the example on Colab only. So most of the time it was K80. But even on P100 it ran out of memory. I've expected that the model was performance expensive, but not this much 😱 

This notebook might help :-) https://github.com/patrickvonplaten/notebooks/blob/master/How_to_evaluate_Longformer_on_TriviaQA_using_NLP.ipynb