device-side assert triggered Error while doing inference on Distilbert and Bert
Author: bhadreshpsavani
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Colab
- Python version: 3.8 
- PyTorch version (GPU?): 
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using Distilbert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
[colab](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/DistilbertPerformance.ipynb)

The tasks I am working on is:

* [x] an official GLUE/SQUaD task: SQUaD v2
* [ ] my own task or dataset: 

## To reproduce

Steps to reproduce the behavior:

1. Get Model and Tokenizer
2. Get SQUAD2 datasets
3. Performance Inference on the validation dataset with GPU
4. get result on SQUAD V2 Metrix 

Run the [colab](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/DistilbertPerformance.ipynb) to reproduce.

## Expected behavior

It should give results in the below format without error on SQUAD V2 Metrix 
```python
{'exact': 79.4660153288975, 'f1': 82.91266052065696, 'total': 11873, 'HasAns_exact': 77.64844804318489, 'HasAns_f1': 84.55162253066118, 'HasAns_total': 5928, 'NoAns_exact': 81.27838519764508, 'NoAns_f1': 81.27838519764508, 'NoAns_total': 5945, 'best_exact': 79.4660153288975, 'best_exact_thresh': 1.0, 'best_f1': 82.91266052065693, 'best_f1_thresh': 1.0}
```

Note: This code is working fine for longformer model.  I found this issue in Distilbert and Bert Model while doing inference on GPU

Tagging SMEs: @LysandreJik


Comments:
If you have a CUDA device-side error, it is advised to run your code on CPU, because then you will receive a more informative error message.

Hi @NielsRogge,
In CPU its working fine, Error is coming while using GPU only


Hi @bhadreshpsavani, this can't work on CPU. You're sending a sequence that is too long to the model so it cannot handle it.

Please replace
```py
inputs = tokenizer(example['question'], example['context'], return_tensors="pt")
```
by
```py
inputs = tokenizer(example['question'], example['context'], return_tensors="pt", truncation=True)
```

This truncates the sequences that are too long.

Your colab should work then.

Thanks @LysandreJik,
It worked!

Glad we could help, closing!