Different output from model on CPU and GPU
Author: jxmorris12
Pull request: no

# 🐛 Bug

I trained models using the `run_glue.py` script and uploaded them to the model hub. I've realized that their output slightly differs between when I do inference on the CPU and the GPU.

Absolute errors are small – on the order of `1e-7` – but that turns out to be too much for my use case.

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MNLI
* [ ] my own task or dataset: (give details below)

## To reproduce
```python
(torch) qcuda8 04:47 PM > python
Python 3.7.7 (default, Mar 26 2020, 15:48:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> model_cpu = transformers.AutoModelForSequenceClassification.from_pretrained('textattack/bert-base-uncased-MNLI')
>>> model_gpu = transformers.AutoModelForSequenceClassification.from_pretrained('textattack/bert-base-uncased-MNLI').to('cuda')
>>>
>>> premise = "Among these are the red brick Royal Palace, which now houses the Patan Museum (Nepal's finest and most modern museum), and, facing the palace across the narrow brick plaza, eight temples of different styles and sizes."
>>> hypothesis = "The Patan Museum is down the street from the red brick Royal Palace."
>>>
>>> tokenizer = transformers.AutoTokenizer.from_pretrained('textattack/bert-base-uncased-MNLI')
>>> encoded_text = tokenizer.encode_plus((premise, hypothesis), return_tensors='pt')
>>> encoded_text['input_ids']
tensor([[101, 100, 100, 102]])
>>> encoded_text_cuda = {k: v.cuda() for k,v in encoded_text.items()}
>>> model_gpu(**encoded_text_cuda)[0].squeeze().tolist()
[-1.0867613554000854, 0.6688923239707947, 0.30274006724357605]
>>> model_cpu(**encoded_text)[0].squeeze().tolist()
[-1.0867608785629272, 0.6688917279243469, 0.3027404248714447]
```
## Expected behavior

I want the model output between the CPU model and GPU model to be the same.

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
Hi! I don't think it's possible to have two different hardware perform *exactly* the same. A 1e-7 precision is already very high. Out of curiosity, why do you need such a high precision?

Nevertheless, this is more of a pytorch-specific question rather than transformers-related. I'll close the issue here but feel free to link to an issue on the PyTorch forums if you do open one there!

@LysandreJik I haven't seen this issue with models not trained with transformers (though I probably just haven't looked hard enough).

Can you give me more info on why this is the case, and maybe point me to some relevant resources?

As to your second question-- we need a high precision because we're searching for adversarial examples that maximize model misprediction in our library [TextAttack](https://github.com/QData/TextAttack). This precision error caused a different search outcome with a CPU and GPU. So, for some pair of sequences $a$ and $b$, the model on the CPU predicted $a$ 'more correctly' than it predicted $b$, and the model on the GPU predicted $b$ more correctly than $a$.  Our automated tests caught this issue. 

Do you have a suggestion on how to fix it? Should we just truncate model scores to 7 decimal places? That feels like a crude fix.