Tensorflow and Torch yield significantly different results for same model
Author: DavidHuebner
Pull request: no

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0 (True)
- Tensorflow version (GPU?): 2.7.4 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help?

I am trying to convert a PyTorch transformer model to TensorFlow. When comparing the model outputs, I observe significant differences in all output values.

To reproduce, I compared the model output for the same model (e.g. `distilbert-base-uncased`) once loaded with Tensorflow (2.7.4) and once with torch (1.13.0), see script below. Results are very different.

- Is this an expected outcome? 
- Are there any strategies to mitigate these differences?

@sgugger @Rocketknight1 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Execute the following Python code.
```py
import numpy as np
import tensorflow as tf
from transformers import TFAutoModel, AutoTokenizer, AutoModel
import torch

model_path = "distilbert-base-uncased"
tf_model = TFAutoModel.from_pretrained(model_path) # The same problems occur with from_pt=True
pt_model = AutoModel.from_pretrained(model_path)

tokenizer = AutoTokenizer.from_pretrained(model_path)
payload = ["This is a great sentence embedding"]
encoded_input_tf = tokenizer(payload, return_tensors='tf')
encoded_input_pt= tokenizer(payload, return_tensors='pt')                 

tf_output = tf_model(**encoded_input_tf)
with torch.no_grad():
    pt_output = pt_model(**encoded_input_pt)

np.testing.assert_allclose(pt_output.last_hidden_state, tf_output.last_hidden_state)         
```  
yields
```
AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0

Mismatched elements: 7680 / 7680 (100%)
Max absolute difference: 0.00231361
Max relative difference: 4.5563045
 x: array([[[-0.259016, -0.081947, -0.052371, ..., -0.021481,  0.184779,
          0.368155],
        [-0.412638, -0.111859, -0.17412 , ..., -0.204094,  0.356639,...
 y: array([[[-0.25897 , -0.081805, -0.052341, ..., -0.021413,  0.185112,
          0.367904],
        [-0.412646, -0.111567, -0.174027, ..., -0.204349,  0.356727,...
```

### Expected behavior

Both models outputs should be similar. The `np.testing.assert_allclose()` should not raise an `AssertionError`.

Comments:
Hi @DavidHuebner, this is a known issue, and it's unfortunately unavoidable! Floating point calculations are inherently imprecise, which means that changing the specific kernels you use and the order of operations will slightly alter the result. Because TF uses different kernels to Torch, and because TF compiles models by default (which means that some operations may be fused or reordered), there will always be a numerical difference in their outputs. In general, we find that the final model outputs (e.g. token logits for `distilbert`) remain relatively similar, even though hidden states can vary by ~1e-3 or even ~1e-2 in some cases.

If you want to reduce the error, one useful tip is that TensorFlow enables TensorFloat-32 computation by default (which increases speed but reduces precision on newer GPUs), but Torch does not. Adding the line `tf.config.experimental.enable_tensor_float_32_execution(False)` to your TF code may make the results more similar to the Torch outputs, but even with that I suspect the error will be in the range 1e-4 to 1e-5, and reducing the error to ~1e-7 for any real model is probably impossible!

Alright. Thanks for the response and explanation.