Non-deterministic training issue on GPU: TF-BERT
Author: MFreidank
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
TF-BERT 
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
SST-2
* [ ] my own task or dataset: (give details below)

## To reproduce

In spite of combining learnings from:
* [the "complete recipe" in NVIDIA's slides from gputechconf](https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9911-determinism-in-deep-learning.pdf) 
* [a recently suggested workaround](https://github.com/tensorflow/tensorflow/issues/38185#issuecomment-643014439) for non-determinism issues with crossentropy loss

... I am still arriving at the following [short, non-deterministic colab notebook example](https://colab.research.google.com/drive/1VSU8lYFD0E1HKZrIL1MvyIRwAktlSF_t?usp=sharing) to train BERT .

My results for the sum of model weights (as computed with [this suggested function](https://github.com/NVIDIA/tensorflow-determinism/issues/2#issuecomment-548210203)) after training **for only 5 steps** is (differences are **`highlighted`** below):


| | Device | Before training | After training |
| ------------- | ------------- | ------------- | ------------- |
| Run 1  | GPU | -641227.5609667897224  | -641237.442 **`5159916282`** |
| Run 2  | GPU | -641227.5609667897224  | -641237.442 **`3093758523`** |
| | |  | |
| Run 1 | CPU | -641227.5609667301178 | -641238.1506845243275 |
| Run 2 | CPU | -641227.5609667301178 | -641238.1506845243275 |

This variance gets increasingly more pronounced when the model is trained for longer periods of time. 
I am expecting a general problem with the computational graph with BERT introducing non-determinism. As a result, this could affect a large part of the huggingface community. 

Please keep in mind that determinism is of key importance in certain industries and also a pre-requisite for reproducible research.

Could you please help identify the source of non-determinism and provide guidance on how we can resolve it?

Steps to reproduce the behavior:

1. Execute colab notebook above on the GPU runtime using Tensorflow 2.2.0, observe non-deterministic behavior
2. Execute colab notebook above on the CPU runtime using Tensorflow 2.2.0, observe deterministic behavior

## Expected behavior
Training should be deterministic both on GPU and CPU runtime for TF 2.2.0.

## Environment info
* tensorflow==2.2.0
* nlp==0.2.1
     
- `transformers` version: 2.11.0
- Platform: Linux, Ubuntu 18.04.3 LTS bionic
- Python version: 3.6.9
- PyTorch version (GPU?): -
- Tensorflow version (GPU?): 2.2.0-gpu
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Comments:
Hi! [This StackOverflow issue](https://datascience.stackexchange.com/questions/14812/making-keras-tensorflow-code-execution-deterministic-on-a-gpu) might be of interest to you.

Namely:

> In fact, the randomness(non-determinstic) is a behavior of GPU.
> 
> The reason behind is that cuDNN(and othere CUDA stuffs) uses a non-deterministic algorithm to compute gradients, thus we can't determine anything.

Note that this issue is being addressed as an [issue in the tensorflow-determinism repo](https://github.com/NVIDIA/tensorflow-determinism/issues/19). I have also added a reference to that repo in the above-mentioned Stack Exchange / Data Science question.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.