Non-Determinism Behavior that cannot reproduce result when evaluate on each epoch
Author: Jacob-Ma
Pull request: no
Labels: name: Help wanted

name: Need more information

I modified the example file `run_classifier.py` a little bit, so that the model could evaluate after each training epoch and save each evaluation results on file. This is good for someone who wants to see how the training epoch number influence the result. It is good to simply set the train_epoch = 50, save a checkpoint model on after each epoch so that you could choose the best model for testing data, based on the evalution results on dev dataset.

The strange phenomena and my question:
(1) even though the seed is correctly set for both Numpy and PyTorch, when evaluating on each step, the result is different from when I only have the final evaluation. 
(2) set the training epoch to a different number, say epoch 3 and epoch 10. The intermediate result of epoch 10 on step 3, is different from the epoch 3 results. 
(3) In the code I found that `num_train_optimization_steps = int(len(train_examples) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs`. 
When `args.gradient_accumulation_steps=1, args.num_train_epochs=1`, let's simply it into `num_train_optimization_steps = int(len(train_examples) / args.train_batch_size))`. 
I think it should be `num_train_optimization_steps = int((len(train_examples) - 1) / args.train_batch_size) + 1`.  If train_examples == 99, train_batch_size=32, it will gives different result.  Does it matters in the original code, it is passed into `BertAdam()`?

Could you guys have a try and see what's wrong with my code and what cause the problem? 
Thank you very much. 

Here is the modified example code and run_script are in the attachment.

### results:
#### [a] should be exactly the same
#### [b] should be exactly the same
=======

epoch 3
final evaluation:
{'eval_loss': 0.7023529836109706, 'eval_accuracy': 0.44, 'train_loss': 0.6765455901622772, 'global_step': 6}  ==> [a]

epoch 3
evaluation every epoch:
{'eval_loss': 0.6758423532758441, 'eval_accuracy': 0.6, 'train_loss': 0.8333011567592621, 'global_step': 2}
{'eval_loss': 0.6573205164500645, 'eval_accuracy': 0.58, 'train_loss': 0.7199544608592987, 'global_step': 4}
{'eval_loss': 0.662639707326889, 'eval_accuracy': 0.58, 'train_loss': 0.70155930519104, 'global_step': 6}  ==> [a]

epoch 10
final evaluation:
{'eval_loss': 0.8280548453330994, 'eval_accuracy': 0.58, 'train_loss': 0.8153044879436493, 'global_step': 20} ==> [b]

epoch 10
evaluation every epoch:
{'eval_loss': 0.6604576451437814, 'eval_accuracy': 0.58, 'train_loss': 0.8333011567592621, 'global_step': 2}
{'eval_loss': 0.6526826364653451, 'eval_accuracy': 0.58, 'train_loss': 0.8114463090896606, 'global_step': 4}
{'eval_loss': 0.6567909887858799, 'eval_accuracy': 0.58, 'train_loss': 0.6695355176925659, 'global_step': 6} ==> [a]
{'eval_loss': 0.6620746084621975, 'eval_accuracy': 0.62, 'train_loss': 0.6175626814365387, 'global_step': 8}
{'eval_loss': 0.6602040699550084, 'eval_accuracy': 0.52, 'train_loss': 0.5784901082515717, 'global_step': 10}
{'eval_loss': 0.667422890663147, 'eval_accuracy': 0.54, 'train_loss': 0.5177579522132874, 'global_step': 12}
{'eval_loss': 0.6945722614015851, 'eval_accuracy': 0.52, 'train_loss': 0.5649124383926392, 'global_step': 14}
{'eval_loss': 0.7062868390764508, 'eval_accuracy': 0.48, 'train_loss': 0.7067148089408875, 'global_step': 16}
{'eval_loss': 0.7712458883013044, 'eval_accuracy': 0.46, 'train_loss': 0.7441326081752777, 'global_step': 18}
{'eval_loss': 1.5845262237957545, 'eval_accuracy': 0.42, 'train_loss': 1.0378091633319855, 'global_step': 20}  ==> [b]

### run the bash script the second time, it is the same as the previous result.
It is deterministic somehow(the result for final evaluation is the same as the final evaluation for the previous run. So as to eval_on_each_epoch), but the difference because of train_epoch num is the same.

epoch 3
final evaluation
{'eval_loss': 0.7023529836109706, 'eval_accuracy': 0.44, 'train_loss': 0.6765455901622772, 'global_step': 6}

epoch 3
evaluation every epoch
{'eval_loss': 0.6758423532758441, 'eval_accuracy': 0.6, 'train_loss': 0.8333011567592621, 'global_step': 2}
{'eval_loss': 0.6573205164500645, 'eval_accuracy': 0.58, 'train_loss': 0.7199544608592987, 'global_step': 4}
{'eval_loss': 0.662639707326889, 'eval_accuracy': 0.58, 'train_loss': 0.70155930519104, 'global_step': 6}

epoch 10
final evaluation
{'eval_loss': 0.8280548453330994, 'eval_accuracy': 0.58, 'train_loss': 0.8153044879436493, 'global_step': 20}

epoch 10
evaluation each step
{'eval_loss': 0.6604576451437814, 'eval_accuracy': 0.58, 'train_loss': 0.8333011567592621, 'global_step': 2}
{'eval_loss': 0.6526826364653451, 'eval_accuracy': 0.58, 'train_loss': 0.8114463090896606, 'global_step': 4}
{'eval_loss': 0.6567909887858799, 'eval_accuracy': 0.58, 'train_loss': 0.6695355176925659, 'global_step': 6}
{'eval_loss': 0.6620746084621975, 'eval_accuracy': 0.62, 'train_loss': 0.6175626814365387, 'global_step': 8}
{'eval_loss': 0.6602040699550084, 'eval_accuracy': 0.52, 'train_loss': 0.5784901082515717, 'global_step': 10}
{'eval_loss': 0.667422890663147, 'eval_accuracy': 0.54, 'train_loss': 0.5177579522132874, 'global_step': 12}
{'eval_loss': 0.6945722614015851, 'eval_accuracy': 0.52, 'train_loss': 0.5649124383926392, 'global_step': 14}
{'eval_loss': 0.7062868390764508, 'eval_accuracy': 0.48, 'train_loss': 0.7067148089408875, 'global_step': 16}
{'eval_loss': 0.7712458883013044, 'eval_accuracy': 0.46, 'train_loss': 0.7441326081752777, 'global_step': 18}
{'eval_loss': 1.5845262237957545, 'eval_accuracy': 0.42, 'train_loss': 1.0378091633319855, 'global_step': 20}

Repeatability is crucial in machine learning, please help with this issue. Thank you very much. 

The code sample dataset is here for SST tasks:
[huggingface_bert_issue2.zip](https://github.com/huggingface/pytorch-pretrained-BERT/files/3077188/huggingface_bert_issue2.zip)



Comments:
There is some non-determinism in cuDNN. Try setting `torch.backends.cudnn.deterministic = True` in your code: with that plus the RNG seeding, you should be able to get deterministic results.

Yes go with @Rocketknight1 suggestion.
Also check that you set model in eval mode to disable the DropOut modules before evaluating.

Hi, thank you for you guys reply. 
I simplify the model to a simple MNIST problem to check if there are similar phenomena.
I also posted in pytorch/examples github, but no one replies. I found that it seems that it is PyTorch's problem. 

Could you please take a look at these posts?

[mnist](https://github.com/pytorch/examples/issues/542), in these post, Yes, you are right `torch.backends.cudnn.deterministic = True` could bring some consistency to the result. 

However, even though you set `torch.backends.cudnn.deterministic = True`, STILL, when you compare evaluate at each epoch and only make final evaluation gives different results, you could see in this post:
[minis final evaluation VS evaluation each epoch](https://github.com/pytorch/examples/issues/543)

What would you think?  Is it indeed pyTorch's problem?  I have code attached in the post for your convenience to run. Thank you very much.



> There is some non-determinism in cuDNN. Try setting `torch.backends.cudnn.deterministic = True` in your code: with that plus the RNG seeding, you should be able to get deterministic results.

Hi, thank you for your reply, when I post this issue, I have searched a lot.  In the code I have already set the random seeding like this:
```
use_cuda = not args.no_cuda and torch.cuda.is_available()

# set seed
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
if use_cuda:
    torch.cuda.manual_seed_all(args.seed)  # if got GPU also set this seed
```
and also put `torch.backends.cudnn.deterministic = True` at the front of the script. 

But the behavior is still the same.

What do you think of this mystic difference?
My this post shows the detail and have code attached to have a try:
[minis final evaluation VS evaluation each epoch](https://github.com/pytorch/examples/issues/543)


> Yes go with @Rocketknight1 suggestion.
> Also check that you set model in eval mode to disable the DropOut modules before evaluating.

Thank you for your reply. 
I have checked that, in my original code. 
In train() and test() fuction, I already set model.train() and model.eval() respectively. 

But the behavior is still the same. 
What do you think of this mystic difference?

My this post shows the detail and have code attached to have a try:
[minis final evaluation VS evaluation each epoch](https://github.com/pytorch/examples/issues/543)

This is really a PyTorch/CUDA issue rather than an issue with this repo, but there is more information here: https://pytorch.org/docs/stable/notes/randomness.html

You can also try setting `torch.backends.cudnn.benchmark = False`

@Rocketknight1 
Thank you for your reply. 
I also tried to add `torch.backends.cudnn.benchmark=False`, the behavior is still the same as above.
Yeah, it seems to be a PyTorch problem. 


The #question channel on the PyTorch slack is a good place to get quick answers on these stuff.
PyTorch's forum is also great.