Non Reproducible result with GPU

jaiabhayk: <p>Hi,<br>
I am running the below script (which sets the manual seed as 1 for both cpu and gpu), but it does not give me reproducible results for gpu (for cpu it works fine), any known issue or am I missing something ?</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/mnist/main.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

# Training settings
parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                    help='input batch size for training (default: 64)')
parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                    help='input batch size for testing (default: 1000)')
parser.add_argument('--epochs', type=int, default=10, metavar='N',
                    help='number of epochs to train (default: 10)')
parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                    help='learning rate (default: 0.01)')
parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/mnist/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>torch.manual_seed(args.seed)<br>
if args.cuda:<br>
torch.cuda.manual_seed(args.seed)</p>

albanD: <p>If you’re using cudnn, if I remember correctly, some of their kernel are non-deterministic.<br>
Maybe disabling cudnn will help.</p>

sohil: <p>Currently, I am also facing the similar issue of reproducibility on pytorch.</p>
<p>As suggested, I tried disabling cudnn. But pytorch slows down the execution by 5-10x.</p>
<p>Edit:<br>
With cudnn disable, I am still not able to reproduce the results. I found out that the issue is because of torchvision transform file. When I disable the following lines in the code,<br>
transforms.RandomCrop(32, padding=4),<br>
transforms.RandomHorizontalFlip(),</p>
<p>then I am able to reproduce the same results. Setting seed using random.seed() doesn’t work either. Is this behavior expected or am I doing something wrong ?</p>

jaiabhayk: <p>Workaround ==&gt; If you set num_workers=0 in the dataloader, it should reproduce.</p>

gquenot: <p>I tried that workaround in the simple case of the cifar10 tutorial:<br>
<a href="http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a><br>
after having added only the Flip transform line:<br>
transforms.RandomHorizontalFlip(),<br>
I also inserted a seed initialization:<br>
torch.manual_seed(0)<br>
and I even disabled the shuffle.<br>
I am still not able to reproduce the results. Any other idea?</p>

gquenot: <p>It seems that ransforms.RandomCrop() and transforms.RandomHorizontalFlip() used the “plain” python seed, not the torch one. I fixed the reproducibilty issue with:<br>
import random<br>
random.seed(0)<br>
Reproducibility is conserved while enabling shuffle but not with num_workers&gt;0.</p>

dlutwy: <p>I got the same problem.<br>
My model was trained on 10-fold cross validation. I tried to save all the random states that may contribute to reproducibility every one fold was trained, the state including:</p>
<pre><code class="lang-auto">    state = {
        'fold_iter_idx': fold_iter_idx,
        'torch_rng_state': torch.get_rng_state(),
        'cuda_rng_state': torch.cuda.get_rng_state(),
        'np_rng_state': np.random.get_state(),
        'random_rng_state': random.getstate()
    }
</code></pre>
<p>Then a new model was initialized to train on next fold. Finally there are 10 results respect with to 10 folds.<br>
I can reproduce same results if I start the program from the very beginning (from the first fold).<br>
But when I tried to reproduce the 7th fold result only with loading the 6th state that saved during previous training, the result was different from that got during previous training.<br>
I tried to disable CUDNN and set</p>
<pre><code class="lang-auto">torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>I didn’t manage to reproduce it.<br>
But the results  with loading 6th states are same,<br>
and I can reproduce the same result as the the results got in <em>from the very beginning training</em>  by loading states on CPU.</p>
<p>I am not sure whether I make my issue clear. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>

albanD: <p>Hi,</p>
<p>You might want to check the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility</a> section of the doc.<br>
In particular, there are few operations that are inherently non-deterministic and so you won’t be able to get reproducible results if you use them.</p>

Barah_Fazili: <p>I’am facing a weird situation where on consecutive runs of a script across multiple gpus results are reproduced but running the same script after a few days is now giving different scores which are now reproducible on short term maybe.</p>