How to get deterministic behavior?

dlmacedo: <p>I am using:</p>
<p>cudnn.benchmark = False<br>
cudnn.deterministic = True</p>
<p>random.seed(1)<br>
numpy.random.seed(1)<br>
torch.manual_seed(1)<br>
torch.cuda.manual_seed(1)</p>
<p>And still not getting deterministic behavior…</p>

ptrblck: <p>How large are the differences?<br>
Could you provide a code snippet showing the error?</p>
<p>If the absolute error is approx. <code>1e-6</code> it might be due to the usage of float values.</p>

dlmacedo: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/dlmacedo/deep-learning-class/blob/master/not_deterministic.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/dlmacedo/deep-learning-class/blob/master/not_deterministic.py" target="_blank" rel="nofollow noopener">dlmacedo/deep-learning-class/blob/master/not_deterministic.py</a></h4>
<pre><code class="lang-py">import argparse
import sys
import os
import time
import random
import numpy
import pandas as pd
import torch
import torch.backends.cudnn as cudnn
import torch.nn as nn
import torch.nn.parallel
import torch.optim
import torch.utils.data
import torchvision.transforms as transforms
import torchvision.models as torchvision_models
import torchnet as tnt

from tqdm import tqdm
from torch.autograd import Variable
from tensorboardX import SummaryWriter
</code></pre>

  This file has been truncated. <a href="https://github.com/dlmacedo/deep-learning-class/blob/master/not_deterministic.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


dlmacedo: <p>The diferences are meanifull… Not of the order of 1e-6…</p>
<p>The diferences are about 1% more or less…</p>

dlmacedo: <p>The problem is in the data augmentation transformations:</p>
<pre><code class="lang-auto">transforms.RandomCrop(32, padding=4),
transforms.RandomHorizontalFlip(),
</code></pre>
<p>After I removed the above code, it worked 100% deterministically…</p>
<p>I am using Pytorch 0.3.1</p>
<p>Does torchvision have a different seed from torch?</p>

dlmacedo: <p>The following appears to be the same issue:</p>
<aside class="quote" data-post="5" data-topic="1831">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/35a633/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/5">Non Reproducible result with GPU</a> 
  </div>
  <blockquote>
    I tried that workaround in the simple case of the cifar10 tutorial: 
<a href="http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">http://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a> 
after having added only the Flip transform line: 
transforms.RandomHorizontalFlip(), 
I also inserted a seed initialization: 
torch.manual_seed(0) 
and I even disabled the shuffle. 
I am still not able to reproduce the results. Any other idea?
  </blockquote>
</aside>

<p>If I use num_workers=0, I can get back the augmentation without losing the deterministic behavior, exactly as reported in the link above.</p>

SimonW: <p>This has been fixed in 0.4, provided you set <code>random.seed</code> in <code>worker_init_fn</code>.</p>
<p>Furthermore, you might want to set <code>torch.backends.cudnn.deterministic=True</code></p>

dlmacedo: <p>It worked. Thanks. Nevertheless, I think we should have a more straightforward way to get determinist behave. This “workaround” in the workers is not so intuitive for beginners.</p>

dlmacedo: <p>Based on my tests, even in PyTorch 0.4 we still need to initialize the workers with the same seed to get deterministic behavior. The following lines are NOT enough:</p>
<blockquote>
<p>cudnn.benchmark = False<br>
cudnn.deterministic = True</p>
<p>random.seed(1)<br>
numpy.random.seed(1)<br>
torch.manual_seed(1)<br>
torch.cuda.manual_seed(1)</p>
</blockquote>
<p>I think this should not be the standard behavior. In my opinion, the above lines should be enough to provide deterministic behavior. It is not obvious to the novice that, besides the above lines, he also need to initialize the workers with the same seed to get deterministic behavior.</p>

SimonW: <p>The problem is <code>numpy</code>. We can’t assume that <code>numpy</code> exists so you’d have to set the seed for <code>numpy</code> in workers yourself.</p>

ndronen: <p>IMO, <code>worker_init_fn</code> allows some flexibility, but why shouldn’t PyTorch’s workers have a reasonable default behavior? Something like the following code block, if it were to execute before <code>worker_init_fn</code>, would be backward compatible and would provide determinism out of the box, whether Numpy is installed or not.</p>
<pre><code class="lang-python">try:
    import numpy
    torch_seed = torch.initial_seed()
    # Numpy expects unsigned integer seeds.
    np_seed = torch_seed // 2**32-1
    numpy.random.seed(np_seed)
except Exception:
    pass
</code></pre>

surojit_sengupta: <p>So does <code>torch.backends.cudnn.deterministic=True</code><br>
make any form of random data pre-processing using torch libraries deterministic?</p>

iainmelvin: <p>Beware running 2 processes on the same machine, even if you set random seeds and set num_workers=0. In my experience running one process on it’s own is deterministic, but running 2 processes side by side is not. The consensus here is that static variables in dynamically linked libraries are to blame. If you need to compare, run on two different machines.<br>
(by 2 processes I mean two training scripts for example)</p>

taromakino: <p>def set_seed(seed):<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)<br>
np.random.seed(seed)<br>
random.seed(seed)</p>
<p>The final line solved this issue for me.</p>

dstanner: <p>I’ve read this thread closely and tried everything I can see on here as suggestions, and I still cannot get deterministic behavior during training. I’m using Pytorch 1.1.0 and Torchvision 0.3.0.</p>
<p>In my code I have the following:</p>
<pre><code>torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(1)
torch.cuda.manual_seed_all(1)
np.random.seed(1)
random.seed(1)
</code></pre>
<p>And my dataloader looks like this:</p>
<pre><code>train_loader = torch.utils.data.DataLoader(
    train_dataset,
    batch_size=128,
    num_workers=0,
    shuffle=True,
    pin_memory=True,
    worker_init_fn=random.seed(1)
)
</code></pre>
<p>Is there a place I’ve missed setting the seed? Looking at the first few mini-batches of training, the differences in accuracy across starts is pretty stark.</p>
<p>Also, I don’t even get deterministic behavior when I disable RandomResizedCrop and RandomHorizontalFlip in my image transforms – so the non-deterministic behavior is happening somewhere else.</p>
<p>Any clues or pointers would be appreciated.</p>

xiaoyanzhuo: <p>Based on previous answers, and official docs, what I did works for me.</p>
<pre><code class="lang-python">def set_seed(seed):
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
</code></pre>
<p><code>set_seed(0)</code>  # 0 or any number you want for the seed</p>
<p>in the  DataLoader(), I also set the <code>num_workers = 0</code></p>
<p>This setting makes my results reproducible.<br>
I am using latest Pytorch version is ‘1.5.0’.</p>

arnabsinha: <p>Will this work for Pytorch version 1.0.1 as well?</p>

xiaoyanzhuo: <p>Not sure about 1.0.1. I suppose if the same options are available, it may work as well. You can try.</p>

arnabsinha: <p>Well I added <code>torch.backend.cudnn.enabled = False</code> and it worked for me. I may have to recheck the results with it  to <code>torch.backend.cudnn.enabled = True</code>.</p>

Tethys_Sun: <p>Hi <a class="mention" href="/u/dstanner">@dstanner</a>, did you solve it? This doesn’t works for me too.</p>