Random seed initialization

Bixqu: <p>I have a problem regarding a large variation in the result I get, by running my model multiple times. The exact same architecture and training gives anywhere from 91.5% to 93.4% accuracy on image classification (cifar 10).</p>
<p>The problem is that I don’t know how to use the torch random seed in order to get the better results, not the worse ones. I tried various values for the random seed, with:</p>
<pre><code>torch.manual_seed(7)
</code></pre>
<p>and I get the lower bound of the results. Any ideas?</p>

smth: <p>if you are using GPU, you might also need to set <code>torch.cuda.manual_seed_all</code>.<br>
<a href="http://pytorch.org/docs/master/cuda.html#random-number-generator" class="onebox" target="_blank">http://pytorch.org/docs/master/cuda.html#random-number-generator</a></p>

vsag: <p><a class="mention" href="/u/smth">@smth</a> do we need to set torch.manual_seed() and torch.cuda.manual_seed_all() or the second is enough? thanks.</p>

smth: <p>with the latest pytorch 0.3 version you only need to set torch.manual_seed which will seed all devices</p>

sid: <p>Is the random number generator platform independent?</p>

smth: <p>the CPU RNG is platform-independent. I am not sure about the CUDA RNG and what guarantees NVIDIA gives across GPU models, CUDA versions and platforms.</p>

voxmenthe: <aside class="quote no-group" data-username="smth" data-post="2" data-topic="7854">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>torch.cuda.manual_seed_all.</p>
</blockquote>
</aside>
<p>Funny, even though I have included both:</p>
<p><code>torch.manual_seed(999)</code></p>
<p>and</p>
<p><code>if torch.cuda.is_available():</code><br>
<code>		torch.cuda.manual_seed_all(999)</code></p>
<p>I am still getting inconsistent results, fluctuating 1-2% by re-running the model. I wonder why that could be?</p>

ptrblck: <p>Could you try to add <code>torch.backends.cudnn.deterministic = True</code> to your code?<br>
CUDNN has some non-deterministic methods, so small fluctuations might come from this.</p>

voxmenthe: <p>I added:</p>
<p><code>torch.backends.cudnn.deterministic = True</code>       in addition to:</p>
<p><code>torch.manual_seed(999)</code>         and</p>
<p><code>if torch.cuda.is_available(): torch.cuda.manual_seed_all(999)</code></p>
<p>but accuracy for same model/same data still varies considerably across runs. I’ve even tried duplicating the above in the code and even tried switching to the latest version of pytorch (3.1) but still getting the same variability in accuracy across runs for same model/same data. Weird.</p>

amirkargar: <p>Hi,</p>
<p>I’m having the same issue, did you figure out a way to make the results consistent across runs?</p>
<p>Thanks,<br>
Amir</p>

Darren_Lee: <p>Hi,</p>
<p>Have you figured out how to make the results reproducible now?</p>
<p>Thanks,<br>
Darren</p>

ivan-bilan: <p>Same problem here, running on PyTorch 0.4. I am using RRelu though, even though I’ve set all the flags mentioned above, results differ by a margin of +/- 0.5% from run to run.</p>

Yufeng_Ma: <p>What’s the number of workers for your dataloader? The following post might be helpful for deterministic results.</p>
<aside class="quote quote-modified" data-post="1" data-topic="9087">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/e8c25b/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/deterministic-non-deterministic-results-with-pytorch/9087">Deterministic/non-deterministic results with PyTorch</a> 
  </div>
  <blockquote>
    Hi, 
I’ve had some trouble in reproducing results, and after reading a few posts there seem to be multiple causes for non-determinism, some which are expected. I’d like to check if I got it right. 
Issue 1 
Dataloaders with multiple threads: They seem to be problematic, even if randoms seeds are set beforehand. [<a href="https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/4">ref</a>]. 
Issue 2 
cuDNN: Apparently cuDNN seems to have non-deterministic kernels [<a href="https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/2">ref1</a>][<a href="https://discuss.pytorch.org/t/non-deterministic-on-gpu/7355/2">ref2</a>] 
Issue 3 
GPU: Apparently, some reductions are non-deterministic in GPU, even without cuDNN. …
  </blockquote>
</aside>


wtsangmed: <p>Was following this post b/c I ran into same issues training an autoencoder. I don’t know if the OP has solved the problem. but I did a test last night on a AWS GPU and cuda on w/ the parameters below gave me consistent results.<br>
<code>torch.backends.cudnn.deterministic = True</code><br>
<code>torch.manual_seed(999)</code></p>
<p>Further I explicitly specify <code>model.eval()</code> after training when computing the decoders and encoders.</p>
<p>Alternatively when I have, below, the results were inconsistent.<br>
<code>torch.backends.cudnn.deterministic = True</code><br>
<code>torch.cuda.manual_seed_all(999)</code></p>
<p>As an above poster mentioned it seems as though torch.manual_seed() applies to both cuda and cpu devices for the latest version. So if you’re not getting consistent result w/ torch.cuda.manual_seed_all, try just torch.manual_seed. This may depend on the pytorch version you have installed…Hope this helps.</p>

mutaku: <p>Good info.</p>
<p>The docs also suggest setting: <code>torch.backends.cudnn.benchmark = False</code></p>
<p>and remember that Numpy should be seeded as well.</p>
<p>–&gt; <a href="https://pytorch.org/docs/master/notes/randomness.html" rel="nofollow noopener">Randomness [Docs]</a></p>

Noam_Salomonski: <p>did anyone solve this yet?</p>

isalirezag: <p>Sounds like there is another question related <a href="https://discuss.pytorch.org/t/how-to-get-deterministic-behavior/18177/11">here</a>.</p>
<p>anyways, I think this can be a solution:</p>
<pre><code class="lang-auto">manualSeed = 1

np.random.seed(manualSeed)
random.seed(manualSeed)
torch.manual_seed(manualSeed)
# if you are suing GPU
torch.cuda.manual_seed(manualSeed)
torch.cuda.manual_seed_all(manualSeed)


torch.backends.cudnn.enabled = False 
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>
<p>also in the dataloader i set <strong>num_workers = 0</strong></p>
<p>based on <a href="https://github.com/facebookresearch/maskrcnn-benchmark/issues/376" rel="nofollow noopener">here</a><br>
you also need to change  <code>worker_init_fn</code> as :</p>
<pre><code class="lang-auto">def _init_fn():
    np.random.seed(manualSeed)
    

DataLoding = data.DataLoader(..., batch_size = ..., 
                             collate_fn = ..., 
                             num_workers =..., 
                             shuffle = ..., 
                             pin_memory = ...,
                             worker_init_fn=_init_fn)


</code></pre>
<p>I noticed if we dont do <code>torch.backends.cudnn.enabled = False</code>  the results are very close, but some times not match <img src="https://discuss.pytorch.org/images/emoji/apple/hushed.png?v=7" title=":hushed:" class="emoji" alt=":hushed:"><br>
p.s. im using pytorch 1.0.1</p>

ShangxuanWu: <p>Thanks!</p>
<p><strong>num_workers = 0</strong> and <strong>torch.backends.cudnn.enabled = False</strong> are the real thing that works! And I also see that if you train one step 10 times,  only  using <strong>num_workers = 0</strong> we can get exactly same output 8 times and different output 2 times.</p>

arnabsinha: <pre><code class="lang-auto">np.random.seed(0)
random.seed(0)
torch.manual_seed(0)
torch.cuda.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.enabled = False
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>
<p>and setting dataloader like the following:</p>
<pre><code class="lang-auto">torch.utils.data.DataLoader(training, shuffle = True, batch_size=BATCH_SIZE, worker_init_fn=np.random.seed(0),num_workers=0)
</code></pre>
<p><strong>WORKED FOR ME!</strong></p>
<p>I am using Pytorch version 1.0.0.</p>

Tethys_Sun: <p>I tried exactly same setting, even with torch.backends.cudnn.enabled =False, the results are not the same… Do you have any idea?</p>