Pytorch reproducibility

username: Ferd
cooked: <p>I meet a problem about reproducibility of pytorch.<br>
I run a network multiple times, but obtain different result each time.<br>
Here is my setting:</p>
<pre><code class="lang-auto">torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
np.random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>
<p>I expect that the output of network should be the same every time I run the network, at least for the first few mini-batches.<br>
But this is not the case, here are the experiment results(I choose the same gpu and seed each time, batchsize=1):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de2890d147aa8e65118c3295e8489e69a71c2062.png" data-download-href="https://discuss.pytorch.org/uploads/default/de2890d147aa8e65118c3295e8489e69a71c2062" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de2890d147aa8e65118c3295e8489e69a71c2062_2_488x500.png" alt="image" data-base62-sha1="vHiUqbgGGhIeVjLCLGQomIQ2xvI" width="488" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de2890d147aa8e65118c3295e8489e69a71c2062_2_488x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/e/de2890d147aa8e65118c3295e8489e69a71c2062_2_732x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/e/de2890d147aa8e65118c3295e8489e69a71c2062.png 2x" data-dominant-color="F0F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">821×841 76 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For random initialization case, loss of batch 1 is the same each time. It indicates that the initial weights are the same. However, if    using Adam, the result becomes different since batch 2.</p>
<p>For resuming from checkpoint case(optimizer is also restored), it produces expected result.</p>
<p>I am confused, I don’t find any random variable in Adam, and I know how to explain the result. Is there anything I didn’t notice when initializing Adam?</p>
<p>Any help is appreciated.</p>

username: Jim_Thompson
cooked: <p>I noticed you did not mention this option setting: <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms" class="inline-onebox" rel="noopener nofollow ugc">torch.use_deterministic_algorithms — PyTorch 1.10.0 documentation</a></p>

username: Ferd
cooked: <p>Thank you for your reply. However, it is introduced into torch since 1.8.0 and I use 1.2.0.</p>
<p>And I use <strong>torch.nn.functional.interpolate()</strong> in the network, so it would definitely throw a RuntimeError if I set <strong>torch.use_deterministic_algorithms</strong> to true.</p>
<p>But I think it is not the reason, because it reproduces the same result when I use SGD or initialize network from a certain checkpoint.</p>

username: tom
cooked: <p>So the errors form the nondeterminism in the PyTorch interpolate function are very small (they’re mostly from numerical precision of different orders of adding floating point numbers). Adam has the somewhat unfortunate property that the first few steps have funny scaling which can amplify random fluctuations. So you might add an Adam warmup phase with a very low learning rate.</p>
<p>Note that in general I would consider your results stable enough to embrace the fact that there is stochastic fluctuation. The rather weak reproducibility that PyTorch can give you is no replacement for stable procedures as other people with different hardware, different OS, different tooling would want to be able to reproduce your results, too.</p>
<p>Best regards</p>
<p>Thomas</p>