Training Reproducibility Problem

baiyu0120: <p><strong>Question about torch training reproducibility.</strong><br>
I set the random seed and cudnn as below:</p>
<blockquote>
<p>random.seed(1)<br>
torch.manual_seed(1)<br>
torch.cuda.manual_seed(1)<br>
np.random.seed(1)<br>
torch.backends.cudnn.deterministic = True</p>
</blockquote>
<p>It works well in torch v0.4.0, but when I upgraded it to v1.0.0, the training loss became irreproducible.<br>
My environment as below:</p>
<ol>
<li>torch v0.4.0, Cuda 8.0, Cudnn  6.0.21, Ubuntu 16.04, GPU GTX1070.</li>
<li>torch v1.0.0, Cuda 10.0, Cudnn 7.3.1.20, Ubuntu 18.04, GPU RTX2080TI</li>
</ol>
<p>Can anyone help me?</p>

rasbt: <aside class="quote no-group quote-modified" data-username="baiyu0120" data-post="1" data-topic="37143">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/54ee81/48.png" class="avatar"> baiyu0120:</div>
<blockquote>
<p>It works well in torch v0.4.0, but when I upgraded it to v1.0.0, the training loss became reproducible.</p>
</blockquote>
</aside>
<p>Assuming you mean irreproducible, do you mean that you get different results if you run the same code on PyTorch 0.4 vs PyTorch 1.0 or if you run the code &gt;= 2 times on PyTorch 1.0?</p>

baiyu0120: <p>Sorry, I mean irreproducible.</p>
<p>Yes, both irreproducible, PyTorch 0.4 vs PyTorch 1.0 and &gt;= 2 times on PyTorch 1.0.</p>

rasbt: <blockquote>
<p>PyTorch 0.4 vs PyTorch 1.0</p>
</blockquote>
<p>this should be expected.</p>
<blockquote>
<p>&gt;= 2 times on PyTorch 1.0.</p>
</blockquote>
<p>This is weird, since you set both random seeds and deterministic cuda behavior. I am currently using the same cards (RTX 2080TIs) and couldn’t find a difference when running code multiple times. Sorry, I have no way to explain this. But one possibility could be that you set the random seeds somewhere later in your script after you initialized the weights?</p>

baiyu0120: <p>My code got an error while training, but it still running.</p>
<p><em>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument</em></p>
<p>Is this related to the irreproducilibity?</p>

rasbt: <p>I haven’t seen this before but it might be</p>

baiyu0120: <p>hi, can you tell me your environments?<br>
Python? pyTorch?Cuda？Cudnn?Ubuntu?</p>

rasbt: <p>I have three different machines with all different cards (1080Ti’s, RTX2018’s, Titan V) and never encountered a problem when using conda. Before, I had a custom compiled versions of everything but I noticed that conda was just a) faster to upgrade b) faster to run (probably because of better compiled MKL stuff)</p>
<ul>
<li>Ubuntu 18.04</li>
<li>Python 3.7</li>
<li>PyTorch 1.0.1 (but PyTorch 0.4 and 1.0 not too long ago worked similarly well)</li>
<li>Cuda 10 (and whatever cuDNN version comes with the PyTorch conda installer EDIT: cuDNN 7.4.0.1)</li>
</ul>

tom: <p>Note that there are known sources of randomness even for this case. The documentation has <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">a section on it</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

baiyu0120: <p>I got reproducibility when replaced the nn.upsample by nn.PixelShuffle in my model.<br>
I tried nearest and bilinear in nn.upsmple,  all of them are training irreproducible.</p>

baiyu0120: <p>And nn.ConvTranspose2d is reproducible.</p>

tom: <p>I think upsample should be part of the known irreproducible list.</p>

baiyu0120: <p>But it was reproducible in v 0.4.0.<br>
What do you mean “known irreproducible list” ?</p>

baiyu0120: <p>you mean here ?<br>
from <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a><br>
<em>A number of operations have backwards that use  <code>atomicAdd</code> , in particular <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.embedding_bag" rel="nofollow noopener"> <code>torch.nn.functional.embedding_bag()</code> </a>,<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.ctc_loss" rel="nofollow noopener"> <code>torch.nn.functional.ctc_loss()</code> </a> and many forms of pooling, padding, and <strong>sampling.</strong> There currently is no simple way of avoiding non-determinism in these functions.</em></p>

tom: <p>Ha, yeah, I thought I should have added it already.<br>
There also is an open issue for enhancing reproducibility. Everyone is in favour of it,but so far none who wants to do it.</p>
<p>Best regards</p>
<p>Thomas</p>

baiyu0120: <p>Which version of Cuda has this problem? From 10.0?<br>
pyTorch v 0.4.0 and Cuda 8.0 seems working well.</p>

tom: <p>As far as I know, it’s not dependent on the CUDA/CuDNN version, but on how the backwards has been implemented. I don’t really know how that changed without digging through the sources.</p>
<p>Best regards</p>
<p>Thomas</p>

rasbt: <blockquote>
<p>Note that there are known sources of randomness even for this case. The documentation has <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">a section on it</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
</blockquote>
<p>On that page, it says:</p>
<blockquote>
<p>Completely reproducible results are not guaranteed across PyTorch releases, individual commits or different platforms. Furthermore, results need not be reproducible between CPU and GPU executions, even when using identical seeds.</p>
</blockquote>
<p>This should be expected though, because cuDNN and PyTorch’s CPU variants use different algorithms for certain operations (e.g., a prominent example would be convolutional operations for which dozens of approximations exist). Also, difference between versions should be expected – could be also attributed to bug fixes etc.</p>
<p>In practice, I find that if I run the same code multiple times with the same cuDNN and PyTorch version (even on different machines; assuming manual seeds for weight init and shuffling are set and cuDNN is set to deterministic), I always get consistent results.</p>

baiyu0120: <p>hi, I got another irreproducibility problem.<br>
I downgraded the RTX2080TI platform from torch v1.0.0 to torch v0.4.0, and now it is reproducible when running the code &gt;= 2 times.<br>
But I found that the batchnorm and nn.linear cause irreproducibility between my two platform below.<br>
If I remove the batchnorm and replace the nn.linear by nn.conv2d, it is reproducible.</p>
<ol>
<li>torch v0.4.0, Cuda 8.0, Cudnn 6.0.21, Ubuntu 16.04, GPU GTX1070.</li>
<li>torch v0.4.0, Cuda 9.0, Cudnn 7.4.2.24, Ubuntu 18.04, GPU RTX2080TI</li>
</ol>
<p>Actually, both CPU-version and GPU-version are irreproducible.</p>

tom: <p>I only know about PyTorch &gt;=1, but if you have a tensor and get different results when sending it through linear several times, I would be most interested.<br>
Linear and and batch norm are generally believed to be reproducible. (But be certain that you’re not applying updated batch norm statistics.)</p>
<p>Best regards</p>
<p>Thomas</p>