Unable to obtain deterministic training

username: michaelshiyu
cooked: <p>Hi all,</p>
<p>I was using the <a href="https://github.com/NVIDIA/pix2pixHD" rel="nofollow noopener">NVIDIA/pix2pixHD</a> and trying to make the training deterministic. Here’s what I set in the beginning of the main training entry script:</p>
<pre><code class="lang-auto">random.seed(123)
torch.manual_seed(123)
np.random.seed(123)
torch.cuda.manual_seed_all(123)
torch.backends.cudnn.deterministic=True
torch.backends.cudnn.benchmark = False
os.environ['PYTHONHASHSEED'] = str(123)
</code></pre>
<p>But the training was not deterministic and the discrepancy was too large to be caused by floating point errors.</p>
<p>Here’s the output from one run:</p>
<pre><code class="lang-auto">(epoch: 1, iters: 1, time: 4.743) G_GAN: 4.459 G_GAN_Feat: 12.911 G_VGG: 10.349 D_real: 4.135 D_fake: 2.747 
(epoch: 1, iters: 2, time: 0.698) G_GAN: 21.762 G_GAN_Feat: 12.063 G_VGG: 8.263 D_real: 26.311 D_fake: 21.320 
(epoch: 1, iters: 3, time: 0.696) G_GAN: 7.435 G_GAN_Feat: 12.372 G_VGG: 8.679 D_real: 8.118 D_fake: 7.287 
(epoch: 1, iters: 4, time: 0.693) G_GAN: 7.552 G_GAN_Feat: 11.537 G_VGG: 7.424 D_real: 6.559 D_fake: 7.466 
(epoch: 1, iters: 5, time: 0.697) G_GAN: 7.928 G_GAN_Feat: 10.878 G_VGG: 8.100 D_real: 6.153 D_fake: 7.848 
(epoch: 1, iters: 6, time: 0.697) G_GAN: 4.726 G_GAN_Feat: 10.149 G_VGG: 8.180 D_real: 3.945 D_fake: 4.801
</code></pre>
<p>Here’s another run:</p>
<pre><code class="lang-auto">(epoch: 1, iters: 1, time: 4.797) G_GAN: 4.459 G_GAN_Feat: 12.911 G_VGG: 10.349 D_real: 4.135 D_fake: 2.747 
(epoch: 1, iters: 2, time: 0.703) G_GAN: 21.762 G_GAN_Feat: 12.063 G_VGG: 8.263 D_real: 26.311 D_fake: 21.320 
(epoch: 1, iters: 3, time: 0.702) G_GAN: 7.434 G_GAN_Feat: 12.372 G_VGG: 8.679 D_real: 8.118 D_fake: 7.287 
(epoch: 1, iters: 4, time: 0.701) G_GAN: 7.590 G_GAN_Feat: 11.538 G_VGG: 7.420 D_real: 6.560 D_fake: 7.503 
(epoch: 1, iters: 5, time: 0.702) G_GAN: 7.839 G_GAN_Feat: 10.882 G_VGG: 8.096 D_real: 6.145 D_fake: 7.759 
(epoch: 1, iters: 6, time: 0.702) G_GAN: 4.819 G_GAN_Feat: 10.133 G_VGG: 8.170 D_real: 3.960 D_fake: 4.899
</code></pre>
<p>Note that the above two runs were obtained with <code>num_workers=0</code> when creating the data loaders. With <code>num_workers&gt;0</code>, the randomness persists.</p>
<p>The original implementation had a bug with using CPU and I tried on my own fork (which has some modifications compared to the original one) and training on CPU is fully deterministic. But my own fork also cannot train deterministically when on GPU. Also, if it matters, I am using a single 2080Ti and CUDA 10.0 for these trainings.</p>
<p>Honestly, I am not sure if this is a pix2pixHD issue or a PyTorch one. But since I was able to obtain deterministic training on CPU, I am suspecting that the problem might be with PyTorch.</p>
<p>Any help would be greatly appreciated. Many thanks in advance!</p>

username: tom
cooked: <p>Have you looked at the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">notes on reproducibility</a>?</p>
<p>Best regards</p>
<p>Thomas</p>

username: michaelshiyu
cooked: <p>Hi Thomas,</p>
<p>I just read it and saw this:</p>
<blockquote>
<p>There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular  <code>atomicAdd</code> , where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result. PyTorch functions that use  <code>atomicAdd</code>  in the forward include <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.index_add_" rel="noopener nofollow ugc"> <code>torch.Tensor.index_add_()</code> </a>, <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.scatter_add_" rel="noopener nofollow ugc"> <code>torch.Tensor.scatter_add_()</code> </a>,<a href="https://pytorch.org/docs/stable/torch.html#torch.bincount" rel="noopener nofollow ugc"> <code>torch.bincount()</code> </a>.</p>
<p>A number of operations have backwards that use  <code>atomicAdd</code> , in particular <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.embedding_bag" rel="noopener nofollow ugc"> <code>torch.nn.functional.embedding_bag()</code> </a>,<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.functional.ctc_loss" rel="noopener nofollow ugc"> <code>torch.nn.functional.ctc_loss()</code> </a> and many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-determinism in these functions.</p>
</blockquote>
<p>Just to make sure, I wonder if this means that as long as my script used one of these operations, there would be no way to guarantee deterministic training? Not even with <code>torch.backends.cudnn.deterministic=True</code>?</p>
<p>Thanks.</p>

username: tom
cooked: <p>Mostly, yes.</p>
<p>Best regards</p>
<p>Thomas</p>

username: michaelshiyu
cooked: <p>I see.</p>
<blockquote>
<p>…and many forms of pooling, padding, and sampling.</p>
</blockquote>
<p>Is there a full list of these operations hosted somewhere?</p>
<p>Thanks.</p>

username: karaspd
cooked: <p>It would be great if there would be a full list of operations that might not get reproducible even by enabling cudnn.detereministic flag.<br>
I am using <code>torch.nn.CTCLoss</code> with cpu the results are reproducible but with gpu and following setting the results from two experiments slowly get diverging. I just want to make sure there is no way for me to get reproducible training (loss values, parameter values) as long as I am using ctc loss.</p>
<pre><code class="lang-auto">    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    import numpy as np
    np.random.seed(seed)
    import os
    os.environ["PYTHONHASHSEED"] = "0"

    import random
    random.seed(seed)

    cudnn.deterministic = True
    cudnn.benchmark = False
</code></pre>
<p>Thanks</p>

username: michaelshiyu
cooked: <p>I feel like right now the only way to figure this out is to actually look at the C++ code, which is a bummer for people that only want to deal with the Python interface of PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>