Effect of torch.backends.cudnn.deterministic=True

username: rezzy
cooked: <p>As far as I understand, if you use <code>torch.backends.cudnn.deterministic=True</code> and with it <code>torch.backends.cudnn.benchmark = False </code> in your code (along with settings seed), it should cause your code to run deterministically.</p>
<p>However, for reasons I don’t understand, if I remove the two lines it will always result in worse results. Even setting deterministric for CUDNN and other places, I still don’t get fully identical results, but removing it causes my loss to not go any lower (top lines in attached image). What am I doing wrong?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/358e61b31f3a0a100eb45516309c609303064b22.png" data-download-href="https://discuss.pytorch.org/uploads/default/358e61b31f3a0a100eb45516309c609303064b22" title="loss_epoch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/358e61b31f3a0a100eb45516309c609303064b22_2_690x256.png" alt="loss_epoch" data-base62-sha1="7DMm92eGVpcgWfVEp8AP8iG4pZU" width="690" height="256" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/358e61b31f3a0a100eb45516309c609303064b22_2_690x256.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/358e61b31f3a0a100eb45516309c609303064b22_2_1035x384.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/358e61b31f3a0a100eb45516309c609303064b22_2_1380x512.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_epoch</span><span class="informations">1409×524 56.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Environment:</p>
<ul>
<li>Pytorch Lighting 1.7</li>
<li>Using DDP accelerator</li>
<li>Settings seeds for random, np, torch (<code>manual_seed_all</code>)</li>
<li>Have the below code for <code>worker_init_fn</code> of DataLoader:</li>
</ul>
<pre><code class="lang-auto">def loader_init_fn(seed):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)   
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You can check our doc on reproducibility for the full details: <a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox">Reproducibility — PyTorch 1.7.1 documentation</a></p>

username: rezzy
cooked: <p>Hi,<br>
Thanks for response. I understand the reproducibility requirements, but my question is, will disabling CuDNN benchmarking have an impact on the result - i.e., the operations themselves, not how fast they are performed?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The different cudnn setups should yield approx. the same final result. If that’s not the case, you might hit a faulty kernel.<br>
To isolate it, we would need to get the model definition, input shapes, used cudnn version, as well as the used GPU. With these information we can create the cudnn logs and check internally with reference implementations for correctness.</p>