Reproduce Training Question

cy-xu: <p>Dear Community,</p>
<p>I’m trying to reproduce a simple over-fitting experiment on autoencoders, I got matching initialization under control but the loss and results would diverge over time.</p>
<p>Focus on the first 10 steps, we could see they are initualizaed the same.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/aa863fd37a33dd259ee54a70a0a68dcba99e849c.png" data-download-href="https://discuss.pytorch.org/uploads/default/aa863fd37a33dd259ee54a70a0a68dcba99e849c" title="init_same_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aa863fd37a33dd259ee54a70a0a68dcba99e849c_2_690x443.png" alt="init_same_1" data-base62-sha1="okwNsOFLxZJhVrZAMKFQjeZHC2g" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/aa863fd37a33dd259ee54a70a0a68dcba99e849c_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/a/aa863fd37a33dd259ee54a70a0a68dcba99e849c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/aa863fd37a33dd259ee54a70a0a68dcba99e849c.png 2x" data-dominant-color="FCFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">init_same_1</span><span class="informations">700×450 34.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/27784254f3b94a971148e8d7717f35bd5b804f8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/27784254f3b94a971148e8d7717f35bd5b804f8a" title="init_same_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27784254f3b94a971148e8d7717f35bd5b804f8a_2_690x443.png" alt="init_same_2" data-base62-sha1="5Dah4fDGHyjWfChErhDS4Cv71vc" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27784254f3b94a971148e8d7717f35bd5b804f8a_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/2/27784254f3b94a971148e8d7717f35bd5b804f8a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/27784254f3b94a971148e8d7717f35bd5b804f8a.png 2x" data-dominant-color="FCFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">init_same_2</span><span class="informations">700×450 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then as training goes further, the curves will diverge a lot more<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0a1b0d47fe5c2729276f4591767517a993c75905.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a1b0d47fe5c2729276f4591767517a993c75905" title="newplot%20(4)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a1b0d47fe5c2729276f4591767517a993c75905_2_690x443.png" alt="newplot%20(4)" data-base62-sha1="1roJrZMFKQn2v7yWphDFWcQv6Wp" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0a1b0d47fe5c2729276f4591767517a993c75905_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/0/0a1b0d47fe5c2729276f4591767517a993c75905.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0a1b0d47fe5c2729276f4591767517a993c75905.png 2x" data-dominant-color="FAF8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot%20(4)</span><span class="informations">700×450 51.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/df1d746f27d073c8b69fadeb4d4617d600fbd5cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/df1d746f27d073c8b69fadeb4d4617d600fbd5cc" title="newplot%20(5)"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/df1d746f27d073c8b69fadeb4d4617d600fbd5cc_2_690x443.png" alt="newplot%20(5)" data-base62-sha1="vPLA7pwNiolInZdQUEGKeKvgzp2" width="690" height="443" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/df1d746f27d073c8b69fadeb4d4617d600fbd5cc_2_690x443.png, https://discuss.pytorch.org/uploads/default/original/2X/d/df1d746f27d073c8b69fadeb4d4617d600fbd5cc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/df1d746f27d073c8b69fadeb4d4617d600fbd5cc.png 2x" data-dominant-color="FAF8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot%20(5)</span><span class="informations">700×450 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I found multiple discussions on reproducing and deterministic here already. For example [<a href="https://discuss.pytorch.org/t/deterministic-non-deterministic-results-with-pytorch/9087">1</a>], [<a href="https://discuss.pytorch.org/t/how-to-get-deterministic-behavior/18177">2</a>], [<a href="https://discuss.pytorch.org/t/random-seed-initialization/7854">3</a>], [<a href="https://discuss.pytorch.org/t/non-reproducible-result-with-gpu/1831/6">4</a>]</p>
<p>From the document we have REPRODUCIBILITY [<a href="https://pytorch.org/docs/master/notes/randomness.html" rel="noopener nofollow ugc">4</a>], MULTIPROCESSING BEST PRACTICES [<a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">5</a>], Asynchronous execution [<a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="noopener nofollow ugc">6</a>].</p>
<p>I have followed all the suggestions and confirmed that I got matching random number and matching initialized weights (diff all the weights and 100% match) in my modules between tests. But my results always end up different.</p>
<p>Combining all previous discussion, here is what I did, at the beginning of my main.py</p>
<pre><code class="lang-auto">import numpy
numpy.random.seed(123)
import random
random.seed(123)

import torch
import torch.nn as nn
import torch.optim as optim
import torch.backends.cudnn as cudnn
from torch.autograd import Variable

torch.backends.cudnn.enabled = False
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.manual_seed(123)
torch.cuda.manual_seed(123)
</code></pre>
<p>In data loader, num_workers=0. There are only two training samples so no shuffle there, flipping turned off.<br>
In the terminal, I set <code>CUDA_LAUNCH_BLOCKING=1 python ./main.py</code></p>
<p>Any other non-deterministic behavior during convolution or back propagation that we need to consider?</p>
<p>Thank you!</p>
<p>Just tried replacing Adam with SGD and the problem remains.</p>

tom: <p>There is a rather long term plan outlined in issue <a href="https://github.com/pytorch/pytorch/issues/15359" rel="nofollow noopener">#15359</a>. We would first flag non-deterministic behaviour and then one could switch to deterministic implementations as desired.<br>
It is, however, hard work, and seems to be one of those things everyone wants to have and no-one wants to do.</p>
<p>Best regards</p>
<p>Thomas</p>

cy-xu: <p>Hi Tom,<br>
Thank you for your reply.</p>
<p>I read your answer to <a href="https://github.com/pytorch/pytorch/issues/12207" rel="nofollow noopener">this issue</a>, regarding non-deterministic behavior during backward(), which is exactly where I located the issue at the moment. All values remain unchanged in forward propagation and the gradient difference appears in both encoder and decoders.</p>
<p>Though I don’t understand how my test machine with a GTX 980T would display 1e-10 or even 1e-12 difference for the gradients, should I trust such minimal difference? Certainly I can’t ignore them as the difference in weights would grow bigger later.</p>
<p>A rookie question - I reproduced example code form pytorch to make sure nothing else caused the problem. But I found <a href="https://github.com/NVlabs/MUNIT/blob/master/networks.py#L206" rel="nofollow noopener">the encoder and decoder pair from here</a> caused most trouble after backward().</p>
<p>I applied all available deterministic methods mentioned in the post, by reducing the conv and res layers in the code I mentioned above, I could see less occurrence of divergence but most of the gradient difference are at level of 1e-4 or 1e-3, isn’t that too big too ignore?</p>
<p>But thank you for your feedback and great work.</p>
<p>Best reagards,<br>
CY</p>

tom: <p>Well, ideally your learning procedure should be robust in the sense that you’d get comparable results from two runs despite the numerical differences.</p>
<p>If you want to have reproducible learning, you need to avoid (or fix!) those functions that introduce the non-linearity. The documentation has a best-knowledge list (I think some particular 2d cross entropy thing is nondeterministic in the forward missing, there is a bug report about that).</p>
<p>If you have one particular function that you want to fix and are willing to dig into that, it’d certainly be interesting, too. For example Chang Rajani in <a href="https://github.com/pytorch/pytorch/issues/17501" rel="nofollow noopener">#17501</a> looking at improving the perfomance and eliminating nondeterminism from <code>torch.fold</code>. Most of the time, function use a cuda function <code>atomicAdd</code> which allows multiple threads to add to the same location. As the order isn’t deterministic, this blows up. There are relatively standard techniques to avoid the atomic add (e.g. changing the order of processing things such that only one thread writes to any given gradient value), but it does take a bit of work.</p>
<p>Best regards</p>
<p>Thomas</p>

cy-xu: <p>Hi Tom, I’m playing around to see how the combination, number and choice of conv layers and residual blocks would affect the results. See if I will find a compromise as a solution for my current network.</p>
<p>Thank you!</p>
<p>Best,<br>
CY</p>