Same model, same parameters, different result for same input

username: fdokic
cooked: <p>Hi all</p>
<p>I have two different instantiations of a model, gn and gx, both consisting of following operators:<br>
I instantiated gx using exactly the same parameters as gn (reference).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25f168d7f638a8391e937be2d359c9a1b68dfd06.png" data-download-href="https://discuss.pytorch.org/uploads/default/25f168d7f638a8391e937be2d359c9a1b68dfd06" title="Image Pasted at 2020-2-19 14-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25f168d7f638a8391e937be2d359c9a1b68dfd06_2_583x500.png" alt="Image Pasted at 2020-2-19 14-47" data-base62-sha1="5pESNZilwIKnN4YHOfYDsqAKtz8" width="583" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25f168d7f638a8391e937be2d359c9a1b68dfd06_2_583x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25f168d7f638a8391e937be2d359c9a1b68dfd06_2_874x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/25f168d7f638a8391e937be2d359c9a1b68dfd06_2_1166x1000.png 2x" data-dominant-color="181818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Image Pasted at 2020-2-19 14-47</span><span class="informations">1668×1430 251 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>what I did:</p>
<ul>
<li>I passed the same input, operator for operator through both models and saved the intermediate results in lists  (Sequential and custom class shouldn’t matter, right? hint that it doesn’t: other model works exactly using that setting)</li>
<li>the results differ from the 5th last layer (from convTrans2d)</li>
<li>the parameters of all operators are the same: checked using:</li>
</ul>
<pre><code class="lang-auto">out_x = list(gx.parameters())
out_n = list(gn.parameters())
for i in range(len(out_x)):
    print(torch.all(out_x[i].eq(out_n[i])).item(), torch.allclose(out_x[i], out_n[i]))
</code></pre>
<p>they are all true (same paramters)</p>
<ul>
<li>storing the intermediate results of each layer in out_x for gx and out_n for gn and comparing the differences I get following:</li>
</ul>
<pre><code class="lang-auto">In [12]: [torch.sum(torch.abs(out_x[i] - out_n[i])).item() for i in range(len(out_x))]
Out[12]:
[0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.0,
 0.00011764606460928917,
 0.00026547431480139494,
 0.00026547431480139494,
 0.0001205947482958436,
 0.00011362379882484674]
</code></pre>
<p>Does anyone have a clue what the reason could be? I don’t unterstand why it changes suddenly in the end, if parameters would be the cause it could differ earlier?</p>
<p>The other model (similar size &amp; operators) also has slightly different hyperparamters e.g. alphas of LeakyReLu but still manages to produce equal outputs…</p>
<p>Any ideas are highly appreciated!</p>

username: ptrblck
cooked: <p>Did you call <code>model.eval()</code> on both models and could you also check the <code>.buffers()</code>?</p>

username: fdokic
cooked: <p>Yes I did, the buffers are also mutually equal. There should not be other sources for non determinism for this model right? So what else could be the reason? Thanks!</p>

username: ptrblck
cooked: <p>Could you post the model definition by wrapping the code into three backticks ```?</p>
<p>Also, are you using e.g. <code>torch.backends.cudnn.deterministic = True</code> as described <a href="https://pytorch.org/docs/stable/notes/randomness.html">here</a>?</p>

username: fdokic
cooked: <p>This was it! Thank you Peter!</p>