Reproduciblity unable to be guaranteed after torch.compile

Horizon: <p>When I add torch.compile to the training pipeline, the results are getting different in evaluation even I fix the random seed in numpy, random, torch, and torch.cuda.</p>
<p>After I removing torch.compile, the results are deterministic across all.</p>

ptrblck: <p>Are you seeing non-deterministic behavior after following the steps mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">reproducibility docs</a> in <code>torch.compile</code> or are you only seeding the code?</p>

Horizon: <p>Yes, I have followed the documentation, not only set the random seed but also set the cudnn.deteministic and torch.Generator.<br>
Since my code had a lot of uncertainty before, I have followed the documentation to make it can be reproduced a long time ago, but this reproducibility failed after adding torch.compile recently.</p>

ptrblck: <p>Would it be possible to share the model so that we could reproduce it?</p>

Horizon: <p>I’m sorry I can’t share our model for some reason.<br>
But I have observed some phenomena:<br>
First of all, the result difference of executing torch.compile on gpu is less than that of executing torch.compile on cpu.<br>
Secondly, I use AMP in my code, does torch.compile requires to run under AMP mode?</p>

ptrblck: <p>No, <code>torch.compile</code> does not require <code>amp</code>.<br>
Just to understand the issue a bit better: are you seeing differences in the outputs for repeated forward passes after calling <code>torch.compile(model)</code> or are you comparing the plain eager mode model vs. the compiled one?</p>

marksaroufim: <p>Are you seeing a results may differ from eager type of warning in your code? If so you might want to enable this flag <a href="https://github.com/pytorch/pytorch/blob/master/torch/_inductor/config.py#L73" class="inline-onebox" rel="noopener nofollow ugc">pytorch/config.py at master · pytorch/pytorch · GitHub</a></p>
<pre><code class="lang-auto">from torch._inductor.config import iconfig
iconfig.fallback_random = True
</code></pre>
<p>There’s been a new PR added to help with determinism as well here <a href="https://github.com/pytorch/pytorch/pull/98339" class="inline-onebox" rel="noopener nofollow ugc">[inductor] fix scatter fallback and fallback in deterministic mode by yuguo68 · Pull Request #98339 · pytorch/pytorch · GitHub</a></p>
<p>Finally if none of those can solve your problem please try out the accuracy minifier which might indicate which line of your code in your model is diverging post compilation <a href="https://pytorch.org/docs/master/compile/troubleshooting.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 Troubleshooting — PyTorch master documentation</a></p>

Horizon: <p>Yes, the difference you mentioned exists. There is a difference between the output of eager mode and compile mode, and the maximum difference is about 0.0003 under the same input.</p>
<p>But the difference I’m talking about is the results of different runs of the same code after torch.compile.<br>
After the first epoch, the loss value for the first run is 0.001052 and the second is 0.001051.<br>
Since my test data are sensitive to different models, this results in two test losses of 0.000195 and 0.000173 respectively.</p>

huzefunb: <p>I have the same issue with PyTorch 2.0.1</p>