Torch.compile drops the performance of validation

username: HelloWorldF
cooked: <p>I encountered that after adding torch.compile(model), the performance of the model on the verification set has been declining as the training progresses.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/31973e81b26f2b05eb8da30725779d1a1851227b.png" data-download-href="https://discuss.pytorch.org/uploads/default/31973e81b26f2b05eb8da30725779d1a1851227b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31973e81b26f2b05eb8da30725779d1a1851227b_2_690x354.png" alt="image" data-base62-sha1="74HqYQBInnmpt5WHtY3HpZhYL5V" width="690" height="354" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31973e81b26f2b05eb8da30725779d1a1851227b_2_690x354.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31973e81b26f2b05eb8da30725779d1a1851227b_2_1035x531.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/31973e81b26f2b05eb8da30725779d1a1851227b_2_1380x708.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1410×724 53.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>When I remove torch.compile(model), the model is back to normal:<br>
epoch 1/1000, train: loss=0.0767, val: psnr=27.5813, 1.4m 1.4m/24.2h<br>
epoch 2/1000, train: loss=0.0438, val: psnr=28.2431, 1.6m 3.0m/25.3h<br>
epoch 3/1000, train: loss=0.0428, val: psnr=28.2909, 1.6m 4.6m/25.5h<br>
epoch 4/1000, train: loss=0.0417, val: psnr=28.4744, 1.6m 6.2m/25.9h<br>
epoch 5/1000, train: loss=0.0406, val: psnr=28.7785, 1.6m 7.8m/26.1h<br>
epoch 6/1000, train: loss=0.0404, val: psnr=28.5917, 1.6m 9.5m/26.3h<br>
epoch 7/1000, train: loss=0.0395, val: psnr=28.6923, 1.6m 11.1m/26.3h<br>
epoch 8/1000, train: loss=0.0393, val: psnr=28.9986, 1.5m 12.6m/26.2h<br>
epoch 9/1000, train: loss=0.0382, val: psnr=28.8534, 1.7m 14.2m/26.4h</p>
<p>I work with torch2.0+cuda11.8+cudnn8700, RTX 4090</p>

username: marksaroufim
cooked: <p>Do you have a repro you can share?</p>
<p>In the meantime here are some things you can try</p>
<ol>
<li>Randomness is different across inductor and torch eager, so set this flag to false <a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/config.py#L81" class="inline-onebox" rel="noopener nofollow ugc">pytorch/config.py at main · pytorch/pytorch · GitHub</a>
</li>
<li>Try using the nightlies, some recent PRs were added to improve determinism  <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>
</li>
<li>If neither of those two things work then it’s likely there’s a bug with some specific operation and you can run the accuracy minifier to point it out <a href="https://pytorch.org/docs/master/compile/troubleshooting.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 Troubleshooting — PyTorch master documentation</a>
</li>
</ol>
<pre><code class="lang-auto">`TORCHDYNAMO_REPRO_AFTER="dynamo" TORCHDYNAMO_REPRO_LEVEL=4`
`TORCHDYNAMO_REPRO_AFTER="aot" TORCHDYNAMO_REPRO_LEVEL=4`
</code></pre>

username: HelloWorldF
cooked: <p>Those two things do not work. You can reproduce the similar result with this <a href="https://github.com/XieQi2015/F-Conv/tree/main/MinistExp" rel="noopener nofollow ugc">repro</a>.</p>
<p>You can implement with the simple case, by adding the model = torch.compile(model)  below <a href="https://github.com/XieQi2015/F-Conv/blob/c55c4184a0cf6e7a30be62217c781264853a0515/MinistExp/Rotated_MNIST_simpleCase_Main.py#L46" rel="noopener nofollow ugc">here</a>.</p>
<p>After training 30 epochs, the model begin to test. If you add the torch.compile, the test accuracy will not increase. While for the version without torch.compile is normal.</p>

username: HelloWorldF
cooked: <p>Here is the requirements.txt <a href="https://drive.google.com/file/d/1ANH1ShiKqsdsdqkeubYj-pHQJ4LcnYTp/view?usp=sharing" rel="noopener nofollow ugc">here</a>.</p>

username: marksaroufim
cooked: <p>So this doesn’t get lost do you mind x-posting to <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> - an oncall can help take a look</p>

username: HelloWorldF
cooked: <p>Could you reproduce the same error?</p>