PyTorch dynamo is unstable in the 1st epoch and took more time than without compile

bilzrd: <p>I tested compile feature of PyTorch 2.0 on my project.</p>
<p>Although 2nd epoch and after is a bit faster, PyTorch dynamo is unstable in the 1st epoch and took more time than without compile.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/9983c9e61092e138e93b5b6fe4227da040186815.png" data-download-href="https://discuss.pytorch.org/uploads/default/9983c9e61092e138e93b5b6fe4227da040186815" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9983c9e61092e138e93b5b6fe4227da040186815_2_690x195.png" alt="image" data-base62-sha1="lU3plN8hHrmjCTE3fUSfF4ZxhZ3" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9983c9e61092e138e93b5b6fe4227da040186815_2_690x195.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9983c9e61092e138e93b5b6fe4227da040186815_2_1035x292.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/9983c9e61092e138e93b5b6fe4227da040186815_2_1380x390.png 2x" data-dominant-color="232220"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1400×396 36.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/4/044ea69816cf5eda4bb5a63ebc4d01d22c50864b.png" data-download-href="https://discuss.pytorch.org/uploads/default/044ea69816cf5eda4bb5a63ebc4d01d22c50864b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/044ea69816cf5eda4bb5a63ebc4d01d22c50864b.png" alt="image" data-base62-sha1="C6pW9upGTfd0eTPY4RB9VX2PlF" width="555" height="500" data-dominant-color="302922"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">744×670 12.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this an expected behavior?<br>
If it isn’t, how to debug this?</p>
<p>In the first epoch, lots of warning messages:</p>
<p>to diagnose recompilation issues, see <a href="https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.torch._inductor.ir:" rel="noopener nofollow ugc">https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.torch._inductor.ir:</a> [WARNING] Using FallbackKernel: torch.ops.aten.randint.default</p>

ptrblck: <p>Could you create an issue on GitHub, please, as the code owners of TorchDynamo are rarely active here?</p>

bilzrd: <p>I see. Should I post issue on pytorch/torchdynamo instead of pytorch/pytorch?</p>

ptrblck: <p>I would create the issue directly in <a href="https://github.com/pytorch/pytorch/issues"><code>pytorch/pytorch</code></a> as dynamo should now be part of the core framework. The code owners could cross-post it into the <code>pytorch/torchdynamo</code> repo if needed.</p>

bilzrd: <p>I see. However, since I found the trouble shooting guide on this page, so I will read this first, and will post issue if needed.<br>
<a href="https://pytorch.org/docs/master/dynamo/troubleshooting.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/dynamo/troubleshooting.html</a></p>
<p>Thank you for the advice.</p>

wconstab: <p>You may be encountering dynamic shapes in your model, causing recompilation.  The periods of low gpu utilization could be compilation time, and the second epoch could be better if the majority of input shapes have been seen before. You can use .explain or recompilation profiler (see Troubleshooting guide)</p>
<p>Posting issues on pytorch/pytorch is also ok, we may end up redirecting all issues there soon, but it’s true we originally asked for issues to be filed on pytorch/torchdynamo since that’s where we developed before launch.</p>