Inconsistent Results with torch.compile on Identical Environments and GPUs

yuchunjia: <p>When using <code>torch.compile</code> in both <code>default</code> and <code>max-autotune</code> compilation modes, the results differ across machines with the same GPU and environment. However, when <code>torch.compile</code> is disabled, the results align perfectly. How can I troubleshoot this issue? Any help would be greatly appreciated.</p>

ptrblck: <p>How large are these differences and is one of the compiled outputs matching PyTorch eager mode?<br>
Also, do you see these issues in the latest release?</p>

yuchunjia: <blockquote>
<p>How large are these differences and is one of the compiled outputs matching PyTorch eager mode?</p>
</blockquote>
<p>We are using the DiT model, and we found that when BF16 is used without enabling compile, different machines can produce identical images. However, after enabling compile, the results become uncontrollable. Even when we align the random seed and use an entirely consistent environment, we cannot obtain exactly the same images.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e94b7715ab2bb332bd42f834af2b7e9231214707.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e94b7715ab2bb332bd42f834af2b7e9231214707" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e94b7715ab2bb332bd42f834af2b7e9231214707_2_690x230.jpeg" alt="image" data-base62-sha1="xhOVUjyRbennTN8LJCV51bx6MBh" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e94b7715ab2bb332bd42f834af2b7e9231214707_2_690x230.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e94b7715ab2bb332bd42f834af2b7e9231214707_2_1035x345.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e94b7715ab2bb332bd42f834af2b7e9231214707_2_1380x460.jpeg 2x" data-dominant-color="A79A92"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1646×550 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>Also, do you see these issues in the latest release?</p>
</blockquote>
<p>Yes, we have followed the issues below and made the corresponding settings, but we still cannot control the results with compile enabled.</p>
<p>torch._inductor.config.fallback_random = True</p>
<aside class="quote" data-post="6" data-topic="176845">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/reproduciblity-unable-to-be-guaranteed-after-torch-compile/176845/6">Reproduciblity unable to be guaranteed after torch.compile</a> <a class="badge-category__wrapper " href="/c/compile/41"><span data-category-id="41" style="--category-badge-color: #0088CC; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category for torch.compile and PyTorch 2.0 related compiler issues.
This includes: issues around TorchDynamo ( torch._dynamo ), TorchInductor (torch._inductor) and AOTAutograd"><span class="badge-category__name">torch.compile</span></span></a>
  </div>
  <blockquote>
    No, torch.compile does not require amp. 
Just to understand the issue a bit better: are you seeing differences in the outputs for repeated forward passes after calling torch.compile(model) or are you comparing the plain eager mode model vs. the compiled one?
  </blockquote>
</aside>

<p>Here is our software and hardware information:<br>
torch version: 2.6.0.dev20241104+cu124<br>
torch cuda version: 12.4<br>
torch git version: 76c297ad6b180015c5955e61357f43ab92d110ea</p>
<p>CUDA Info:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2024 NVIDIA Corporation<br>
Built on Thu_Jun__6_02:18:23_PDT_2024<br>
Cuda compilation tools, release 12.5, V12.5.82<br>
Build cuda_12.5.r12.5/compiler.34385749_0</p>
<p>GPU Hardware Info:<br>
NVIDIA L20 : 2</p>
<p>How can I troubleshoot this issue? Any help would be greatly appreciated.</p>

ptrblck: <p>The <a href="https://pytorch.org/docs/stable/torch.compiler_troubleshooting.html#pytorch-2-0-troubleshooting">Troubleshooting Guide</a> might be a good start allowing you to find the smallest subgraph which reproduces accuracy issues between eager mode and the compiled model.</p>