Subtle Numerical Mismatch in C++ LibTorch: Eager GPU Mode vs. CUDA Graph/CPU

prateekp: <p>We are debugging a very subtle and persistent numerical mismatch issue with a custom C++ LibTorch inference server and would appreciate any expert insight.</p>
<p><strong>Summary of the Problem:</strong> Our C++ server loads a TorchScript model and runs it on an L4 GPU. When running in standard <strong>Eager GPU mode</strong>, it produces a small (<code>~0.15%</code>) but deterministic score mismatch on specific requests.</p>
<p>The mismatch is compared against our “golden” scores, which are generated from a Python script. Crucially, these golden scores are also perfectly matched by the <strong>Triton Inference Server</strong> (using its LibTorch backend) running on the exact same machine and L4 GPU.</p>
<h3><a name="p-474324-environment-details-1" class="anchor" href="#p-474324-environment-details-1"></a>Environment Details</h3>
<ul>
<li>
<p><strong>LibTorch:</strong> 2.7.1 (CUDA build)</p>
</li>
<li>
<p><strong>CUDA Toolkit (for C++):</strong> 12.9</p>
</li>
<li>
<p><strong>cuDNN (in Python/Triton env):</strong> 9.1.0</p>
</li>
<li>
<p><strong>GPU:</strong> NVIDIA L4</p>
</li>
<li>
<p><strong>C++ Compiler:</strong> Clang++</p>
</li>
</ul>
<h3><a name="p-474324-the-core-anomaly-2" class="anchor" href="#p-474324-the-core-anomaly-2"></a>The Core Anomaly</h3>
<p>Here is the behavior we can reproduce 100% of the time:</p>
<ol>
<li>
<p><strong>Triton Server (LibTorch Backend) vs. Python:</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=14" title=":white_check_mark:" class="emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"> Scores match perfectly.</p>
</li>
<li>
<p><strong>C++ CPU Mode vs. Python:</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=14" title=":white_check_mark:" class="emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"> Scores match perfectly.</p>
</li>
<li>
<p><strong>C++ CUDA Graph Mode vs. Python:</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=14" title=":white_check_mark:" class="emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"> Scores match perfectly.</p>
</li>
<li>
<p><strong>C++ Eager GPU Mode vs. Python:</strong> <img src="https://discuss.pytorch.org/images/emoji/apple/cross_mark.png?v=14" title=":cross_mark:" class="emoji" alt=":cross_mark:" loading="lazy" width="20" height="20"> Fails with a <code>~0.15%</code> relative error on specific, repeatable requests.</p>
</li>
</ol>
<p>We’ve also observed a “warm-up” effect: the first run after a server restart fails on a different set of requests than all subsequent runs, which then become consistent.</p>
<h3><a name="p-474324-what-we-have-proven-and-ruled-out-3" class="anchor" href="#p-474324-what-we-have-proven-and-ruled-out-3"></a>What We Have Proven and Ruled Out</h3>
<p>We have done extensive debugging and have confirmed the following with profiler logs and tests:</p>
<ol>
<li>
<p><strong>C++ Kernels are Identical in Eager vs. Graph Mode:</strong> This is the most confusing finding. We used <code>nsys</code> to capture the kernel launches in both the failing (Eager GPU) and passing (CUDA Graph) modes. The profiler reports show that the <strong>exact same set of non-cuDNN kernels</strong> are being executed in both modes. This rules out the theory that the graph capture selects a “simpler” or “different” kernel. Althought the profile may iteself in kernel tracing mode cause this</p>
</li>
<li>
<p><strong>Synchronization Doesn’t Help:</strong> Running the C++ server with <code>export CUDA_LAUNCH_BLOCKING=1</code> to force synchronous execution <strong>does not</strong> solve the mismatch.</p>
</li>
<li>
<p><strong>JIT Optimizations Don’t Seem to be the Cause:</strong> Disabling the JIT Graph Executor (<code>torch::jit::setGraphExecutorOptimize(false)</code>) <strong>does not</strong> solve the mismatch.</p>
</li>
<li>
<p><strong>TF32 is Not the Cause:</strong> We confirmed that TF32 for <code>matmul</code> (cuBLAS) is disabled in both the Python and C++ environments.</p>
</li>
</ol>
<p>Given that our C++ Eager and Graph modes use the  <strong>same set of GPU kernels</strong>, why would the Eager mode produce a numerical mismatch while the Graph mode does not?</p>
<p>The failure seems to be caused by the <strong>execution context</strong> of launching kernels eagerly from multiple threads, but we can’t find the mechanism. It seems that the non-cuDNN kernels, when launched eagerly, are sensitive to some host-side “noise” (like scheduling jitter) that is eliminated by the rigid context of a CUDA Graph.</p>
<p>Has anyone seen behavior like this before? Is there a known issue or a runtime setting we might be missing that controls the numerical stability of eager-mode execution in a multi-threaded LibTorch server?</p>