Torch.compile(mode="max-autotune") produces different inference result from eager mode — is this expected?

username: tinywisdom
cooked: <h2><a name="p-467387-inference-result-mismatch-between-eager-mode-and-torchcompilemodemax-autotune-1" class="anchor" href="#p-467387-inference-result-mismatch-between-eager-mode-and-torchcompilemodemax-autotune-1"></a>Inference result mismatch between eager mode and torch.compile(mode=“max-autotune”)</h2>
<p>Hi team,</p>
<p>I’m encountering a result mismatch when running inference using <code>torch.compile</code>.</p>
<h3><a name="p-467387-what-im-doing-2" class="anchor" href="#p-467387-what-im-doing-2"></a>What I’m doing:</h3>
<ul>
<li>I have a PyTorch model defined with standard <code>nn.Module</code>.</li>
</ul>
<pre data-code-wrap="python"><code class="lang-python"># torch.rand(1, 3, 224, 224, dtype=input_dtype)

import torch.nn as nn

class BaseConv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, stride, padding, conv_layer):
        super().__init__()
        self.conv = conv_layer(in_channels, out_channels, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)

    def forward(self, x):
        return self.conv(x)

class ActivatedConv(BaseConv):
    def __init__(self, in_channels, out_channels, kernel_size, stride, padding, conv_layer, activation):
        super().__init__(in_channels, out_channels, kernel_size, stride, padding, conv_layer)
        self.activation = activation

    def forward(self, x):
        return self.activation(self.conv(x))

class NormalizedConv(ActivatedConv):
    def __init__(self, in_channels, out_channels, kernel_size, stride, padding, conv_layer, norm, activation):
        super().__init__(in_channels, out_channels, kernel_size, stride, padding, conv_layer, activation)
        self.norm = norm(out_channels)

    def forward(self, x):
        return self.activation(self.norm(self.conv(x)))

class Conv2DBNReLU(NormalizedConv):
    def __init__(self, in_channels, out_channels, kernel_size, stride, padding):
        super().__init__(in_channels, out_channels, kernel_size, stride, padding, nn.Conv2d, nn.BatchNorm2d, nn.ReLU())

class MyModel(nn.Module):
    def __init__(self, in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1):
        super().__init__()
        self.conv1 = Conv2DBNReLU(in_channels, out_channels, kernel_size, stride, padding)

    def forward(self, x):
        return self.conv1(x)

def my_model_function(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1):
    return MyModel(in_channels, out_channels, kernel_size, stride, padding)

if __name__ == "__main__":
    model = my_model_function()
    print(model)

</code></pre>
<ul>
<li>I run inference in <strong>two different modes</strong>:
<ol>
<li><strong>Eager mode</strong>: <code>output_eager = model(input_tensor)</code></li>
<li><strong>Compiled mode</strong>: <code>output_compiled = torch.compile(model, mode="max-autotune")(input_tensor)</code></li>
</ol>
</li>
</ul>
<h3><a name="p-467387-problem-3" class="anchor" href="#p-467387-problem-3"></a>Problem:</h3>
<p>The two outputs are <strong>not equal</strong>, and the difference exceeds typical tolerances like <code>torch.allclose(output_eager, output_compiled, atol=1e-5)</code>.</p>
<p>This happens consistently on my model when compiled using <code>torch.compile</code> with <code>mode="max-autotune"</code> (and default backend = “inductor”).</p>
<pre><code class="lang-auto">=== Detailed comparison ===
Total number of elements: 3,211,264
Max absolute error: 0.00128412
Mean absolute error: 0.000100889
Max relative error: 23,868.7
Mean relative error: 0.285904
Number of elements exceeding tolerance: 98,102
Percentage of out-of-tolerance elements: 3.05%
Result of torch.allclose(output_eager, output_compiled, atol=1e-5): False
</code></pre>
<h3><a name="p-467387-my-question-4" class="anchor" href="#p-467387-my-question-4"></a>My Question:</h3>
<ul>
<li>Is this difference expected due to aggressive optimizations under <code>max-autotune</code>?</li>
<li>Or is this possibly a bug or unsupported pattern in my model?</li>
</ul>
<p>I’d appreciate any clarification, or suggestions on how to debug this further.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Which PyTorch version are you using and do you see the same or similar error in the latest nightly?</p>

username: tinywisdom
cooked: <p>Hi, thanks for the follow-up.</p>
<p>Previously I was using <strong>PyTorch 2.5.1</strong>, and I have just tested it again with the latest nightly build: <strong>2.6.0.dev20241112+cu121</strong>.</p>
<p>Unfortunately, the issue still persists. Below is the output difference observed when running the model using <code>OriginalModel + Compile(max-autotune)</code>:</p>
<p>=== Output difference details ===</p>
<ul>
<li>Total number of elements: 3,211,264</li>
<li>Max absolute error: 0.00102174</li>
<li>Mean absolute error: 0.000106659</li>
<li>Max relative error: 20,524.3</li>
<li>Mean relative error: 0.344893</li>
<li>Number of elements exceeding tolerance: 106,797</li>
<li>Percentage of elements exceeding tolerance: 3.3257%</li>
<li>Result of <code>torch.allclose(output_eager, output_compiled, atol=1e-5)</code>: False</li>
</ul>
<p>Please let me know if I can provide any additional information to help diagnose this further.</p>

username: richard
cooked: <p>Do you have a script that reproduces the problem? That is, what is the shape/dtype/device of your input tensor?</p>

username: tinywisdom
cooked: <p>Thanks for the response!</p>
<p>Sure — I’ll prepare and share a minimal script that reproduces the issue, along with full details of the input tensor (shape, dtype, and device). Will update here shortly.</p>
<p>By the way, may I be listed as the original reporter if this turns out to be a confirmed issue?</p>
<p>Thanks again!</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import importlib.util
import os

def load_model_from_file(module_path, model_function_name="my_model_function"):
    model_file = os.path.basename(module_path)[:-3]
    spec = importlib.util.spec_from_file_location(model_file, module_path)
    model_module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(model_module)

    model_function = getattr(model_module, model_function_name)
    model = model_function()
    return model



def compare_outputs(a: torch.Tensor, b: torch.Tensor, atol=1e-5, rtol=1e-3):
    print("=== Output difference comparison ===")
    diff = a - b
    abs_diff = diff.abs()
    rel_diff = abs_diff / (a.abs() + 1e-8)
    total_elements = a.numel()
    print(f"- Total elements: {total_elements}")
    print(f"- Max absolute error: {abs_diff.max().item():.8f}")
    print(f"- Mean absolute error: {abs_diff.mean().item():.8f}")
    print(f"- Max relative error: {rel_diff.max().item():.8f}")
    print(f"- Mean relative error: {rel_diff.mean().item():.8f}")
    num_exceed = (~torch.isclose(a, b, atol=atol, rtol=rtol)).sum().item()
    print(f"- Elements exceeding tolerance: {num_exceed}")
    print(f"- Percentage exceeding tolerance: {100.0 * num_exceed / total_elements:.4f}%")
    print(f"- torch.allclose: {torch.allclose(a, b, atol=atol, rtol=rtol)}")



if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    input_tensor = torch.rand(1, 3, 224, 224, device=device)

    model_path = "xxx/xxx/xxx/xxx.py"
    model = load_model_from_file(model_path).to(device).eval()

    with torch.no_grad():
        output_eager = model(input_tensor)

    compiled_model = torch.compile(model, mode="max-autotune")
    with torch.no_grad():
        output_compiled = compiled_model(input_tensor)

    compare_outputs(output_eager, output_compiled)

</code></pre>

username: richard
cooked: <p><a class="mention" href="/u/tinywisdom">@tinywisdom</a> this sounds like a bug to me so far. Please feel free to open an issue directly on GitHub: <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox">GitHub · Where software is built</a></p>

username: tinywisdom
cooked: <p>Thank you so much for the response — I really appreciate it!</p>
<p>I’ll keep all future updates, findings, and reproductions in this thread to keep everything in one place.</p>
<p>Thanks again for your time and help!</p>