Does dynamo trigger real kernel execution?

yjguo: <p>I think the answer is NO according to “<a href="https://pytorch.org/docs/stable/torch.compiler_dynamo_overview.html" class="inline-onebox" rel="noopener nofollow ugc">Dynamo Overview — PyTorch 2.5 documentation</a>”:</p>
<blockquote>
<p>Dynamo hooks into the frame evaluation API in CPython (<a href="https://peps.python.org/pep-0523/" rel="noopener nofollow ugc">PEP 523</a>) to dynamically modify Python bytecode right <code>before</code> it is executed.<br>
It creates this FX Graph <code>through bytecode analysis</code></p>
</blockquote>
<p>With above statement, I think dynamo does the python bytecode analysis w/o executing the real kernels, to generate a computation graph in torch IR.</p>
<p>But, I tried the below code, and I think it does not match my understanding. It looks that dynamo triggers real kernel execution on device.</p>
<p>Since I only care about dynamo, so I choose the backend as ‘eager’ below, to not involve AOTAutograd.</p>
<p>Case 1:</p>
<pre><code class="lang-auto">import torch

class TestNet(torch.nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.linear = torch.nn.Linear(3, 2)

    def forward(self, x):
        x = x * 2
        if x.sum() &gt; 0:
            x = torch.sin(x)
        if x.mean() &gt; 0:
            x = torch.cos(x)
        x = self.linear(x)
        return x

m = TestNet().cuda()
m = torch.compile(m, backend="eager")
inputs = torch.ones(3).cuda()
m(inputs)
</code></pre>
<p>with <code>TORCH_LOGS="+dynamo" python -u eagerbackend.py</code>, we can see from the log that <code>sin</code> and <code>cos</code> are in the computation graphs.</p>
<p>Case 2:</p>
<pre><code class="lang-auto">import torch

class TestNet(torch.nn.Module):
    def __init__(self):
        super(TestNet, self).__init__()
        self.linear = torch.nn.Linear(3, 2)

    def forward(self, x):
        x = x * 2
        if x.sum() &gt; 0:
            x = torch.sin(x)
        if x.mean() &gt; 0:
            x = torch.cos(x)
        x = self.linear(x)
        return x

m = TestNet().cuda()
m = torch.compile(m, backend="eager")
inputs = torch.ones(3).cuda() * -1.0
m(inputs)
</code></pre>
<p>With <code>TORCH_LOGS="+dynamo" python -u eagerbackend.py</code>, we do not see anything about <code>sin</code> and <code>cos</code> in the log.</p>
<p>So, putting the two cases together, I think that case 1 implies that the code goes into the <code>if</code> body, and case 2 implies that the code does not go into the ‘if’ body.  That’s correct, BUT, we can only know it by really executing the kernel, and so we can know the value of <code>x.sum()</code> and <code>x.mean()</code>.</p>
<p>This experiment gives me the answer that dynamo triggers the real kernel execution, but I think it conflicts with the document of ‘dynamo overview’.  Could you help to clarification? thanks.</p>

ptrblck: <p>On your code snippets you are not using pure Dynamo operations but are executing the model after calling <code>torch.compile</code> on it. As the docs also explain, after Dynamo’s processing, a backend will be used to code-gen and/or execute the kernels. So of course the forward pass of a compiled model will show executed kernels.</p>

yjguo: <p>thanks for the replay.</p>
<p>My understanding is that <code>m(inputs)</code> will trigger two steps:</p>
<ol>
<li>dynamo converts the python bytecode to FX graph.  And <code>TORCH_LOGS="+dynamo" </code> shows what the FX graph looks like.</li>
<li>the FX graph is executed. I understand there’s real kernel execution at this step.</li>
</ol>
<p>My question is if there’s real kernel execution in the step 1.  From the log of case 1 and case 2, the generated FX graphs from Dynamo (in step 1) are different, and so I think it implies that there’s real kernel execution in step 1. Otherwise, we should see the same FX graph for both case 1 and case 2 in step 1.</p>

yjguo: <p><a href="https://pytorch.org/tutorials/intermediate/compiled_autograd_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Compiled Autograd: Capturing a larger backward graph for torch.compile — PyTorch Tutorials 2.6.0+cu124 documentation</a> mentions that:</p>
<ul>
<li>Dynamo intercepts the Python bytecode, “simulates” their execution and records the operations into a graph.</li>
</ul>
<p>what does “simulates” mean, is there a real kernel execution within Dynamo?</p>