Questions about Pytorch 2.0 tutorial and TORCH_COMPILE_DEBUG=1

Phil_Tomson: <p>I’m going through the Pytorch 2.0 tutorial here <a href="https://pytorch.org/blog/Accelerating-Hugging-Face-and-TIMM-models/" class="inline-onebox" rel="noopener nofollow ugc">Accelerating Hugging Face and TIMM models with PyTorch 2.0 | PyTorch</a> ( Accelerating Hugging Face and TIMM models with PyTorch 2.0).</p>
<p>When I get to the section “A toy example” there’s a point where they say you can run the toy model with:</p>
<pre><code class="lang-auto">TORCH_COMPILE_DEBUG=1 python trig.py
</code></pre>
<p>And this apparently is supposed to generate the output:</p>
<pre><code class="lang-auto">@pointwise(size_hints=[16384], filename=__file__, meta={'signature': {0: '*fp32', 1: '*fp32', 2: 'i32'}, 'device': 0, 'constants': {}, 'configs': [instance_descriptor(divisible_by_16=(0, 1, 2), equal_to_1=())]})
@triton.jit
def kernel(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
    xnumel = 10000
    xoffset = tl.program_id(0) * XBLOCK
    xindex = xoffset + tl.reshape(tl.arange(0, XBLOCK), [XBLOCK])
    xmask = xindex &lt; xnumel
    x0 = xindex
    tmp0 = tl.load(in_ptr0 + (x0), xmask)
    tmp1 = tl.sin(tmp0)
    tmp2 = tl.sin(tmp1)
    tl.store(out_ptr0 + (x0 + tl.zeros([XBLOCK], tl.int32)), tmp2, xmask)

</code></pre>
<p>However, I do not see that output. I see a lot of debug output. I also notice that a torch_compile_debug directory has been created so I went into that to see if that’s where I coudl find the specified output. There’s a timestamp directory that’s created and under that directory I see two subdirectories: aot_torchinductor and torchdynamo.</p>
<p>In torchdynamo there’s a debug.log with the following:</p>
<pre><code class="lang-auto">skipping __init__ /usr/lib/python3.10/contextlib.py
skipping __enter__ /usr/lib/python3.10/contextlib.py
skipping __init__ /usr/lib/python3.10/contextlib.py
skipping __enter__ /usr/lib/python3.10/contextlib.py
skipping enable_dynamic /home/phil/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
Step 1: torchdynamo start tracing fn
TRACE starts_line /home/phil/devel/pytorch/trig.py:3
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR sin [TorchVariable(&lt;module 'torch' from '/home/phil/.local/lib/python3.10/site-packages/torch/__init__.py'&gt;)]
TRACE LOAD_FAST x [TorchVariable(&lt;built-in method sin of type object at 0x7fa801eb5080&gt;)]
TRACE CALL_FUNCTION 1 [TorchVariable(&lt;built-in method sin of type object at 0x7fa801eb5080&gt;), TensorVariable()]
TRACE LOAD_ATTR cuda [TensorVariable()]
TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), cuda)]
TRACE STORE_FAST a [TensorVariable()]
TRACE starts_line /home/phil/devel/pytorch/trig.py:4
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR sin [TorchVariable(&lt;module 'torch' from '/home/phil/.local/lib/python3.10/site-packages/torch/__init__.py'&gt;)]
TRACE LOAD_FAST y [TorchVariable(&lt;built-in method sin of type object at 0x7fa801eb5080&gt;)]
TRACE CALL_FUNCTION 1 [TorchVariable(&lt;built-in method sin of type object at 0x7fa801eb5080&gt;), TensorVariable()]
TRACE LOAD_ATTR cuda [TensorVariable()]
TRACE CALL_FUNCTION 0 [GetAttrVariable(TensorVariable(), cuda)]
TRACE STORE_FAST b [TensorVariable()]
TRACE starts_line /home/phil/devel/pytorch/trig.py:5
TRACE LOAD_FAST a []
TRACE LOAD_FAST b [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), TensorVariable()]
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing fn (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to None
Step 2: calling compiler function debug_wrapper
Step 2: done compiler function debug_wrapper
skipping _fn /home/phil/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
skipping nothing /home/phil/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py
skipping __exit__ /usr/lib/python3.10/contextlib.py
skipping __exit__ /usr/lib/python3.10/contextlib.py
~                                                      
</code></pre>
<p>In the aot_torchinductor dir there’s a file called aot_model__0_debug.log that has:</p>
<pre><code class="lang-auto">[aot_autograd.py:1054 DEBUG] ====== Forward (only) graph 0 ======
[aot_autograd.py:1055 DEBUG] class &lt;lambda&gt;(torch.nn.Module):
    def forward(self, arg0_1: f32[10000], arg1_1: f32[10000]):
        # File: /home/phil/devel/pytorch/trig.py:3, code: a = torch.sin(x).cuda()
        sin: f32[10000] = torch.ops.aten.sin.default(arg0_1);  arg0_1 = None

        # File: /home/phil/devel/pytorch/trig.py:4, code: b = torch.sin(y).cuda()
        sin_1: f32[10000] = torch.ops.aten.sin.default(arg1_1);  arg1_1 = None

        # File: /home/phil/devel/pytorch/trig.py:5, code: return a + b
        add: f32[10000] = torch.ops.aten.add.Tensor(sin, sin_1);  sin = sin_1 = None
        return (add,)
</code></pre>
<p>Which looks a bit closer to what was supposed to be expected, but still quite different.</p>
<p>Also, further down in the tutorial it on the Huggingface/BERT example it says:</p>
<pre><code class="lang-auto">If you remove the `to(device="cuda:0")` from the model and `encoded_input` then PyTorch 2.0 will generate C++ kernels that will be optimized for running on your CPU.
</code></pre>
<p>I tried that, but don’t see any C++ output anywhere.</p>

EstherBear: <p>Hi,</p>
<p>I think you can find what you expect at …/aot_torchinductor/model__0_inference_0.0/output_code.py.</p>

marksaroufim: <p><a class="mention" href="/u/estherbear">@EstherBear</a> is correct there should be a <code>tmp</code> folder printed in your console that shows where the inductor code is printed, we could make this clearer and package everythign in the same directory cc <a class="mention" href="/u/mlazos">@mlazos</a></p>

mlazos: <aside class="quote no-group" data-username="Phil_Tomson" data-post="1" data-topic="171150">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phil_tomson/48/56747_2.png" class="avatar"> Phil_Tomson:</div>
<blockquote>
<p>Accelerating Hugging Face and TIMM models with PyTorch 2.0 | PyTorch</p>
</blockquote>
</aside>
<p>By default the python output code should be written in /torch_compile_debug/run_/aot_torchinductor/model_/outputcode.py</p>
<p>There will be a separate model_… directory for each subgraph that is compiled.</p>