Fusing operators in torch.compile for Codegen

username: adityat
cooked: <p>I was trying to understand the exact logic for fusing a set of operators before codegen.</p>
<p>Consider this sub-graph:</p>
<pre><code class="lang-auto">class FuseSiLU(nn.Module):
    def __init__(self):
        super(SiLU, self).__init__()
        

    def forward(self, x):
        x = x * F.silu(x)
        return x

</code></pre>
<p>I am running this model using inductor backend in torch.compile. On further analysis with profiler, with this flag enabled:</p>
<blockquote>
<p>from torch._inductor import config<br>
config.cpp.enable_kernel_profile = True</p>
</blockquote>
<p>I can see this operator in profile log.</p>
<blockquote>
<p><strong>graph_0_cpp_fused_mul_silu_</strong></p>
</blockquote>
<p>I want to understand where is the decison being taken to fuse a bunch of op’s before codegen.<br>
So far I’ve looked at some of the earlier answers for torch.inductor pattern-matcher but could’t pin the logic for this one. Any help would be much appreciated.</p>

username: adityat
cooked: <p>I’ve followed some threads , but  it did’t clarify this:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://towardsdatascience.com/how-pytorch-2-0-accelerates-deep-learning-with-operator-fusion-and-cpu-gpu-code-generation-35132a85bd26?gi=185e1d63ccac">
  <header class="source">
      <img src="https://miro.medium.com/v2/resize:fill:256:256/1*VzTUkfeGymHP4Bvav-T-lA.png" class="site-icon" width="207" height="206">

      <a href="https://towardsdatascience.com/how-pytorch-2-0-accelerates-deep-learning-with-operator-fusion-and-cpu-gpu-code-generation-35132a85bd26?gi=185e1d63ccac" target="_blank" rel="noopener nofollow ugc" title="05:35PM - 20 April 2023">Medium – 20 Apr 23</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/395;"><img src="https://miro.medium.com/v2/resize:fit:1003/1*m3hHjaqAY_Rqwcvgfg4IoQ.png" class="thumbnail" width="690" height="395"></div>

<h3><a href="https://towardsdatascience.com/how-pytorch-2-0-accelerates-deep-learning-with-operator-fusion-and-cpu-gpu-code-generation-35132a85bd26?gi=185e1d63ccac" target="_blank" rel="noopener nofollow ugc">How Pytorch 2.0 accelerates deep learning with operator fusion and CPU/GPU...</a></h3>

  <p>A primer on deep learning compiler technologies in PyTorch for graph capture, intermediate representations, operator fusion, and more</p>

  <p>
    <span class="label1">Reading time: 17 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox discoursetopic" data-onebox-src="https://dev-discuss.pytorch.org/t/how-is-pattern-matching-in-inductor-fx-implemented/1720">
  <header class="source">
      <img src="https://global.discourse-cdn.com/standard10/uploads/pytorch1/optimized/1X/6cd7da56682d360e2c6006ff3e31eb250c5a8675_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://dev-discuss.pytorch.org/t/how-is-pattern-matching-in-inductor-fx-implemented/1720" target="_blank" rel="noopener nofollow ugc" title="03:56PM - 09 December 2023">PyTorch Developer Mailing List – 9 Dec 23</a>
  </header>

  <article class="onebox-body">
    <img src="https://global.discourse-cdn.com/standard10/uploads/pytorch1/original/1X/6cd7da56682d360e2c6006ff3e31eb250c5a8675.png" class="thumbnail onebox-avatar" width="500" height="500">

<div class="title-wrapper">
  <h3><a href="https://dev-discuss.pytorch.org/t/how-is-pattern-matching-in-inductor-fx-implemented/1720" target="_blank" rel="noopener nofollow ugc">How is pattern matching in inductor/fx implemented?</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #BF1E2E;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">compiler</span>
        </span>
      </span>
  </div>
</div>

  <p>Inductor and FX heavily use pattern matching to match and replace subgraph patterns for optimization. However, subgraph matching (finding a subgraph that is isomorphic to a given graph) is a well-known NP-hard problem. What is the mental model for...</p>

  <p>
    <span class="label1">Reading time: 1 mins 🕑</span>
      <span class="label2">Likes: 7 ❤</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>