NvFuser with torch.compile

username: trusira
cooked: <p>Hi,<br>
What is the status of Dynamo support for NvFuser? I could not find many references/tutorials on the topic and the ones I found seem deprecated and/or use TorchScript as the graph format.<br>
Is it possible to use NvFuser with torch.compile?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p><code>nvFuser</code> is deprecated and the default backend for <code>TorchScript</code> (which is also in maintenance mode) was switched back to NNC in <a href="https://github.com/pytorch/pytorch/pull/105185">this PR</a>.</p>

username: trusira
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the clarification. I see NvFuser listed as a prospective codegen backend for Pytorch 2.0 here (<a href="https://pytorch.org/get-started/pytorch-2.0/#developervendor-experience" class="inline-onebox" rel="noopener nofollow ugc">PyTorch 2.0 | PyTorch</a>) and I believe that support is not available yet.<br>
Can you elaborate on what either nvFuser or NNC can offer in this context given that we already have Triton generating code for GPUs?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="trusira" data-post="3" data-topic="187829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/trusira/48/63387_2.png" class="avatar"> trusira:</div>
<blockquote>
<p>and I believe that support is not available yet.</p>
</blockquote>
</aside>
<p>This is correct and the <code>nvFuser</code> support in <code>TorchDynamo</code> was removed <a href="https://github.com/pytorch/pytorch/pull/105789">here</a>.</p>
<aside class="quote no-group" data-username="trusira" data-post="3" data-topic="187829">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/trusira/48/63387_2.png" class="avatar"> trusira:</div>
<blockquote>
<p>Can you elaborate on what either nvFuser or NNC can offer in this context given that we already have Triton generating code for GPUs?</p>
</blockquote>
</aside>
<p>As mentioned before <code>nvFuser</code> is also deprecated in <code>TorchScript</code> and <code>NNC</code> is used as the default backend. Note that <code>TorchScript</code> is in “maintenance mode” and I believe won’t get any major updates or fixes anymore.</p>
<p><code>nvFuser</code> development continues out-of-core in a standalone repository and we are exploring different directions.</p>

username: andreigh
cooked: <p>Is there any torch dynamo backend that generates CUDA code (.cu) and compiles it using nvcc (similar to  how inductor uses g++ to compile fused c++) ? AFAIK, triton generates ptx code directly</p>

username: trusira
cooked: <p>AFAIK, the backend you’re describing doesn’t exist. Can you explain the motivation for having this capability, given we can already generate PTX?</p>

username: andreigh
cooked: <p>Was just curious if this ever was a part of the project. I guess one advantage would have been to use the nvcc compiler optimizations.</p>

username: yiakwy-xpu-ml-framew
cooked: <p>Hi, Ptrblck I think this is not accurate. The PR has been reverted. Also I frequently see the warning in Megatron-LM that the nvfuser is no longer supported in torch script. It does not tell me whether NNC is used when compilation optimization is applied with dynomo.</p>

username: ptrblck
cooked: <p>The PR was initially reverted but relanded in <a href="https://github.com/pytorch/pytorch/commit/c913f3857fb6a8ff8a32174db1da1b4f808f178f"><code>c913f385</code></a> as also shown in the same PR further down.</p>