Learning PyTorch with Examples, Defining New Autograd Functions (1.7.1)

olszewskip: <p>Hi!<br>
I’ve just only started with Pytorch. I’ve got a question about the “Defining New Autograd Function” section of one of the first tutorials in 1.7.1 docs:</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-defining-new-autograd-functions" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-defining-new-autograd-functions" target="_blank" rel="noopener nofollow ugc">Learning PyTorch with Examples — PyTorch Tutorials 1.7.1 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
This section explains how to define custom forward and backward methods, which is nice, but the presented definition doesn’t seem to connect with the training in the same example: Do I understand correctly that it doesn’t matter for the training loop in this section that the <code>P3</code> function is defined  as <code>LegendrePolynomial3.apply</code> rather than simply as</p>
<pre><code class="lang-auto">def P3(x):
    return 0.5 * (5 * x ** 3 - 3 * x)
</code></pre>
<p>?<br>
If so, what is the advantage here of the approach with inheriting from <code>torch.autograd.Function</code>?</p>

InnovArul: <aside class="quote no-group" data-username="olszewskip" data-post="1" data-topic="107009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/f1d935/48.png" class="avatar"> olszewskip:</div>
<blockquote>
<p>Do I understand correctly that it doesn’t matter for the training loop in this section that the <code>P3</code> function is defined as <code>LegendrePolynomial3.apply</code> rather than simply as</p>
<pre><code class="lang-auto">def P3(x):
    return 0.5 * (5 * x ** 3 - 3 * x)
</code></pre>
<p>?</p>
</blockquote>
</aside>
<p>Yes, you are right, that in this example, it doesn’t matter for the training loop if the function is defined as  <code>LegendrePolynomial3.apply</code> or just a subroutine <code>P3</code>. This is a simple example to show how to define a custom instantiation of <code>autograd.Function</code>. Note that, the elementary <code>operators</code> involved in this subroutine are <code>*, **, -</code>. Under the hood, Pytorch has already implemented with <code>backward</code> (or) gradient calculators for these operations.</p>
<aside class="quote no-group" data-username="olszewskip" data-post="1" data-topic="107009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/f1d935/48.png" class="avatar"> olszewskip:</div>
<blockquote>
<p>what is the advantage here of the approach with inheriting from <code>torch.autograd.Function</code>?</p>
</blockquote>
</aside>
<p>In certain scenarios, when there is a need for some complicated custom operations to be performed (that is not defined in pytorch), typically we write those operations (both <code>forward</code> and <code>backward</code>) as a Cuda C++ op and connect python &amp; Cuda C++ via <code>autograd.Function</code>.</p>