How the autograd is implmented in pytorch

moha657: <p>Hi,</p>
<p>I am currently trying to understand the PyTorch codebase. For now, the implementations of the Linear layer, for example, are described by these two files in GitHub repos, but I can’t understand how the operations are stored for the computational graph.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/linear.py#L50">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/linear.py#L50" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/linear.py#L50" target="_blank" rel="noopener nofollow ugc">torch/nn/modules/linear.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/linear.py#L50" rel="noopener nofollow ugc"><code>v2.6.0</code></a>
</div>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="40" style="counter-reset: li-counter 39 ;">
          <li></li>
          <li>    """</li>
          <li></li>
          <li>    def __init__(self, *args: Any, **kwargs: Any) -&gt; None:</li>
          <li>        super().__init__()</li>
          <li></li>
          <li>    def forward(self, input: Tensor) -&gt; Tensor:</li>
          <li>        return input</li>
          <li></li>
          <li></li>
          <li class="selected">class Linear(Module):</li>
          <li>    r"""Applies an affine linear transformation to the incoming data: :math:`y = xA^T + b`.</li>
          <li></li>
          <li>    This module supports :ref:`TensorFloat32&lt;tf32_on_ampere&gt;`.</li>
          <li></li>
          <li>    On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision&lt;fp16_on_mi200&gt;` for backward.</li>
          <li></li>
          <li>    Args:</li>
          <li>        in_features: size of each input sample</li>
          <li>        out_features: size of each output sample</li>
          <li>        bias: If set to ``False``, the layer will not learn an additive bias.</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/nn/modules/linear.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/nn/modules/linear.cpp" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/nn/modules/linear.cpp" target="_blank" rel="noopener nofollow ugc">torch/csrc/api/src/nn/modules/linear.cpp</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/nn/modules/linear.cpp" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-cpp">#include &lt;torch/nn/functional/linear.h&gt;
#include &lt;torch/nn/init.h&gt;
#include &lt;torch/nn/modules/linear.h&gt;

#include &lt;torch/types.h&gt;
#include &lt;torch/utils.h&gt;

#include &lt;cmath&gt;
#include &lt;cstdint&gt;

namespace F = torch::nn::functional;

namespace torch::nn {

void IdentityImpl::reset() {}

void IdentityImpl::pretty_print(std::ostream&amp; stream) const {
  stream &lt;&lt; "torch::nn::Identity()";
}

</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/api/src/nn/modules/linear.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


soulitzer: <p>You might be interested in<br>
<a href="https://blog.ezyang.com/2019/05/pytorch-internals/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://blog.ezyang.com/2019/05/pytorch-internals/</a><br>
<a href="http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/" class="onebox" target="_blank" rel="noopener nofollow ugc">http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/</a></p>

soulitzer: <p>Short answer is that there will be some additional logic that interposes to track the operations everytime you run an operation.</p>

moha657: <aside class="quote no-group quote-modified" data-username="moha657" data-post="1" data-topic="219022">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/moha657/48/76599_2.png" class="avatar"> moha657:</div>
<blockquote>
<p>I am currently trying to understand the PyTorch codebase. For now, the implementations of the Linear layer, for example, are described by these two files in GitHub repos, but I can’t understand how the operations are stored for the computational graph.</p>
<p><a href="https://github.com/pytorch/pytorch/blob/v2.6.0/torch/nn/modules/linear.py#L50" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/nn/modules/linear.py at v2.6.0 · pytorch/pytorch · GitHub</a></p>
</blockquote>
</aside>
<p>Thanks for the response the articles are very interesting in fact what i am interested in is knowing at a deep level the in and out of pytorch as i wan to go from a intermediate level into upper intermediate in both pytorch and deep learning</p>