Structure of Pytorch

username: 0xFFFFFFFF
cooked: <p>I’ve made some pull requests to Pytorch’s C++ / CUDA functions, but I was never able to fully comprehend the general structure of Pytorch to the details. I’ve dug into the directory many times, but after a attempts, I thought the best would be to hear from the hoarse’s mouth.</p>
<p>I want to clairfy the general function call stack, from when the end user uses <code>nn.Conv2d</code> all the way down to when <code>cudnnConvolutionForward</code> is called.</p>
<p><code> nn.Conv2d -&gt; ??? -&gt; cudnnConvolutionForward</code></p>
<p>Here is what I’ve figured out so far:</p>
<ul>
<li>
<code>nn.Conv2d</code> (defined in <code>torch/nn/modules/conv.py</code>) is just a thin wrapper around <code>F.conv2d</code>
</li>
<li>
<code>F.conv2d</code> (defined in <code>torch.nn.functional.py</code>) just calls <code>_add_docstr(torch.conv2d, "really long doc string...")</code> to add docstring.</li>
<li>
<code>_add_docstr</code> (defined <code>torch/csrc/Module.cpp</code>) function is a C++ function. Looking at the actual definition of the function <code>THPModule_addDocStr</code> (defined in the same file), all it does is check if <code>ml_doc</code> field is occupied, and if not it adds the <code>"really long doc..."</code> to the field. So <code>_add_docstr</code> doesn’t define the function. As the name suggests, it only adds docstring to the pre-defined function.
<ul>
<li>Pytorch seems to use the standard way to  extends the CPython’s interpreter described <a href="https://docs.python.org/3/extending/extending.html#a-simple-example" rel="nofollow noopener">here</a>.</li>
</ul>
</li>
<li>This makes me believe that <code>torch.conv2d</code> is another C++ function that is defined somewhere. <strong>Here where I’m lost</strong>
</li>
<li>The only fucnction that calls <code>cudnnConvolutionForward</code> is <code>raw_cudnn_convolution_forward_out</code>  (defined <code>pytorch/aten/src/ATen/native/cudnn/Conv.cpp</code>). So I assume calling <code>nnConv2d</code> eventually calls <code>raw_cudnn_convolution_forward_out</code>. So I climb up the call stack starting from <code>raw_cudnn_convolution_forward_out</code>.</li>
<li>After climbing up a few functions, we reach <code>conv2d</code> (defined in <code>aten/src/ATen/native/Convolution.cpp</code>). <code>conv2d</code> is called in many functions, but I couldn’t find a function that directly adds <code>conv2d</code> to CPython’s interpreter.</li>
</ul>
<p>To summarize, from top down, I have:</p>
<p><code>nn.Conv2d -&gt; F.conv2d -&gt; torch.conv2d -&gt; ???</code></p>
<p>And from bottom up I have:</p>
<p><code>??? -&gt; conv2d -&gt; convolution -&gt; _convolution-&gt; cudnn_convolution-&gt; cudnn_convolution_forward-&gt; raw_cudnn_convolution_forward_out-&gt; cudnnConvolutionForward</code></p>
<p>My assumption is <code>torch.conv2d</code> is <code>conv2d</code> defined in <code>aten/src/ATen/native/Convolution.cpp</code>.  But this is a specualtion. Please help me out how the two branches connect. Thanks!</p>

username: tom
cooked: <p>I have an <a href="https://lernapparat.de/selective-excursion-into-pytorch-internals/" rel="nofollow noopener">old blog post</a> describing how this works from a “where does my PyTorch function come from?” perspective. The key is the native_functions.yaml. From that the dispatching works to Convolution.cpp via autogenerated wrappers (torch/autograd/generated for the Python bindings, and also to go from at::… to at::native::…).</p>
<p>Best regards</p>
<p>Thomas</p>

username: 0xFFFFFFFF
cooked: <p>Your blog post was extremely helpful. Thanks</p>