How to reach the exact source code in Pytorch?

Poornima_Jain: <p>I wish to implement fourier domain based convolution, for which I wished to compare the complexity of my approach with the inbuilt conv2d layer’s complexity.<br>
In the docs, the following formula is given for computing convolutional layer -<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d</a></p>
<p>To get to the source code, I followed the ‘source’ button on the right of the formula which lead to the following page -<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv2d</a></p>
<p>This code used ‘functional.conv2d’, but the definition on the functional conv2d page did not have a link to the source code.</p>
<p>Some answers online suggested this is the entry point to the code for conv2d -<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/c780610f2d8358297cb4e4460692d496e124d64d/aten/src/ATen/native/Convolution.cpp#L481</a></h4>
<pre class="onebox"><code class="lang-cpp"><ol class="start lines" start="471" style="counter-reset: li-counter 470 ;">
<li>}</li>
<li>
</li>
<li>
</li>
<li>at::Tensor conv1d(</li>
<li>    const Tensor&amp; input, const Tensor&amp; weight, const Tensor&amp; bias,</li>
<li>    IntArrayRef stride, IntArrayRef padding, IntArrayRef dilation, int64_t groups) {</li>
<li>  return at::convolution(input, weight, bias, stride, padding, dilation,</li>
<li>                         false, {0}, groups);</li>
<li>}</li>
<li>
</li>
<li class="selected">at::Tensor conv2d(</li>
<li>    const Tensor&amp; input, const Tensor&amp; weight, const Tensor&amp; bias,</li>
<li>    IntArrayRef stride, IntArrayRef padding, IntArrayRef dilation, int64_t groups) {</li>
<li>  return at::convolution(input, weight, bias, stride, padding, dilation,</li>
<li>                         false, {{0, 0}}, groups);</li>
<li>}</li>
<li>
</li>
<li>at::Tensor conv3d(</li>
<li>    const Tensor&amp; input, const Tensor&amp; weight, const Tensor&amp; bias,</li>
<li>    IntArrayRef stride, IntArrayRef padding, IntArrayRef dilation, int64_t groups) {</li>
<li>  return at::convolution(input, weight, bias, stride, padding, dilation,</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>But this link further points to mkldnn and cudnn implementations, which maybe implemented in the following files -<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mkldnn/Conv.cpp" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mkldnn/Conv.cpp" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/aten/src/ATen/native/mkldnn/Conv.cpp</a></h4>
<pre><code class="lang-cpp">#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;ATen/Config.h&gt;

#if !AT_MKLDNN_ENABLED()

namespace at { namespace native {

at::Tensor mkldnn_convolution(
    const at::Tensor&amp; input, const at::Tensor&amp; weight, const at::Tensor&amp; bias,
    IntArrayRef padding, IntArrayRef stride, IntArrayRef dilation, int64_t groups) {
  AT_ERROR("mkldnn_convolution_forward: ATen not compiled with MKLDNN support");
}

at::Tensor mkldnn_convolution_backward_input(
    IntArrayRef input_size, const at::Tensor&amp; grad_output, const at::Tensor&amp; weight,
    IntArrayRef padding, IntArrayRef stride, IntArrayRef dilation, int64_t groups, bool bias_defined) {
  AT_ERROR("mkldnn_convolution_backward_input: ATen not compiled with MKLDNN support");
}

</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/mkldnn/Conv.cpp" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
and<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp</a></h4>
<pre><code class="lang-cpp">#include &lt;limits&gt;
#include &lt;ATen/ATen.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#include &lt;ATen/Config.h&gt;
#include &lt;ATen/cuda/CUDAConfig.h&gt;
#include &lt;ATen/cuda/Exceptions.h&gt;
#include &lt;ATen/native/ConvUtils.h&gt;

#if !AT_CUDNN_ENABLED()

namespace at { namespace native {

// See Note [ATen preprocessor philosophy]

at::Tensor cudnn_convolution(
    const at::Tensor&amp; input, const at::Tensor&amp; weight,
    IntArrayRef padding, IntArrayRef stride, IntArrayRef dilation,
    int64_t groups, bool benchmark, bool deterministic) {
  AT_ERROR("cudnn_convolution: ATen not compiled with cuDNN support");
}
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cudnn/Conv.cpp" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I tried to probe the mkldnn file further but it points to some algorithm called -<br>
‘ideep::algorithm::convolution_direct’, which seems to be defined in some Intel github account.</p>
<p>So I could not get to any code in which we can see the implementation of the convolution function directly - as in in terms of plain multiplication and addition.</p>
<p>Could anyone kindly suggest can/ how can we get to the base code in Pytorch?<br>
Or can we assume that the implementation directly uses the given formula in torch.nn.conv2d to compute convolution?<br>
Thanks in advance.</p>

tom: <p>Like most open-source libraries that PyTorch uses, this is “vendored” as a git submodule (cloning with <code>--recursive</code> gets you these, too, or use <code>git submodule update --init --recursive</code> and <code>git submodule sync</code> or some combination). Then the code you are most likely interested in is under <code>third_party/ideep/mkl-dnn/src/cpu/</code> in your PyTorch checkout.</p>
<p>Best regards</p>
<p>Thomas</p>

BruceDai003: <aside class="quote no-group" data-username="Poornima_Jain" data-post="1" data-topic="68404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/poornima_jain/48/14833_2.png" class="avatar"> Poornima_Jain:</div>
<blockquote>
<p>mplement fourier dom</p>
</blockquote>
</aside>
<p>I might should open up a new post, but I just want to ask if you can help me to find where is the cudnn version of this at::cudnn_convolution, I searched the whole project but found no definition. I know it will eventually call into cudnn library’s APIs, but I just need to find the missing link part. Guess it might be some code auto generated?<br>
Here is where it gets called:<br>
<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp#L955" rel="noopener nofollow ugc">cudnn_convolution</a></p>
<pre><code class="lang-auto">    } else {
      output = at::cudnn_convolution(
          input.contiguous(backend_memory_format), weight,
          params.padding, params.stride, params.dilation, params.groups, params.benchmark, params.deterministic, params.allow_tf32);
      if (bias.defined()) {
        output.add_(reshape_bias(input.dim(), bias));
      }
    }
</code></pre>
<p>Thank you very much.</p>

ptrblck: <p>The cuDNN calls for the currently used v7 API can be found <a href="https://github.com/pytorch/pytorch/blob/a5b57c9433eb25947292b49458ce73783f51b7c2/aten/src/ATen/native/cudnn/Conv_v7.cpp">here</a>.</p>

BruceDai003: <p>Thanks a lot. Now I can pull them all together.</p>

rzjhdi: <aside class="quote no-group" data-username="Poornima_Jain" data-post="1" data-topic="68404">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/poornima_jain/48/14833_2.png" class="avatar"> Poornima_Jain:</div>
<blockquote>
<p>the implementation of the convolution function directly - as in in terms of plain multiplication and addition.</p>
</blockquote>
</aside>
<p>Did you manage to find the implementation of the convolution function in in terms of plain multiplication and addition, finally?</p>
<p>I am still struggling to find it. Thanks.</p>