Where are `at::_ops::***::call` implementations?

collinmccarthy: <p>Hi, I’m trying to understand where / how to find the CUDA implementations for various operators. Specifically, the <code>_pad_enum</code> op right now. In my code I call:</p>
<pre data-code-wrap="c++"><code class="lang-plaintext">auto options =
    F::PadFuncOptions({/*left*/ 0, /*right*/ 0, /*top*/ 0, /*bottom*/ pad_prob_m})
        .mode(torch::kConstant)
        .value(0);
F::pad(tensor, options);
</code></pre>
<p>which <a href="https://github.com/pytorch/pytorch/blob/885122b7dc3367c1feaf01410ad625acccf816dc/torch/csrc/api/include/torch/nn/functional/padding.h#L34" rel="noopener nofollow ugc">calls at::_pad_enum()</a>. In my version of libtorch (<code>libtorch-cxx11-abi-shared-with-deps-1.13.1+cu117.zip</code>), that implementation is in <code>libtorch/include/ATen/ops/_pad_enum.h</code> and calls <code>at::_ops::_pad_enum::call(self, pad, mode, value)</code>. If I look at <code>at::_ops::_pad_enum_ops.h</code> I see this struct:</p>
<pre data-code-wrap="c++"><code class="lang-plaintext">struct TORCH_API _pad_enum {
  using schema = at::Tensor (const at::Tensor &amp;, at::IntArrayRef, int64_t, c10::optional&lt;double&gt;);
  using ptr_schema = schema*;
  // See Note [static constexpr char* members for windows NVCC]
  STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(name, "aten::_pad_enum")
  STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(overload_name, "")
  STATIC_CONSTEXPR_STR_INL_EXCEPT_WIN_CUDA(schema_str, "_pad_enum(Tensor self, int[] pad, int mode, float? value=None) -&gt; Tensor")
  static at::Tensor call(const at::Tensor &amp; self, at::IntArrayRef pad, int64_t mode, c10::optional&lt;double&gt; value);
  static at::Tensor redispatch(c10::DispatchKeySet dispatchKeySet, const at::Tensor &amp; self, at::IntArrayRef pad, int64_t mode, c10::optional&lt;double&gt; value);
};
</code></pre>
<p>So my questions are:</p>
<ol>
<li>Where is the underlying implementation here so I can look at the CUDA code?</li>
<li>If I set a breakpoint there with cuda-gdb, and build from source with Debug build type, will it build with nvcc -g -G and will I be able to hit my breakpoint? (I assume I can’t step into from, e.g. <code>torch::nn::functional</code> because of this dispatch mechanism)</li>
<li>Is there any documentation describing how things like these include files are generated and how these calls are dispatched so I can understand this all better and be able to find any CUDA code I want?</li>
</ol>
<p>Thanks,<br>
-Collin</p>

eqy: <p>Ah, I believe that’s a consequence of there being some codegen that happens between ATen ops and their underlying implementations, which could create some not-intended-for-human-consumption source files like <code>_pad_enum.h</code> to reduce code duplication at the expense of readability. I checked the git blame for that file which reveals this PR to be the last one to do major changes:</p><aside class="onebox githubcommit" data-onebox-src="https://github.com/pytorch/pytorch/commit/7f051b4d2beea6a2c4e7581f53f5c9deddbbe4cb#diff-9d658d0f73248419934f3750fc742cbafa29b4484870857a52322d6d74dbe36c">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/commit/7f051b4d2beea6a2c4e7581f53f5c9deddbbe4cb#diff-9d658d0f73248419934f3750fc742cbafa29b4484870857a52322d6d74dbe36c" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/7f051b4d2beea6a2c4e7581f53f5c9deddbbe4cb" target="_blank" rel="noopener nofollow ugc">Implement F.pad in ATen</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2022-04-01" data-time="01:10:12" data-timezone="UTC">01:10AM - 01 Apr 22 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/peterbell10" target="_blank" rel="noopener nofollow ugc">
          <img alt="peterbell10" src="https://avatars.githubusercontent.com/u/13238737?v=4" class="onebox-avatar-inline" width="20" height="20">
          peterbell10
        </a>
      </div>

      <div class="lines" title="changed 5 files with 105 additions and 42 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/7f051b4d2beea6a2c4e7581f53f5c9deddbbe4cb" target="_blank" rel="noopener nofollow ugc">
          <span class="added">+105</span>
          <span class="removed">-42</span>
        </a>
      </div>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">This moves the C++ torch pad function into ATen proper. Once the
forward-compati<span class="show-more-container"><a href="https://github.com/pytorch/pytorch/commit/7f051b4d2beea6a2c4e7581f53f5c9deddbbe4cb" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">bility period is over, the python interface can use
this directly.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/73431

Approved by: https://github.com/ezyang</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>From there you can infer the underlying names are probably things like replicationpad and reflectionpad, which are probably implemented in the <code>ATen/native/cuda/</code> directory e.g., for GPU:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReflectionPad.cu">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReflectionPad.cu" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReflectionPad.cu" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReflectionPad.cu</a></h4>


      <pre><code class="lang-cu">#define TORCH_ASSERT_ONLY_METHOD_OPERATORS
#include &lt;ATen/core/Tensor.h&gt;
#include &lt;ATen/ceil_div.h&gt;
#include &lt;ATen/Dispatch.h&gt;
#include &lt;ATen/cuda/Atomic.cuh&gt;
#include &lt;ATen/cuda/detail/IndexUtils.cuh&gt;
#include &lt;ATen/cuda/CUDAContext.h&gt;
#include &lt;ATen/TensorUtils.h&gt;
#include &lt;ATen/Utils.h&gt;

#ifndef AT_PER_OPERATOR_HEADERS
#include &lt;ATen/Functions.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#else
#include &lt;ATen/ops/empty.h&gt;
#include &lt;ATen/ops/zeros_like.h&gt;
#include &lt;ATen/ops/reflection_pad1d_native.h&gt;
#include &lt;ATen/ops/reflection_pad2d_native.h&gt;
#include &lt;ATen/ops/reflection_pad3d_native.h&gt;
#include &lt;ATen/ops/reflection_pad1d_backward_native.h&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReflectionPad.cu" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReplicationPadding.cu">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReplicationPadding.cu" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReplicationPadding.cu" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReplicationPadding.cu</a></h4>


      <pre><code class="lang-cu">#define TORCH_ASSERT_ONLY_METHOD_OPERATORS
#include &lt;ATen/core/Tensor.h&gt;
#include &lt;ATen/ceil_div.h&gt;
#include &lt;ATen/Dispatch.h&gt;
#include &lt;ATen/cuda/Atomic.cuh&gt;
#include &lt;ATen/cuda/detail/IndexUtils.cuh&gt;
#include &lt;ATen/cuda/CUDAContext.h&gt;
#include &lt;ATen/TensorUtils.h&gt;
#include &lt;ATen/Utils.h&gt;
#include &lt;c10/util/Exception.h&gt;

#ifndef AT_PER_OPERATOR_HEADERS
#include &lt;ATen/Functions.h&gt;
#include &lt;ATen/NativeFunctions.h&gt;
#else
#include &lt;ATen/ops/empty_like.h&gt;
#include &lt;ATen/ops/replication_pad1d_native.h&gt;
#include &lt;ATen/ops/replication_pad1d_backward_native.h&gt;
#include &lt;ATen/ops/replication_pad2d_native.h&gt;
#include &lt;ATen/ops/replication_pad2d_backward_native.h&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/8f1c3c68d3aba5c8898bfb3144988aab6776d549/aten/src/ATen/native/cuda/ReplicationPadding.cu" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>In general understanding the dispatching mechanism entirely is a bit daunting (for me), so I typically just check <code>git blame</code> which would hopefully reveal when corresponding files have moved around in PRs or e.g., use a profiler like <code>nsys nvprof</code> to reveal the names of the kernels being called.</p>

Realtyxxx: <p>Hi，I’m also want to see it’s dispatcher mechanism;</p>
<p>I wonder this is generated by gen.py ??</p>