CUDA `add_kernel`

Jerome_Ku: <p>Can anyone point me to where the <code>add_kernel</code> is implemented specifically for <code>CUDA</code>?  Looking at CUDA traces (through <code>nsys</code>), I can see the dispatcher ultimately calls out to this kernel which has a <code>gpu_kernel_impl</code>.</p>
<pre><code class="lang-auto">libcuda.so.535.113.01!0x7f2c80680076
libcudart.so.12!cudaLaunchKernel
libtorch_cuda.so!void at::native::gpu_kernel_impl&lt;...&gt;(...)
libtorch_cuda.so!void at::native::gpu_kernel&lt;...&gt;(...)
libtorch_cuda.so!at::native::add_kernel(...)::{lambda()#1}::operator()() const
libtorch_cuda.so!at::native::add_kernel(...)
libtorch_cuda.so!at::(...)::wrapper_CUDA_add__Tensor(...)
libtorch_cpu.so!at::_ops::add__Tensor::call(...)
libtorch_cpu.so!at::native::_convolution(...)
libtorch_cpu.so!at::(...)::(...)::wrapper_CompositeExplicitAutograd___convolution(...)
libtorch_cpu.so!c10::impl::wrap_kernel_functor_unboxed_&lt;...&gt;::call(...)
libtorch_cpu.so!at::_ops::_convolution::call(...)
libtorch_cpu.so!at::native::convolution(...)
libtorch_cpu.so!at::(...)::(...)::wrapper_CompositeExplicitAutograd__convolution(...)
libtorch_cpu.so!c10::impl::wrap_kernel_functor_unboxed_&lt;...&gt;::call(...)
libtorch_cpu.so!at::_ops::convolution::redispatch(...)
libtorch_cpu.so!torch::autograd::VariableType::(...)::convolution(...)
libtorch_cpu.so!c10::impl::wrap_kernel_functor_unboxed_&lt;...&gt;::call(...)
libtorch_cpu.so!at::_ops::convolution::call(...)
libtorch_cpu.so!at::native::conv2d_symint(...)
libtorch_cpu.so!c10::impl::wrap_kernel_functor_unboxed_&lt;...&gt;::call(...)
libtorch_cpu.so!at::_ops::conv2d::call(...)
libtorch_python.so!torch::autograd::THPVariable_conv2d(...)
</code></pre>
<p>In past versions of <code>PyTorch</code> the <code>add</code> op was implemented in a file <a href="https://github.com/pytorch/pytorch/blob/e1bd4963e23835e307792eb3e49cb5f1b8f771c5/aten/src/ATen/native/cuda/BinaryAddSubKernel.cu" rel="noopener nofollow ugc"><code>BinaryAddSubKernel.cu</code></a> in <code>ATen/native/cuda</code> and registered under <a href="https://github.com/pytorch/pytorch/blob/e1bd4963e23835e307792eb3e49cb5f1b8f771c5/aten/src/ATen/native/cuda/BinaryAddSubKernel.cu#L66" rel="noopener nofollow ugc"><code>add_kernel_cuda</code></a>.  However, this file no longer exists on more recent versions of pytorch (&gt;= 2.1).</p>
<p>The <code>native_functions.yaml</code> entry for <code>add</code> is:</p>
<pre><code class="lang-auto">- func: add.Tensor(Tensor self, Tensor other, *, Scalar alpha=1) -&gt; Tensor
  device_check: NoCheck   # TensorIterator
  structured_delegate: add.out
  variants: function, method
  dispatch:
    SparseCPU, SparseCUDA: add_sparse
    SparseCsrCPU, SparseCsrCUDA: add_sparse_csr
    MkldnnCPU: mkldnn_add
    ZeroTensor: add_zerotensor
    NestedTensorCPU, NestedTensorCUDA: NestedTensor_add_Tensor
  tags: [core, pointwise]
</code></pre>
<p>Any thoughts on the refactoring / re-implementation rationale would be greatly appreciated!</p>

ptrblck: <p>I would assume the <code>TensorIterator</code> is used from <a href="https://github.com/pytorch/pytorch/blob/919c9b713e65aa1489123259a0ce7e79322d4d9d/aten/src/ATen/native/cuda/Loops.cuh#L108-L129">here</a>.</p>