Where is the source code for MulBackward1

BruceDai003: <p>I’m using pytorch 1.8.1. As I’m running a testcase in test_autograd.py, e.g. <code>addcmul</code>. I see there is a gradgradcheck to check the second order derivatives. I just want to know how the backward is done.<br>
So I used <code>torchviz</code> to generate the backward graph below:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a156b23c6384386fb71eed26e161dfbb8a8f29ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/a156b23c6384386fb71eed26e161dfbb8a8f29ba" title="addcmul"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a156b23c6384386fb71eed26e161dfbb8a8f29ba_2_689x493.png" alt="addcmul" data-base62-sha1="n1gClLOpCZpiSd3cDqFWmUf4DAm" width="689" height="493" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a156b23c6384386fb71eed26e161dfbb8a8f29ba_2_689x493.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a156b23c6384386fb71eed26e161dfbb8a8f29ba_2_1033x739.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a156b23c6384386fb71eed26e161dfbb8a8f29ba.png 2x" data-dominant-color="EDECE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">addcmul</span><span class="informations">1106×791 44.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>(This graph is generated in an pytorch 1.9 environment)<br>
So, I guess these are the called backward functions, right? I want to know how <code>MulBackward1</code> is done in pytorch c/cuda source code. But I greped and only found it here:build/lib.linux-x86_64-3.6/torch/include/torch/csrc/autograd/generated/Functions.h.(And from the directory hierachy, it seems this file is auto generated, maybe from some macros?) So it’s superclass is TraceableFunction and the superclass of that is Node. But I cann’t find anything like how <code>MulBackward1</code> is implemented, like an op. Or am I looking into the wrong direction?</p>

BruceDai003: <p>By using nvprof, I managed to trace it down to this kernel information:</p>
<pre><code class="lang-auto">void at::native::unrolled_elementwise_kernel&lt;at::native::MulScalarFunctor&lt;double, double&gt;, at::detail::Array&lt;char*, 2&gt;, Trivi...(int, at::native::MulScalarFunctor&lt;double, double&gt;, at::detail::Array&lt;char*, 2&gt;, TrivialOffsetCalculator&lt;1, unsigned int&gt;, TrivialOffsetCalculator&lt;1, unsigned int&gt;, at::native::memory::LoadWithCast&lt;1&gt;, at::native::memory::StoreWithCast)
</code></pre>
<p>And another kernel is</p>
<pre><code class="lang-auto">void at::native::unrolled_elementwise_kernel&lt;at::native::MulFunctor&lt;float&gt;, at::detail::Array&lt;char*, 3&gt;, OffsetCalculator&lt;2, ...(int, at::native::MulFunctor&lt;float&gt;, at::detail::Array&lt;char*, 3&gt;, OffsetCalculator&lt;2, unsigned int&gt;, OffsetCalculator&lt;1, unsigned int&gt;, at::native::memory::LoadWithoutCast, at::native::memory::StoreWithoutCast)
</code></pre>
<p>I guess MulFunctor is for same shape multiplication? (MulBackward),<br>
MulScalarFunctor is for an array times a scalar? (MulBackward1).<br>
But I’m also quite curious as how did pytorch auto-generated <code>MulBackward</code> and <code>MulBackward1</code>, etc as in torch/csrc/autograd/generated/Functions.h file.</p>
<p>Edit:<br>
for adcmul op, the torchviz result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3742dec8d69754f1d80145bb6d6f9e3e7774d8a.png" alt="image" data-base62-sha1="uaBESY89QIu12GxIg0GsiHRbFbA" width="665" height="414"><br>
There is a <code>AddcmulBackward</code>, by inspecting into the kernels called using <code>nvprof</code> again, I also see that there is the aforemetioned <code>at::native::MulScalarFunctor</code> like for MulBackward1, but this time the template class is float:</p>
<pre><code class="lang-auto">void at::native::vectorized_elementwise_kernel&lt;4, at::native::MulScalarFunctor&lt;float, float&gt;, at::detail::Array&lt;char*, 2&gt; &gt;(int, at::native::MulScalarFunctor&lt;float, float&gt;, at::detail::Array&lt;char*, 2&gt;)
</code></pre>
<p>So, I’m curious why in the first case, it’s a double, in the second case, it’s a float?</p>
<p>And here is the simple python test code:</p>
<pre><code class="lang-auto">import torch

from torch import nn

from torchviz import make_dot, make_dot_from_trace

# %%

S = 2

device = 'cuda:0'

a = torch.randn(S, S, requires_grad=True, device=device)

b = torch.tensor(0.3, requires_grad=True, device=device)

c = torch.tensor(0.5, requires_grad=True, device=device)

value = 0.5

y = torch.addcmul(a, b, c, value=value)

# %%

# Use a fn to do addcmul/addcdiv, etc

def fn(*inputs):

    output = getattr(inputs[0], 'addcmul')(*inputs[1:], value=value)

    return output

num_outputs = 1

tupled_inputs = (a, b, c)

output = fn(*tupled_inputs)

# torch.allclose(output, a.addcmul(b, c, value=value))

# %%

grad_out = torch.ones_like(output, requires_grad=True)

def new_func(*args):

    input_args = args[:-num_outputs]

    grad_outputs = args[-num_outputs:]

    outputs = fn(*input_args)

    input_args = tuple(x for x in input_args if isinstance(x, torch.Tensor) and x.requires_grad)

    grad_inputs = torch.autograd.grad(outputs, input_args, grad_outputs, create_graph=True)

    return grad_inputs

tupled_inputs = (a, b, c, grad_out)

# %%

grad_inputs = new_func(*tupled_inputs)

# print(grad_inputs)

# Now try gradgrad

g_b = grad_inputs[1]

gradgrad_out = torch.ones_like(g_b, memory_format=torch.legacy_contiguous_format)

gradgrad_input = torch.autograd.grad(g_b, tupled_inputs, gradgrad_out,

                                              retain_graph=True, allow_unused=True)

# print(gradgrad_input[2])                                           

# # %%

# make_dot(output)

# # %%

# grad_inputs = new_func(*tupled_inputs)

# # %%

# grad_inputs[0]

# # %%

# make_dot(new_func(*tupled_inputs)[1], params={'a': a, 'b': b, 'c': c, 'grad_out': tupled_inputs[-1]})

# # %%

# make_dot(new_func(*tupled_inputs)[2], params={'a': a, 'b': b, 'c': c, 'grad_out': tupled_inputs[-1]})

# # %%

# make_dot(new_func(*tupled_inputs), params={'a': a, 'b': b, 'c': c, 'grad_out': tupled_inputs[-1]})

# # %%

# make_dot(new_func(*tupled_inputs), params={'a': a, 'b': b, 'c': c, 'grad_out': tupled_inputs[-1]}, show_attrs=True)

# # %%

# torch.__version__
</code></pre>
<p>With nvprof:</p>
<pre><code class="lang-auto">nvprof --print-gpu-trace python test_addcmul.py
</code></pre>

tom: <aside class="quote no-group" data-username="BruceDai003" data-post="1" data-topic="129634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brucedai003/48/41490_2.png" class="avatar"> BruceDai003:</div>
<blockquote>
<p>But I cann’t find anything like how <code>MulBackward1</code> is implemented, like an op. Or am I looking into the wrong direction?</p>
</blockquote>
</aside>
<p>It’s generated during the build in torch/csrc/autograd/generated. In <code>Functions.cpp</code>:</p>
<pre data-code-wrap="C++"><code class="lang-C++">variable_list MulBackward1::apply(variable_list&amp;&amp; grads) {
  IndexRangeGenerator gen;
  auto self_ix = gen.range(1);
  variable_list grad_inputs(gen.size());
  auto&amp; grad = grads[0];
  bool any_grad_defined = any_variable_defined(grads);
  if (should_compute_output({ self_ix })) {
    auto grad_result = any_grad_defined ? (mul_tensor_backward(grad, at::scalar\
_to_tensor(other), self_scalar_type)) : Tensor();
    copy_range(grad_inputs, self_ix, grad_result);
  }
  return grad_inputs;
}
</code></pre>
<p>and some not so interesting Python wrapping in <code>python_functions.cpp</code></p>
<p>The above is generated by the things in <code>tools/autograd</code> processing <code>derivatives.yaml</code>.</p>

BruceDai003: <aside class="quote no-group" data-username="tom" data-post="3" data-topic="129634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p>build in torch/csrc/autograd/generated. In <code>Functions.cpp</code>:</p>
</blockquote>
</aside>
<p>Thank you very much. But I just checked my build files, I only see header files under torch/csrc/autograd/generated:</p>
<ul>
<li>Functions.h</li>
<li>python_functions.h</li>
<li>variable_factories.h</li>
<li>VariableType.h</li>
</ul>
<p>Is there environment variables which might controll if <code>cpp</code> files will be generated or not?</p>

tom: <p>I never know, but it would seem that PyTorch should generate a number of .cpps as well:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/773c8b6440eee7d903f3d3a80d1fa866370b1ba2/caffe2/CMakeLists.txt#L365">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/773c8b6440eee7d903f3d3a80d1fa866370b1ba2/caffe2/CMakeLists.txt#L365" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/773c8b6440eee7d903f3d3a80d1fa866370b1ba2/caffe2/CMakeLists.txt#L365" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/773c8b6440eee7d903f3d3a80d1fa866370b1ba2/caffe2/CMakeLists.txt#L365</a></h4>



  <pre class="onebox">    <code class="lang-txt">
      <ol class="start lines" start="355" style="counter-reset: li-counter 354 ;">
          <li>
          </li>
<li># Generate header with version info</li>
          <li>configure_file("${TORCH_SRC_DIR}/csrc/api/include/torch/version.h.in"</li>
          <li>  "${TORCH_SRC_DIR}/csrc/api/include/torch/version.h"</li>
          <li>  @ONLY)</li>
          <li>
          </li>
<li>set(GENERATED_CXX_TORCH</li>
          <li>  "${TORCH_SRC_DIR}/csrc/autograd/generated/Functions.cpp"</li>
          <li>  )</li>
          <li>
          </li>
<li class="selected">if(NOT INTERN_DISABLE_AUTOGRAD AND NOT BUILD_LITE_INTERPRETER)</li>
          <li>  list(APPEND GENERATED_CXX_TORCH</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/VariableType_0.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/VariableType_1.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/VariableType_2.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/VariableType_3.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/VariableType_4.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/TraceType_0.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/TraceType_1.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/TraceType_2.cpp"</li>
          <li>    "${TORCH_SRC_DIR}/csrc/autograd/generated/TraceType_3.cpp"</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


BruceDai003: <aside class="quote no-group" data-username="tom" data-post="5" data-topic="129634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tom/48/3162_2.png" class="avatar"> tom:</div>
<blockquote>
<p><code>BUILD_LITE_INTERPRETER</code></p>
</blockquote>
</aside>
<p>Thanks a lot. I was using VSCode to search for this “Functions.cpp” file, the reason why I can’t find it is due to the directory is makred in .gitignore file. Now I commented it out from .gitignore file, I can find all the generated files. Again, thanks.<br>
Besides, by reading the source code, for my case, I just figured out why the second case the templated parameters are all double, but not the first.</p>
<ol>
<li>For the first MulScalar&lt;float, float&gt; call, this happens during addcmul’s backward function. By checking the backward formula in “derivatives.yaml” file for addcmul:</li>
</ol>
<pre><code class="lang-auto">- name: addcmul(Tensor self, Tensor tensor1, Tensor tensor2, *, Scalar value=1) -&gt; Tensor

  self: handle_r_to_c(self.scalar_type(), grad)

  tensor1: handle_r_to_c(tensor1.scalar_type(), grad * (tensor2 * value).conj())

  tensor2: handle_r_to_c(tensor2.scalar_type(), grad * (tensor1 * value).conj())
</code></pre>
<p>It didn’t call another implemented backward function, instead, it writes down the calculation formula using already known operators, here it is mul, specifically, for <code>tensor2 * value</code>, it would be a tensor * Scalar, so it will call <strong>aten/src/ATen/native/BinaryOps.cpp  mul()</strong>:</p>
<pre><code class="lang-auto">Tensor mul(const Tensor&amp; self, Scalar other) {
  return native::mul(self, wrapped_scalar_tensor(other));
}
</code></pre>
<p>Notice that it will call a <code>wrapped_scalar_tensor()</code> API to the Scalar variable first, by checking the source code for this, it will in fact call a <code>set_wrapped_numer()</code> function, which means when calculating the common_types for self(float) and other(double), the other’s type won’t participate in the calculation, so the result is float.<br>
However, if we don’t use <code>wrapped_scalar_tensor()</code> API, the result of common_type would be double. That is the case of what happened when we do a backward for Mul for Scalar. Here in derivatives.yaml, you can find the following:</p>
<pre><code class="lang-auto">- name: mul.Scalar(Tensor self, Scalar other) -&gt; Tensor
  self: mul_tensor_backward(grad, at::scalar_to_tensor(other), self.scalar_type())
</code></pre>
<p>Notice that in this backward function, it just directly called at::scalar_to_tensor(other), without calling <code>tensor.unsafeGetTensorImpl()-&gt;set_wrapped_number(true);</code>. Thus, the common_type of <code>double</code> and <code>float</code> would be double now. But on my platform, I can’t use double on GPU, that’s why it fails for me.</p>

tom: <p>Glad you solved it and thank you for sharing your insights.</p>
<p>Best regards</p>
<p>Thomas</p>