Change the tanh backward code

username: hohohihi
cooked: <p>Hi, there.</p>
<p>I want to change the backward behavior of tanh.<br>
So, i have to touch the source of torch.<br>
But i don’t know where my downloaded torch code exist.<br>
So,</p>
<ol>
<li>where is my torch source code exist in my computer? (I am using anaconda)</li>
<li>Where is the directory of tanh’s backward function?</li>
<li>do i have to do something after changing the source code? (i.e. recompile? re-source?)</li>
</ol>
<p>here is my bashrc file.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e044ed0d558cec159a14f6c6ec2308190e77c4b.png" data-download-href="https://discuss.pytorch.org/uploads/default/4e044ed0d558cec159a14f6c6ec2308190e77c4b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e044ed0d558cec159a14f6c6ec2308190e77c4b_2_690x259.png" alt="image" data-base62-sha1="b8athpMCNdcHlIf3HpboRxE59nt" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e044ed0d558cec159a14f6c6ec2308190e77c4b_2_690x259.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e044ed0d558cec159a14f6c6ec2308190e77c4b_2_1035x388.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e044ed0d558cec159a14f6c6ec2308190e77c4b_2_1380x518.png 2x" data-dominant-color="393B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1556×585 76.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>You won’t be able to change the pre-built binaries so would need to build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.<br>
The easier approach to change the backward method of a specific operation would be to implement a custom <code>autograd.Function</code> and write the <code>backward</code> method manually as described e.g. <a href="https://pytorch.org/tutorials/beginner/examples_autograd/polynomial_custom_function.html">here</a>.</p>

username: hohohihi
cooked: <p>Oh, thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
Let me try that way.</p>
<p>But, before doing it, can you specify where the original tanh’s backwar function is?<br>
I could find the forward function of it, but i cannot find the backward of tanh’s.</p>
<p>I want to check the exact math expression of tanh’s backward function.<br>
How backward of tanh is coded. So, I want where is this directory is.</p>
<p>Thx.</p>

username: ptrblck
cooked: <p>The CPU kernel is defined <a href="https://github.com/pytorch/pytorch/blob/c5872e6d6d8fd9b8439b914c143d49488335f573/aten/src/ATen/native/cpu/BinaryOpsKernel.cpp#L793-L836">here</a> and the CUDA kernel <a href="https://github.com/pytorch/pytorch/blob/a0a2b755659a0fd0e12d209ac84c402538a75e1b/aten/src/ATen/native/cuda/BinaryMiscBackwardOpsKernels.cu#L86-L100">here</a>.</p>

username: hohohihi
cooked: <p>Thank you, <a class="mention" href="/u/ptrblck">@ptrblck</a> .<br>
now, I know  what i have to do.</p>

username: hohohihi
cooked: <p>Hi, again. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks for the kernel code. But I want more. in the referenced page,</p>
<pre><code class="lang-auto">void tanh_backward_kernel_cuda(TensorIteratorBase&amp; iter) {
  if(isComplexType(iter.dtype())) {
    AT_DISPATCH_COMPLEX_TYPES(iter.dtype(), "tanh_backward_complex_cuda", [&amp;]() {
      gpu_kernel(iter, [] GPU_LAMBDA(scalar_t a, scalar_t b) -&gt; scalar_t {
        return a * std::conj(scalar_t{1.} - b * b);
      });
    });
  } else {
    AT_DISPATCH_FLOATING_TYPES_AND2(at::ScalarType::Half, at::ScalarType::BFloat16, iter.dtype(), "tanh_backward_cuda", [&amp;]() {
      gpu_kernel(iter, [] GPU_LAMBDA(scalar_t a, scalar_t b) -&gt; scalar_t {
        return a * (scalar_t{1.} - b * b);
      });
    });
  }
}
</code></pre>
<p>In these code, I want to find the where the ‘b’ came from.<br>
So, can you give me the “upper code” of that cuda kernel?</p>

username: ptrblck
cooked: <p><code>a</code> and <code>b</code> would be the inputs to <code>tanh_backward</code>, which should be <code>grad_output</code> and <code>output</code>.</p>

username: hohohihi
cooked: <p>Yes, <a class="mention" href="/u/ptrblck">@ptrblck</a>  i also presume that <code>a</code> and <code>b</code> are <code>grad_output</code> and <code>output</code> from these kind of mathematical sources (<a href="https://imaddabbura.github.io/post/coding-nn-fwd-bckwd-prop/" class="inline-onebox" rel="noopener nofollow ugc">Coding Neural Network - Forward Propagation and Backpropagtion | Imad Dabbura</a>).<br>
But i want to know how these cuda function are connected to python code(“upper code”) just to study this.</p>
<p>And i have to know exactly what <code>output</code> is.<br>
For example, (tanh function) <code>y</code> = tanh(x), <code>output</code> should be <code>y</code>, right?</p>
<p>Then, at the forward path, it must be something like ‘ctx.save_for_backward(output)’.<br>
So, i presume those kind of things. But i want to see it.</p>
<p>I searched myself through all the directories of pytorch github. But I could not understand the structure of it. So, i lost myself and don’t know where to go.</p>
<p>Thank you.</p>