What is PyTorch's Backwards function for Tanh

VolcanoJ: <p>Aloha,<br>
I’m trying to explore alternatives to the Tanh backwards function and I started by setting up a baseline for the experiment by overwriting the Backwards function with 1 −  tanh^2(x)</p>
<p>However, I did not get the same results as when I used the autograd version of tanh’s derivative.  Can anyone shed some light on this?</p>
<p>Mahalo,<br>
Jonathan</p>

KFrank: <p>Aloha Jonathan!</p>
<aside class="quote no-group" data-username="VolcanoJ" data-post="1" data-topic="119389" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/volcanoj/48/29898_2.png" class="avatar"> VolcanoJ:</div>
<blockquote>
<p>overwriting the Backwards function with 1 −  tanh^2(x)</p>
<p>However, I did not get the same results as when I used the autograd version of tanh’s derivative.</p>
</blockquote>
</aside>
<p>It works for me:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.7.1'
&gt;&gt;&gt; t = torch.arange (1.0, 4.0)
&gt;&gt;&gt; t.requires_grad = True
&gt;&gt;&gt; t_tanh = torch.tanh (t)
&gt;&gt;&gt; t_tanh.sum().backward()
&gt;&gt;&gt; t
tensor([1., 2., 3.], requires_grad=True)
&gt;&gt;&gt; t_tanh
tensor([0.7616, 0.9640, 0.9951], grad_fn=&lt;TanhBackward&gt;)
&gt;&gt;&gt; t.grad
tensor([0.4200, 0.0707, 0.0099])
&gt;&gt;&gt; 1.0 - t_tanh**2
tensor([0.4200, 0.0707, 0.0099], grad_fn=&lt;RsubBackward1&gt;)
&gt;&gt;&gt; t.grad - (1.0 - t_tanh**2)
tensor([0., 0., 0.], grad_fn=&lt;SubBackward0&gt;)
</code></pre>
<p>Could you tell us how you are invoking autograd?  (Also let us know<br>
what version of pytorch you are using.)</p>
<p>Best.</p>
<p>K. Frank</p>

VolcanoJ: <p>Thanks for getting back to me so quickly.  Wow, looks like im on version 0.4.1 so i’ll upgrade shortly.  This is my implementation, perhaps I misunderstood the examples?</p>
<pre><code class="lang-auto">
class TanhControl(T.autograd.Function):

    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return calcForward(input)

    @staticmethod
    def calcForward(input):
        return T.tanh(input)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.clone()
            grad_input = calcBackward(input)
        return grad_input

    @staticmethod
    def calcBackward(input):
           return 1 - pow(T.tanh(input),2)
</code></pre>

KFrank: <p>Aloha Jonathan!</p>
<aside class="quote no-group quote-modified" data-username="VolcanoJ" data-post="3" data-topic="119389" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/volcanoj/48/29898_2.png" class="avatar"> VolcanoJ:</div>
<blockquote>
<pre><code class="lang-auto">class TanhControl(T.autograd.Function):
...
    @staticmethod
    def backward(ctx, grad_output):
    ...
            grad_input = grad_output.clone()
            grad_input = calcBackward(input)
        return grad_input
</code></pre>
</blockquote>
</aside>
<p>Your problem seems to be here.</p>
<p>This line, <code>grad_input = grad_output.clone()</code>, looks like some<br>
leftover code from some other autograd function that isn’t relevant<br>
here.</p>
<p>The next line, <code>grad_input = calcBackward(input)</code>, doesn’t use<br>
<code>grad_output</code> which is needed to implement the chain rule, if you will.</p>
<p>Because <code>TanhControl</code> is a scalar function that just gets applied<br>
element-wise to a tensor, you just need to multiply <code>grad_output</code><br>
by the derivative of <code>tanh()</code>, element-wise:<br>
<code>grad_input = calcBackward(input) * grad_output</code></p>
<p>Here is a script that compares pytorch’s <code>tanh()</code> with a tweaked<br>
version of your <code>TanhControl</code> and a version that uses<br>
<code>ctx.save_for_backward()</code> to gain (modest) efficiency by saving<br>
<code>tanh (input)</code> (rather than just <code>input</code>) so that it doesn’t have to<br>
recomputed it during <code>backward()</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

_ = torch.manual_seed (2021)

class TanhControl (torch.autograd.Function):
    @staticmethod
    def forward (ctx, input):
        ctx.save_for_backward (input)
        return torch.tanh (input)
    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        return (1.0 - torch.tanh (input)**2.0) * grad_output

class TanhControlB (torch.autograd.Function):
    @staticmethod
    def forward (ctx, input):
        th = torch.tanh (input)
        ctx.save_for_backward (th)
        return th
    @staticmethod
    def backward(ctx, grad_output):
        th, = ctx.saved_tensors
        return (1.0 - th**2.0) * grad_output

t1 = torch.randn (2, 3)
print ('t1 = ...')
print (t1)
t2 = t1.clone()
t3 = t1.clone()
t1.requires_grad = True
t2.requires_grad = True
t3.requires_grad = True

l1 = (torch.tanh (t1)**2).sum()
l2 = (TanhControl.apply (t2)**2).sum()
l3 = (TanhControlB.apply (t3)**2).sum()
print ('l1 =', l1)
print ('l2 =', l2)
print ('l3 =', l3)

l1.backward()
l2.backward()
l3.backward()
print ('t1.grad = ...')
print (t1.grad)
print ('t2.grad - t1.grad = ...')
print (t2.grad - t1.grad)
print ('t3.grad - t1.grad = ...')
print (t3.grad - t1.grad)
</code></pre>
<p>Here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
t1 = ...
tensor([[ 2.2871,  0.6413, -0.8615],
        [-0.3649, -0.6931,  0.9023]])
l1 = tensor(2.7625, grad_fn=&lt;SumBackward0&gt;)
l2 = tensor(2.7625, grad_fn=&lt;SumBackward0&gt;)
l3 = tensor(2.7625, grad_fn=&lt;SumBackward0&gt;)
t1.grad = ...
tensor([[ 0.0792,  0.7693, -0.7168],
        [-0.6137, -0.7680,  0.6963]])
t2.grad - t1.grad = ...
tensor([[0., 0., 0.],
        [0., 0., 0.]])
t3.grad - t1.grad = ...
tensor([[0., 0., 0.],
        [0., 0., 0.]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

VolcanoJ: <p>Thank you so much K. Frank. With your help I was able to fix my code and test out my theory in Pytorch. My initial tests (not conclusive yet) yielded a significant improvement in learning speed!</p>