Pytorch autograd not matching with computed derivatives?

nitesr: <p>I am not able to figure out why by hand derivatives are not matching that of pytorch’s.</p>
<p>Equations:</p>
<pre data-code-wrap="Math"><code class="lang-plaintext">h0 = 0.5
w_h = 2.0
w_y = 40.0
h1_raw = w_h * h0
h1 = tanh(h1_raw)
h2_raw = w_h * h1
h2 = tanh(h2_raw)
y2 = w_y * h2

find dy2/dw_h &amp; dy2/dw_y ?
</code></pre>
<p>W_h.grad : pythorch autograd on Y2 backward<br>
dy2_dwh: computed using analytic method<br>
dw_h: computed using backward pass on computation graph (CS231 standford course)</p>
<p>Here is the code and output.</p>
<pre><code class="lang-auto">import torch

W_h = torch.tensor(2.0, requires_grad=True)
w_h = W_h.detach().item()

W_y = torch.tensor(40.0, requires_grad=True)
w_y = W_y.detach().item()

H0 = torch.tensor(0.5, requires_grad=False)
h0 = H0.item()

H1_raw = W_h * H0
h1_raw = H1_raw.item()
H1 = torch.tanh(H1_raw)
h1 = H1.detach().item()

H2_raw = W_h * H1
h2_raw = H2_raw.detach().item()
H2 = torch.tanh(H2_raw)
Y2 = W_y * H2
h2 = H2.detach().item()

Y2.backward()

# analytic method for derivatives
#   dy/dw_y = h2 * dw_y/dw_y = h2 (product rule)
#   dy/dw_h = h2 * dw_y/dw_h + w_y * dh2/dw_h = w_y * dh2/dw_h (product rule &amp; w_y is contant for w_h)
#   dh2/dw_h = 1 - h2_raw**2 * dh2_raw/dw_h (chain rule)
#   dh2_raw/dw_h = h1 * dw_h/dw_h + w_h * dh1/dw_h = h1 + w_h * dh1/dw_h (product rule)
#   dh1/dw_h = 1 - h1_raw**2 * dh1_raw/dw_h (chain rule)
#   dh1_raw/dw_h = h0 * dw_h/dw_h + w_h * dh0/dw_h = h0 (product rule &amp; h0 is constant)
dh0_dwh = 0 #because H0 is constant for w_h
dh1raw_dwh = h0 + w_h * dh0_dwh
dh1_dwh = (1 - h1_raw**2) * dh1raw_dwh
dh2raw_dwh = h1 + w_h * dh1_dwh
dh2_dwh = (1 - h2_raw**2) * dh2raw_dwh
dwy_dwh = 0
dy2_dwh = h2 * dwy_dwh + w_y * dh2_dwh

dh2_dwy = 0
dy2_dwy = h2 + w_y * dh2_dwy

# backward pass over computation graph
dy2 = 1
dw_y = h2 * 1
dh2 = w_y * 1
dh2_raw = (1 - h2_raw**2) * dh2
dh1 = w_h * dh2_raw
dh1_raw = (1 - h1_raw**2) * dh1
dw_h = h1 * dh2_raw + h0 * dh1_raw

print(W_h.grad, "&lt;&gt;", dy2_dwh, "&lt;&gt;", dw_h)
print(W_y.grad, "&lt;&gt;", dy2_dwy, "&lt;&gt;", dw_y)
</code></pre>
<p>tensor(8.1888) &lt;&gt; -40.21530288726289 &lt;&gt; -40.21530288726289<br>
tensor(0.9093) &lt;&gt; 0.9092516899108887 &lt;&gt; 0.9092516899108887</p>
<p>Note at <a href="http://desmos.com/calculator" class="inline-onebox" rel="noopener nofollow ugc">Desmos | Graphing Calculator</a>, <code>d/dx(40 * tanh(x * tanh(x * 0.5)))</code> at x=2 is 8ish</p>

KFrank: <p>Hi Nitesh!</p>
<aside class="quote no-group" data-username="nitesr" data-post="1" data-topic="187917" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nitesr/48/63771_2.png" class="avatar"> nitesr:</div>
<blockquote>
<pre><code class="lang-auto">#   dh1/dw_h = 1 - h1_raw**2 * dh1_raw/dw_h (chain rule)
</code></pre>
</blockquote>
</aside>
<p>I haven’t looked at the details of your code, but it appears that your analytic<br>
formula for the derivative of <code>tanh()</code> is incorrect.</p>
<p>Best.</p>
<p>K. Frank</p>

nitesr: <aside class="quote no-group" data-username="nitesr" data-post="1" data-topic="187917">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nitesr/48/63771_2.png" class="avatar"> nitesr:</div>
<blockquote>
<p><code>dh1/dw</code></p>
</blockquote>
</aside>
<p>I am applying the chain rule,</p>
<pre><code class="lang-auto">  f(g(w_h) = f'(g) . g'(w_h) where f = tanh() = h1 &amp; g = h1_raw
   f' = 1 - f*f = 1 - h1*h1
   g' = dh1_raw/dw_h
</code></pre>
<p>I realized what I did wrong. Thank you <a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>buggy code</p>
<pre><code class="lang-auto">dh1_dwh = (1 - h1_raw**2) * dh1raw_dwh
dh2raw_dwh = h1 + w_h * dh1_dwh
dh2_dwh = (1 - h2_raw**2) * dh2raw_dwh
</code></pre>
<pre><code class="lang-auto">dh2_raw = (1 - h2_raw**2) * dh2
dh1 = w_h * dh2_raw
dh1_raw = (1 - h1_raw**2) * dh1
</code></pre>
<p>correction</p>
<pre><code class="lang-auto">dh1_dwh = (1 - h1**2) * dh1raw_dwh
dh2raw_dwh = h1 + w_h * dh1_dwh
dh2_dwh = (1 - h2**2) * dh2raw_dwh
</code></pre>
<pre><code class="lang-auto">dh2_raw = (1 - h2**2) * dh2
dh1 = w_h * dh2_raw
dh1_raw = (1 - h1**2) * dh1
</code></pre>