Why .backward() is giving different gradients as compared to analytical computation when retain_graph = True?

aknirala: <p>Here is my code and reasoning, kindly let me know if my calculation is incorrect.</p>
<pre><code class="lang-auto">#Consider Y = 3X; (We take X = 1 and Y = 4 and perurb X by alpha = 0.001 to get new X)
X = torch.ones(1).requires_grad_()
Y1 = torch.ones(1).requires_grad_()*4
Y2 = torch.ones(1).requires_grad_()*4
p_y1 = 3*X
l1 = F.mse_loss(p_y1, Y1)
grad = torch.autograd.grad(l1, [X], retain_graph=True)[0]
print("grad1:", grad)
nX = X + grad.data*0.001
p_y2 = 3*nX
final_y = (p_y1 + p_y2)/2
print("Final_y:", final_y)
l2 = F.mse_loss(final_y, Y2)
grad = torch.autograd.grad(l2, [X])[0]
print("Loss: ",l2, "Final grad: ", grad)
</code></pre>
<p>This gives output:</p>
<pre><code class="lang-auto">grad1: tensor([-6.])
Final_y: tensor([2.9910], grad_fn=&lt;DivBackward0&gt;)
Loss:  tensor(1.0181, grad_fn=&lt;MseLossBackward0&gt;) Final grad:  tensor([-6.0540])
</code></pre>
<p>Now, I try to compute this analytically as:</p>
<pre><code class="lang-auto">#loss = (y - 3x)^2 ==&gt; grad_x = 2(y-3x)(-3) = -6(y - 3x)
y = 4
x = 1
grad1 = -6*(y-3*x)
grad1
</code></pre>
<p>This gives output as:<br>
<code>-6</code><br>
This is as expected.</p>
<pre><code class="lang-auto">"""
Now, x get's modified as: x2 = x + 0.001*grad1
we get new Y as: 3*( x + 0.001*grad1) = 3*(x -0.006* (y - 3x) ) = 3x -0.018y +0.054x
And this is averaged with old y: 3*x to get final output: (3x + 3x -0.018y +0.054x)/2 = 3x +0.027x - 0.009y
"""
final_p = 3*x +0.027*x - 0.009*y
final_p #This is final y, so far so good
</code></pre>
<p>This gives output as:<br>
<code>2.991</code><br>
This is also as expected.</p>
<pre><code class="lang-auto">"""
Now, wrt this what will be grad X?
loss = (y - 3x -0.027x + 0.009y)^2 #Note here two y's have same value...so...
grad_x = 2(y - 3x -0.027x + 0.009y) (-3 - 0.027)
"""
(y - 3*x -0.027*x + 0.009*y)**2 #This is loss also seems reasonable
</code></pre>
<p>This gives output as:<br>
<code>1.0180809999999998</code><br>
This is also as expected (more or less)</p>
<p><code>2*(y - 3*x -0.027*x + 0.009*y)*(-3 - 0.027)  #This is final grad on X, WHY THE DIFF?</code><br>
This gives output as:<br>
<code>-6.108486</code><br>
This is different <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"> Why, pyTorch calculated -6.0540?</p>
<p>EDIT: It seems that pyTorch is considering first addition to X as constant!! as evident from code below. How can I make it treat as a function of X?</p>
<pre><code class="lang-auto">X = torch.ones(1).requires_grad_()
Y1 = torch.ones(1).requires_grad_()*4
Y2 = torch.ones(1).requires_grad_()*4
p_y1 = 3*X
"""l1 = F.mse_loss(p_y1, Y1)
#l1.backward(retain_graph=True)
grad = torch.autograd.grad(l1, [X], retain_graph=True)[0]
print("grad1:", grad)
delX = grad.data*0.001"""
delX = torch.autograd.Variable(torch.ones(1)*-0.006)
nX = X + delX.detach().clone()
p_y2 = 3*nX
final_y = (p_y1 + p_y2)/2
print("Final_y:", final_y)
l2 = F.mse_loss(final_y, Y2)
grad = torch.autograd.grad(l2, [X])[0]
#l2.backward()
print("Loss: ",l2, "Final grad: ", grad)
</code></pre>
<p>This gives the same output as before:</p>
<pre><code class="lang-auto">Final_y: tensor([2.9910], grad_fn=&lt;DivBackward0&gt;)
Loss:  tensor(1.0181, grad_fn=&lt;MseLossBackward0&gt;) Final grad:  tensor([-6.0540])
</code></pre>
<p>This matches my analytical calculations as well, but how can I get the actual gradient using pyTorch?</p>