Autograd.grad accumulates gradients on sequence of Tensor making it hard to calculate Hessian matrix

tianle: <p>I encountered some unexpected behavior when using torch.autograd.grad <a href="https://pytorch.org/docs/stable/_modules/torch/autograd.html#grad" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd — PyTorch 2.1 documentation</a></p>
<p>Here is a toy example:</p>
<pre><code class="lang-auto">x = torch.tensor([5, 3], dtype=torch.float, requires_grad=True)
y = x
print(torch.autograd.grad(outputs=y, inputs=x, grad_outputs=torch.ones_like(y), create_graph=True))
</code></pre>
<p>The output is correct:<br>
<code>(tensor([ 1.,  1.]),)</code></p>
<p>However, if I tried to handle a sequence of tensors at a time, the result is unexpected:</p>
<pre><code class="lang-auto">torch.autograd.grad(outputs=[y,y], inputs=[x,x], 
                          grad_outputs=[torch.ones_like(y), torch.ones_like(y)], 
                          create_graph=True)
</code></pre>
<p>I would expect the result to be:<br>
<code>(tensor([ 1.,  1.]), tensor([ 1.,  1.]))</code><br>
But, the actual result is:<br>
<code>(tensor([ 2.,  2.]), tensor([ 2.,  2.]))</code></p>
<p>I would assume the calculation of gradients is independent when the inputs/outputs of autograd.grad function are sequence of Tensor, but it appears not.<br>
Here is a slightly more complicated example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/cdae313bca54c66333a91427faa2d5c56411bb7a.png" data-download-href="https://discuss.pytorch.org/uploads/default/cdae313bca54c66333a91427faa2d5c56411bb7a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdae313bca54c66333a91427faa2d5c56411bb7a_2_690x201.png" alt="image" data-base62-sha1="tlx5nw9exfnWLluRMHmfgiJxQvw" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdae313bca54c66333a91427faa2d5c56411bb7a_2_690x201.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/cdae313bca54c66333a91427faa2d5c56411bb7a_2_1035x301.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/cdae313bca54c66333a91427faa2d5c56411bb7a.png 2x" data-dominant-color="ECEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1358×396 52.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am not sure if this is the intended behavior. Any suggestion is appreciated.</p>

alexis-jacq: <p>Because, for a given variable, it will accumulate the computed gradients.</p>
<p>So if you give to <code>autograd.grad</code> two times y (that’s what you do by giving a list [y,y] as outputs), it understands you did two times the operation y=x, so the grad of y will sum – two times – the corresponding gradient. And it will return the gradient of each output of the list, hence two times the gradient of y, which was computed as so <code>[1+1, 1+1]</code>.</p>
<p>If instead, you try:</p>
<pre><code class="lang-auto">x = torch.tensor([5, 3], dtype=torch.float, requires_grad=True)
y = x
z = x
torch.autograd.grad(outputs=[y,z], inputs=[x,x], 
                          grad_outputs=[torch.ones_like(y), torch.ones_like(y)], 
                          create_graph=True)
</code></pre>
<p>You obtain what you expected :</p>
<pre><code class="lang-auto">([1, 1], [1, 1])
</code></pre>
<p>because here no variable was accumulating two operations in the given sequence</p>

tianle: <p>Thanks for your reply <a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> .<br>
I tried your code, but still get unexpected result on my machine (pytorch0.4):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4efb7af261b28d74b671047f8fc86c5abea2a49f.png" data-download-href="https://discuss.pytorch.org/uploads/default/4efb7af261b28d74b671047f8fc86c5abea2a49f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4efb7af261b28d74b671047f8fc86c5abea2a49f_2_690x192.png" alt="image" data-base62-sha1="bgI2gXiJ2mAzXbVaxiAM8IhQaar" width="690" height="192" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4efb7af261b28d74b671047f8fc86c5abea2a49f_2_690x192.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4efb7af261b28d74b671047f8fc86c5abea2a49f_2_1035x288.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4efb7af261b28d74b671047f8fc86c5abea2a49f.png 2x" data-dominant-color="EFF0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1101×307 25.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Besides, after autograd.grad() is called, x.grad is None.<br>
However, gradients must have been accumulated somewhere somehow during the calculation, but I don’t know how it is done (my second example in the original post is more tricky to find out how gradient is accumulated).</p>
<p>I know torch.autograd.backward() will accumulate gradients, should autograd.grad() also accumulate gradients? Apparently it does during the call, but the gradients won’t be save to variables.</p>
<p>Besides, my original goal of using autograd.grad is to calculate Hessian matrix (not a Hessian vector: <a href="https://github.com/pytorch/pytorch/blob/master/test/test_autograd.py#L195" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/test_autograd.py at main · pytorch/pytorch · GitHub</a>). Any suggestion on this is appreciated.</p>

alexis-jacq: <p>Oh sorry, I edited my post:<br>
It’s <code>outputs=[y,z]</code>, not  <code>outputs=[y,y]</code> that I meant !</p>
<pre><code class="lang-auto">torch.autograd.grad(outputs=[y,z], inputs=[x,x], 
                          grad_outputs=[torch.ones_like(y), torch.ones_like(y)], 
                          create_graph=True)
</code></pre>
<p>So, if in the sequence you give, you have two times the same variable, the gradient for this variable will be accumulated two times. That’s as simple as this!</p>

tianle: <p>Thanks <a class="mention" href="/u/alexis-jacq">@alexis-jacq</a> .</p>
<p>I tried you modified code, but still got the same result on my machine:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/25e94ec1978783edbf0ef47ed03944618dcd70a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/25e94ec1978783edbf0ef47ed03944618dcd70a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/25e94ec1978783edbf0ef47ed03944618dcd70a9_2_690x186.png" alt="image" data-base62-sha1="5pnwASxZfrwurYJC4ELGOcgCPNL" width="690" height="186" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/25e94ec1978783edbf0ef47ed03944618dcd70a9_2_690x186.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/25e94ec1978783edbf0ef47ed03944618dcd70a9_2_1035x279.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/25e94ec1978783edbf0ef47ed03944618dcd70a9.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1105×299 25.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For this simple example, it is clear to see the gradient has been accumulated two times as you said. However, for the second example in my original post, it is not clear how it works. I am still not clear about how the computation graph is constructed when autograd.grad is called.</p>

alexis-jacq: <p>Try print(y.grad)… Of course, x has again accumulated two gradients.</p>
<p>In your second example, the accumulation is more complexe, but still simple :</p>
<pre><code class="lang-auto">grad(y) = dy/dx = 2,2;        grad(z) = ones_like(y) = [1,1] # call for grad(y,x)
  |                             |
  +                             +
  |                             |
grad(y)=dy/dx*dz/dy = 40,24 ; grad(z) = dz/dy = 20,12 # call for grad(z,y)
  ||                            ||
42,26                          21,13  # call for grad([z,y],[y,x])</code></pre>

tianle: <p>Now I understand how gradients are accumulated, thank you very much <a class="mention" href="/u/alexis-jacq">@alexis-jacq</a>!</p>
<p>I think when autograd.grad is called, the currently computation graph is copied, and then backward() is called on the copied graph. That’s why the mechanism of gradient accumulation in autograd.grad() is the same as that in backward(). The difference is that when autograd.grad() is called, computation is done on the newly created graph (setting all gradients of leaf nodes to None if there is any), and the gradients won’t be accumulated or returned on the original graph.</p>
<p>I found the only way to avoid accumulation of gradients using autograd.grad on a sequence of Tensor is to create separate graphs without sharing nodes. For example:</p>
<pre><code class="lang-auto">x = torch.tensor([5, 3], dtype=torch.float, requires_grad=True)
y = x
w = torch.tensor([5, 3], dtype=torch.float, requires_grad=True)
z = w
torch.autograd.grad(outputs=[y,z], inputs=[x,w], 
                          grad_outputs=[torch.ones_like(y), torch.ones_like(z)], 
                          create_graph=True)
</code></pre>
<p>Then the result will be<br>
<code>(tensor([ 1., 1.]), tensor([ 1., 1.]))</code></p>
<p>However, in order to calculate a Hessian matrix, we have to calculate second order partial derivatives separately, but there is no way to call zero_grad() during the the call of autograd.grad() on a sequence of Tensor. The only way to get around is to call autograd.grad() on one Tensor at a time, instead of on a sequence of Tensor at a time.<br>
Please correct me if I am wrong.</p>
<p>Thanks!</p>

Johnny-Wish: <p>I don’t think this will help. Because setting</p>
<pre><code class="lang-auto">y = x
z = x
</code></pre>
<p>is simply copying the <strong>reference</strong> of the Tensor object <code>x</code> to another two variable names <code>y</code> and <code>z</code>. All of the three hold reference to exactly the same object.</p>