What's the difference between torch.autograd.grad and backward()?

michaelfang: <ul>
<li>Are there any functional differences between the two pieces of code?</li>
</ul>
<pre><code class="lang-auto">dydx = torch.autograd.grad(outputs=y,
                           inputs=x,
                           grad_outputs=weight,
                           retain_graph=True,
                           create_graph=True,
                           only_inputs=True)

**and**
y.backward()
</code></pre>
<p>the both code is to compute the grad. When use optimizer.step() after executing y.backward(), value of x will be updated, but if i use  torch.autograd.grad instead of y.backward(), value of x will not be updated. Is it right?</p>

albanD: <p>Hi,</p>
<p>The difference is that <code>autograd.grad()</code> is returning the gradients to you.<br>
While <code>.backward()</code> is populating the <code>.grad</code> field on the different leaf Tensors that were used to compute <code>y</code>.</p>
<p>In particular, this <code>.grad</code> field is used by the optimizers to update the weights. So if you use <code>autograd.grad()</code> you will need to populate these fields yourself based on the gradients it returned before calling <code>optimizer.step()</code>.</p>

michaelfang: <p>thank you so much！I finally figure it out.</p>

Nazila-H: <p>Hi,<br>
Thanks for your explanation.<br>
I have to use <code>autograd.grad()</code> to return the gradients for me as following:</p>
<pre><code class="lang-auto">def loss_fn(pred, target):
    return F.cross_entropy(input=pred, target=target)
data = data.clone().detach().requires_grad_(True)
output= model(data)
loss = loss_fn(pred=output, target=label)
grad = torch.autograd.grad(outputs=loss, inputs=data, allow_unused=True)
</code></pre>
<p>I want to update the weights of the network as well .<br>
As you told I need to populate <code>.grad</code> field myself based on the gradients it returned, before calling <code>optimizer.step()</code>. Could you please tell me how can I do it?</p>
<p>Thanks in advance.</p>

albanD: <p>Hi,</p>
<p>If you want the gradients both for the parameters and the input, then you only need to make sure that your inputs is a leaf that requires grad. Which you already do by doing <code>.detach().requires_grad_(True)</code>.<br>
Then you can just call <code>.backward()</code> to get the gradient for all the leafs (so the input as well as all the weights in the net).<br>
You can get the <code>grad</code> that correspond to the gradient of the input by accessing <code>data.grad</code>.</p>

Nazila-H: <p>Thanks alot. It works.</p>

Shivani_Malhotra: <p>Hello</p>
<p>I have been working on VQ-VAE and facing some issues on .grad function. If you look at the attachment, it shows loss.backward(). I wish to access the gradients of loss before they reach the input again through backpropagation. But loss.grad() isnt working when I apply it after backward. Kindly help.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/827cbf65c9cb5836a563463ac7db3b20b0671fbc.png" alt="doubt" data-base62-sha1="iCllbINzSiwCbJkWgHFTsTgUI0I" width="537" height="241"></p>

bpfrd: <p>If we want to populate the grad manually how do you suggest to do that¿</p>

pietrolesci: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, coming a bit late to this thread. Why do we need to <code>.detach()</code> the input tensor to get the gradient?</p>

albanD: <p>The <code>.grad</code> field is only populated for Tensors that are leafs (no gradient history). And you can use .detach() to break the link with the history, thus making the Tensor a leaf.</p>
<p>Note that if you don’t want that, you can use <code> t.retain_grad()</code> to force the .grad field to be populated on a non-leaf.<br>
You can also use autograd.grad() to get the gradients wrt any Tensor you want without the .grad field being updated.</p>