What is the grad_outputs kwarg in autograd.grad?

jwillette: <p>I am having trouble understanding exactly what this line means in the <a href="https://pytorch.org/docs/stable/autograd.html" rel="nofollow noopener">docs</a>…</p>
<pre><code class="lang-auto">grad_outputs (sequence of Tensor) – The “vector” in the Jacobian-vector product. Usually gradients w.r.t. each output. None values can be specified for scalar Tensors or ones that don’t require grad. If a None value would be acceptable for all grad_tensors, then this argument is optional. Default: None.
</code></pre>
<p>I see this <a href="https://discuss.pytorch.org/t/what-does-grad-outputs-do-in-autograd-grad/18014">thread</a> which partially explains it (<code>None</code> is equivalent to passing in <code>torch.ones(...)</code> of the proper size) but I still don’t really understand what it is for or what it should be used for.</p>
<p>Any input? Thanks</p>

albanD: <p>Hi,</p>
<blockquote>
<p><code>None</code> is equivalent to passing in <code>torch.ones(...)</code> of the proper size</p>
</blockquote>
<p>This is only true for an output with a single element!</p>
<p>Otherwise, you can see these outputs as providing <code>dL/dout</code> (where <code>L</code> is your loss) so that the autograd can compute <code>dL/dw</code> (where <code>w</code> are the parameters for which you want the gradients) as <code>dL/dw = dL/dout * dout/dw</code>.</p>
<p>Another way to see this as mentioned in the doc is that autograd only computes a vector matrix product between a vector v and the Jacobian of the function. <code>grad_outputs</code> allow you to specifiy this vector <code>v</code>.</p>

jwillette: <p>Thanks for your answer, so the vector passed in will not be mutated, but it will have an effect on the final gradients that come out of the <code>grad</code> function?</p>
<p>Is there a simple use case to illustrate why someone would need this?</p>

albanD: <p>In most cases, you can do without it, but for example, you can replace:</p>
<pre><code class="lang-auto">loss = l1 + 2 * l2
autograd.grad(loss, inp)
</code></pre>
<p>by</p>
<pre><code class="lang-auto">autograd.grad((l1, l2), inp, grad_outputs=(torch.ones_like(l1), 2 * torch.ones_like(l2))
</code></pre>
<p>Which is going to be slightly faster.<br>
Also some algorithms require you to compute <code>x * J</code> for some <code>x</code>. You can avoid having to compute the full Jacobian J by simply providing <code>x</code> as a grad_output.</p>

jwillette: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="94378">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>This is only true for an output with a single element!</p>
</blockquote>
</aside>
<p>Thanks for the help. Just one more thing. It seems that by the code you posted, passing in <code>torch.ones(...)</code> will not have a material affect on the final outcome, right? seems like that conflicts with the comment about a single element, but I am not sure</p>

albanD: <p>I assume above that l1 and l2 are scalar value! Sorry <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"><br>
I just use <code>ones_like()</code> to get a Tensor with a 1 on the right device and with the right dtype.</p>

sxcai188: <p>this example really useful for me to understand the grad_outputs argument, I think it could be added to the document of autograd to help more people like me</p>

rhossein: <p>Thanks for that answer, I would add that torch.ones could be seen as the derivative of the identity map, in this way the backward differentiation can be initialized. It acts as a seed in some sense !</p>