Why do we need to pass the gradient parameter to the backward function in PyTorch?

AbishekBashyal: <p>According to the docs, when we call the backward function to the tensor if the tensor is non-scalar (i.e. its data has more than one element) and requires gradient, the function additionally requires specifying <strong>gradient</strong>.</p>
<pre><code class="lang-auto">import torch
a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([1.,1.])) 

print(a.grad)
</code></pre>
<p><code>Output: tensor([20., 20.])</code></p>
<p>Now scaling the external gradient:</p>
<pre><code class="lang-auto">a = torch.tensor([10.,10.],requires_grad=True)
b = torch.tensor([20.,20.],requires_grad=True)

F = a * b
F.backward(gradient=torch.tensor([2.,2.])) #modified

print(a.grad)
</code></pre>
<p><code>Output: tensor([40., 40.])</code></p>
<p>So, passing the gradient argument to backward seems to scale the gradients.<br>
Also, by default <code>F.backward()</code> is <code>F.backward(gradient=torch.Tensor([1.]))</code></p>
<p>Apart from scaling the grad value how does the <code>gradient</code> parameter passed to the backward function helps to compute the derivatives when we have a non-scalar tensor?<br>
Why can’t PyTorch calculate the derivative implicitly without asking explicit gradient parameter as it did for the scalar-tensor?</p>
<p>Also, In the case of scalar value, <code>.backward()</code> w/o parameters is equivalent to <code>.backward(torch.tensor(1.0))</code>  then why can’t we use broadcasting of <code>torch.tensor(1.0)</code> so that it works for not-scalar inputs also instead of explicitly having to pass external <code>gradient</code> parameter while calculating the vector Jacobian product?</p>

AlphaBetaGamma96: <p>I’m pretty sure when you do something like <code>y.backward(torch.ones_like(y))</code> you’re just telling autograd to repeat <code>.backward()</code> for each element in <code>y</code> under the hood.</p>

AbishekBashyal: <p><a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> can you share some resources showing that describes:</p>
<blockquote>
<p><code>y.backward(torch.ones_like(y))</code> is telling autograd to repeat <code>.backward()</code> for each element in <code>y</code> under the hood.</p>
</blockquote>

albanD: <p>Hi,</p>
<p>If you consider a function f that has n_input and n_output. And a Jacobian matrix containing all its partial derivatives J_f of size (n_output x n_input).<br>
Then what backpropagation (or AD whichever way you want to name it) does is to compute <code>v^T J_f</code> for a given <code>v</code>.<br>
If your function has a single output, then it makes sense to take <code>v = 1.</code> so that backprop will return <code>J_f</code>.<br>
But if you have multiple outputs, there is no good default and so we requirethe user to provide the <code>v</code> value they want.</p>
<p>If you want to reconstruct the full J_f, you will have to do as many backwards as there are outputs in your function. You can use the autograd.functional.jacobian function if you need that.</p>

AbishekBashyal: <p><a class="mention" href="/u/alband">@albanD</a>  thanks, its clear now :D.</p>