'gradient' argument in out.backward(gradient)

username: xu-song
cooked: <h2><a name="q1-what-is-the-purpose-of-gradient-argument-1" class="anchor" href="#q1-what-is-the-purpose-of-gradient-argument-1"></a>Q1: what is the purpose of <code>gradient</code> argument?</h2>
<p><code>gradient</code> will perform a dot product on <code>out</code>, right?</p>
<p>what is the objective of this operation?</p>
<h2><a name="q2-why-gradient-argument-is-required-for-tensor-outputs-2" class="anchor" href="#q2-why-gradient-argument-is-required-for-tensor-outputs-2"></a>Q2:  Why <code>gradient</code> argument is required for tensor outputs?</h2>
<p>Can we use the following backward, which not specify the <code>gradient</code> argument?</p>
<pre data-code-wrap="python"><code class="lang-python">out.backward()
</code></pre>
<p>It will cause the following error</p>
<blockquote>
<p>“grad can be implicitly created only for scalar outputs”</p>
</blockquote>
<p>Why is the gradient argument required in tensor outputs? We can set a tensor of ones as the default value of <code>gradient</code>.</p>

username: jpeg729
cooked: <ol>
<li>The shape of the params has absolutely no relation to the required shape of the argument to out.backward.<br>
<code>params</code> is a list containing the weight tensors of the various modules in your net, so in this case the net has two Conv2d layers which have 2 tensors of parameters each (<code>.weight</code> and <code>.bias</code>), and three Linear layers which each have 2 tensors of parameters (<code>.weight</code> and <code>.bias</code>). That makes 10 tensors in total.</li>
</ol>
<p>The last layer of <code>net</code> has 10 outputs which is why <code>out.backward</code> needs 10 inputs. One per output.</p>
<ol start="2">
<li>Calling <code>out.backward()</code> will --calculate-the-gradient-of-the-output–. EDIT: <code>out.backward()</code> is equivalent to <code>out.backward(torch.Tensor([1]))</code><br>
Usually we need the gradient of the loss. e.g.</li>
</ol>
<pre><code class="lang-auto">out = net(input)
loss = torch.nn.functional.mse_loss(out, target)
loss.backward()
</code></pre>
<ol start="3">
<li>Each time you run <code>.backward()</code> the stored gradients for each parameter are updated by <em>adding</em> the new gradients. This allows us to cumulate gradients over several samples or several batches before using the gradients to update the weights. Once you have updated the weights you don’t want to keep those gradients around because if you reuse them, then you will push the weights too far.</li>
</ol>

username: xu-song
cooked: <aside class="quote no-group" data-username="jpeg729" data-post="2" data-topic="12742">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<p>The shape of the params has absolutely no relation to the required shape of the argument to out.backward.</p>
</blockquote>
</aside>
<p>Yes,  The shape of the argument should same with the shape of out</p>

username: jpeg729
cooked: <p>Concerning <code>out.backward()</code>, I was mistaken, you are right. It is equivalent to doing <code>out.backward(torch.Tensor([1]))</code></p>
<p>The params are all declared using <code>Variable(.., requires_grad=True)</code> or something equivalent. This means that whenever you use those params in a calculation pyTorch assumes you are going to want to calculate the gradient with respect to those params and it saves all the details it is going to need in order to calculate those gradients.</p>
<p>The value <code>out</code> was calculated using the params listed in <code>net.parameters()</code>, so whenever you call <code>out.backward</code> the gradient will be calculated with respect to every one of those parameters.</p>

username: richard
cooked: <p>For your first question, the output of your network <code>out = net(input)</code> has size <code>(1, 10)</code>, so you want the gradient you pass in to match that size.</p>

username: xu-song
cooked: <h2>what does <code>gradient</code> argument do in backward()?</h2>
<p>I make a test by the following case.</p>
<pre><code class="lang-python">import torch
from torch.autograd import Variable

x = Variable(torch.ones(2, 3), requires_grad=True)
a = Variable(torch.ones(3, 4))
out = torch.mm(x,a)    # shape: (2,4)

gradient = torch.FloatTensor([[1,2,3,4],[5,6,7,8]]) # shape=(2,4)
#gradient = torch.ones(4, 4)
out.backward(gradient)
print(x.grad)

</code></pre>
<p><code>x</code></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>x_11</th>
<th>x_12</th>
<th>x_13</th>
</tr>
</thead>
<tbody>
<tr>
<td>x_21</td>
<td>x_22</td>
<td>x_23</td>
</tr>
</tbody>
</table>
</div><p><code>out</code></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>x_11+x_12+x_13</th>
<th>x_11+x_12+x_13</th>
<th>x_11+x_12+x_13</th>
<th>x_11+x_12+x_13</th>
</tr>
</thead>
<tbody>
<tr>
<td>x_21+x_22+x_23</td>
<td>x_21+x_22+x_23</td>
<td>x_21+x_22+x_23</td>
<td>x_21+x_22+x_23</td>
</tr>
</tbody>
</table>
</div><p><code>gradient*out</code></p>
<div class="md-table">
<table>
<thead>
<tr>
<th>x_11+x_12+x_13</th>
<th>2x_11+2x_12+2x_13</th>
<th>3x_11+3x_12+3x_13</th>
<th>4x_11+4x_12+4x_13</th>
</tr>
</thead>
<tbody>
<tr>
<td>5x_21+5x_22+5x_23</td>
<td>6x_21+6x_22+6x_23</td>
<td>7x_21+7x_22+7x_23</td>
<td>8x_21+8x_22+8x_23</td>
</tr>
</tbody>
</table>
</div><p><code>x.grad</code></p>
<pre><code class="lang-auto">Variable containing:
 10  10  10
 26  26  26
</code></pre>
<p>10 = 1+2+3+4+0+0+0+0<br>
26=0+0+0+0+5+6+7+8</p>
<hr>
<p>I find that, gradient will perform a <strong>dot product</strong> on output in the above case.<br>
What is the objective of this operation?</p>

username: jpeg729
cooked: <p>Might I suggest a <a href="https://youtu.be/GZTvxoSHZIo?list=PLlJy-eBtNFt6EuMxFYRiNRS07MCWN5UIA&amp;t=192" rel="nofollow noopener">lecture on backpropagation by Andrej Karpathy</a></p>

username: richard
cooked: <p>What do you mean? I don’t think that’s what it does.</p>
<p>Assume that you have some <code>input</code> and a function <code>f</code> such that <code>f(input) = output</code></p>
<p>The gradient you pass to <code>output.backward()</code> is the “gradient of the output”. What happens with backpropagation is that we use the gradient of the output to compute the gradient of the input.</p>

username: xu-song
cooked: <aside class="quote no-group quote-modified" data-username="richard" data-post="11" data-topic="12742">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p>“The gradient you pass to output.backward() is the “gradient of the output”.</p>
</blockquote>
</aside>
<p>sorry, I don’t quite understand the gradient of output?<br>
<code>the gradient of output w.r.t. model parameters</code> = <code>d(out)/d(param)</code> = lim(Δout)/(Δparam).</p>
<p><code>the gradient of output</code> = d(out)?  or   <code>Δout</code> ?   or  <code>backward propagation errors</code>?<br>
<a class="mention" href="/u/richard">@richard</a> <a class="mention" href="/u/jpeg729">@jpeg729</a>  can u explain it mathmatically?</p>

username: richard
cooked: <p>Let’s say your <code>output</code> is not a tensor with size 1. In addition, assume that there’s some function <code>f</code> that you’re acting on the <code>output</code>, so that you get something like <code>newOutput = f(output)</code>.</p>
<p>You can pass a gradient <code>grad</code> to <code>output.backward(grad)</code>. The idea of this is that if you’re doing backpropagation manually, and you know the gradient of the input of the next layer (<code>f</code> in this case), then you can pass the gradient of the input of the next layer to the previous layer that had output <code>output</code>.</p>

username: wuuw
cooked: <p>Thanks!!! You really helped me!</p>

username: singhvishal0209
cooked: <p>suppose I have a image A of size 720*720 and I want to calculate its gradient.</p>
<p>Hence I will use A.backward() but since A is not scalar I need to give gradient as an argument to backward. I understand that the shape of gradient should also be 720*720 but what should be the value of it?</p>
<p>Should I randomly initialize a tensor of shape 720*720</p>

username: humble_D
cooked: <p>You don’t calculate gradients of a tensor, you calculate gradients of an operation done on that tensor.</p>
<p>Now coming back to your question, now if you actually came up with an operation that returned a 720^2 tensor then yes, you would want to pass in a gradient tensor equivalent to that shape. The reason why you pass this tensor is explained in Andrej Karpathy’s lecture. Go take a look.</p>