Clarification - Using backward() on non-scalars

username: Mattie
cooked: <p>Hey,</p>
<p>Sorry if this is obvious, but I find the description of the <code>torch.autograd.backward(variables, grad_variables, retain_variables=False)</code> function quite confusing.</p>
<p>I’m working on a project where I have a vector of variables that I would like to differentiate to find the Jacobian. When it comes to implementing this, I’m not sure what form <code>grad_variables</code> should be or what a ‘sequence of Tensor’ is. I’ve tried many things, but all throw an error.</p>
<p>Would anyone be able to point me in the direction of an example if one exists? If not, say I had the following super simple example:</p>
<p><code>x = Variable(torch.FloatTensor([[2,1]]), requires_grad=True) M = Variable(torch.FloatTensor([[1,2],[3,4]])) y = torch.mm(x,M)</code></p>
<p>What should the arguments for <code>y.backward()</code> be so that I can find <code>[[dy1/dx1, dy1/dx2], [dy2/dx1, dy2/dx2]]</code> (i.e. recover M)?</p>

username: colesbury
cooked: <p>The naming of <code>grad_variables</code> might be a little bit confusing. In the context of neural networks, it’s the “loss”.</p>
<p>To recover M requires two calls to backwards. Here’s how with <code>Variable.backward()</code>:</p>
<pre><code class="lang-auto">x = Variable(torch.FloatTensor([[2,1]]), requires_grad=True)
M = Variable(torch.FloatTensor([[1,2],[3,4]]))
y = torch.mm(x, M)
jacobian = torch.FloatTensor(2, 2).zero_()
y.backward(torch.FloatTensor([[1, 0]]), retain_variables=True)
jacobian[:,0] = x.grad.data
x.grad.data.zero_()
y.backward(torch.FloatTensor([[0, 1]]), retain_variables=True)
jacobian[:,1] = x.grad.data
</code></pre>
<p>You can also replace the <code>y.backward()</code> calls are equivalent to:</p>
<p><code>torch.autograd.backward([y], [torch.FloatTensor([[1, 0]])], retain_variables=True)</code></p>

username: Mattie
cooked: <p>Fab, that’s a great help. Thanks for your time.</p>

username: linlin
cooked: <blockquote>
<p>The naming of grad_variables might be a little bit confusing. In the context of neural networks, it’s the “loss”.</p>
</blockquote>
<p>I am a bit confused by this. In this example, isn’t that the “loss” is y?</p>
<p><code>torch.FloatTensor([[1, 0]])</code>  is passed to <code>grad_variables</code> and specifies that we’ll use the first column of y’s gradients, and <code>torch.FloatTensor([[0, 1]])</code> for the second column.</p>
<p>By default, <code>grad_variables</code> is <code>torch.Tensor([1])</code> which means we will just keep calculated gradients.</p>
<p>Is that correct?</p>
<p>Thanks a lot!</p>

username: jsm
cooked: <p>Thanks for your intuitive example.<br>
I’m getting a bit lost on the usage of the following line:<br>
<code>y.backward(torch.FloatTensor([[1, 0]]), retain_variables=True)</code><br>
especially when it comes to the <code>gradient</code> argument.</p>
<p>As <a class="mention" href="/u/linlin">@linlin</a> already mentioned, in the provided example the argument <code>gradient=torch.FloatTensor([[1,0]])</code><br>
is used to acquire information for a column (e.g. the first one in the above scenario) or to define the variable<br>
that we compute the gradient with respect to (as seen from: <a href="http://pytorch.org/docs/master/autograd.html#torch.autograd.Variable.backward" rel="nofollow noopener">http://pytorch.org/docs/master/autograd.html#torch.autograd.Variable.backward</a>) ?</p>
<p>If we use it with respect to a given variable why<br>
<code>y.backward(torch.FloatTensor([[1, 0]]), retain_variables=True)</code><br>
<code>jacobian[:,0] = x.grad.data</code><br>
<code>x.grad.data.zero_()</code><br>
<code>y.backward(torch.FloatTensor([[0, 1]]), retain_variables=True)</code><br>
<code>jacobian[:,1] = x.grad.data</code></p>
<p>is used instead of</p>
<p><code>y.backward(x * torch.FloatTensor([[1, 0]]), retain_variables=True)</code><br>
<code>jacobian[:,0] = x.grad.data</code><br>
<code>x.grad.data.zero_()</code><br>
<code>y.backward(x * torch.FloatTensor([[0, 1]]), retain_variables=True)</code><br>
<code>jacobian[:,1] = x.grad.data</code></p>
<p>for computing  <code>[[dy1/dx1, dy1/dx2], [dy2/dx1, dy2/dx2]]</code> ?</p>
<p>Thank you in advance!</p>

username: jdhao
cooked: <p>I think derivative of y w.r.t to x is not M but transpose of M.<br>
dy1/dx = [M11, M21], dy2/dx = [M12, M22].</p>
<p>Also, you can see <a href="https://stackoverflow.com/questions/43451125/pytorch-what-are-the-gradient-arguments/47026836#47026836">here</a> for the meaning of <code>gradient</code> argument in <code>backward()</code> method.</p>

username: jsm
cooked: <p>Great answer <a class="mention" href="/u/jdhao">@jdhao</a>! This solves my ambiguities. Thanks a lot!!!<br>
I guess this answers <a class="mention" href="/u/linlin">@linlin</a> question.</p>

username: jdhao
cooked: <p>cheers! Stackoverflow is your friend <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: linlin
cooked: <p>Thanks to <a class="mention" href="/u/jsm">@jsm</a>, <a class="mention" href="/u/jdhao">@jdhao</a>. It clarified the cloud away.</p>
<p>But I still think the naming of <code>grad_variables</code> is a bit misleading. Something like <code>grad_weighting</code> would be more intuitive.</p>

username: yottabytt
cooked: <p>Stackoverflow’s question and answers were perfect. Thanks for the link to SO thread <a class="mention" href="/u/jdhao">@jdhao</a>.</p>

username: jdhao
cooked: <p>You can give upvote to support that answer if you have stackoverflow account <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: saan77
cooked: <p>Thanks for the explanation <a class="mention" href="/u/colesbury">@colesbury</a>.<br>
I just want to know what if my <code>y</code> is an output of the inceptionv3 model. In that case, I have to run a loop (1000 iterations). Is there a better way to do this?</p>

username: Jolyon
cooked: <p>OMG, thank u so much! I wasted tons of time on understanding so-called gradient on the non-scalar output and you make it clear using one word: “loss” <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=6" title=":laughing:" class="emoji" alt=":laughing:"></p>

username: Huimin_ZENG
cooked: <p><a class="mention" href="/u/saan77">@saan77</a> Hi! Did you solve it?</p>