Grad returns None or throws error

username: kuzand
cooked: <p>I have a simple input tensor <code>x</code> and a function <code>f(x) = 2*x</code>. The gradient of x is df/dx = 2. How to get this result using pytorch? This is what I tried:</p>
<pre><code class="lang-auto">x = torch.tensor([0.0, 1.0], requires_grad = True)
f = 2*x
f.backward()
print(x.grad)
</code></pre>
<p>This throws the following error:</p>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
<p>Then I tried:</p>
<pre><code class="lang-auto">x = torch.tensor([0.0, 1.0], requires_grad = True)
f = 2*x
f.backward(torch.FloatTensor([1, 1]))
print(x.grad)
</code></pre>
<p>This works, returning the following result: <code>tensor([2., 2.])</code>. I am still not sure though why <code>f.backward()</code> doesn’t work and what is the explanation of <code>torch.FloatTensor([1, 1])</code>. Ideally I would expect to get <code>2.</code> and not <code>[2., 2.]</code>.</p>
<p>However, the following returns None:</p>
<pre><code class="lang-auto">x = torch.tensor([0.0, 1.0], requires_grad = True)
y = 1*x
f = 2*y
f.backward(torch.FloatTensor([1, 1]))
print(y.grad)
</code></pre>
<p>Why is this happening?</p>

username: albanD
cooked: <p>Hi,</p>
<p>Few things:</p>
<ul>
<li>The .grad field is only populated for leafs (tensor with no history) that require gradients when you call <code>.backward()</code>. In particular that is why you <code>y</code> at the end does not have its .grad field populated: it is not a leaf. If you want its field to be populated, you can call <code>y.retain_grad()</code> before the backward call.</li>
<li>Pytorch uses AD. Which, for multi-dimensional functions, only computes vector Jacobian product. In your first example, you function is R^2 -&gt;R^2 (EDIT: 2D -&gt; R2). So the Jacobian is a 2x2 matrix. You thus need to provide the vector by which it should be multiplied. Since your Jacobian here is diagonal, you can give [1., 1.] as you did to get that diagonal.</li>
<li>For more mathematical computations like this, I would recommend using the autograd.grad API as it is simpler for higher level derivatives and makes it easier to know exactely what you’re doing:</li>
</ul>
<pre><code class="lang-auto">x = torch.tensor([0.0, 1.0], requires_grad = True)
f = 2*x
grad = autograd.grd(outputs=f, inputs=x, grad_outputs=torch.tensor([1.0, 1.0]))[0]
print(grad)
</code></pre>

username: kuzand
cooked: <p>Hi albanD. Thank you for your reply.</p>
<aside class="quote no-group quote-modified" data-username="albanD" data-post="2" data-topic="73408">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>In your first example, you function is 2D → 2D. So the Jacobian is a 2x2 matrix. You thus need to provide the vector by which it should be multiplied. Since your Jacobian here is diagonal, you can give [1., 1.] as you did to get that diagonal.</p>
</blockquote>
</aside>
<p>Isn’t the <code>f = 2*x</code> a 1D tensor, since <code>x</code> is 1D? The Jacobian is then also 1D, i.e. <code>J = tensor([2, 2])</code>. So what is returned is <code>J*grad_outputs</code> where <code>grad_outputs=torch.tensor([1.0, 1.0])</code>, i.e. <code>tensor([2., 2.])</code>.<br>
For a 2D case, e.g. J being a 2x2 tensor, we would have <code>torch.mm(J, grad_outputs)</code> (instead of simple multiplication) where <code>grad_outputs=torch.tensor([[1.0, 1.0], [1.0, 1.0]])</code>. Am I right?</p>

username: albanD
cooked: <p>Sorry my comment was a bit misleading. It’s not 2D it’s 1D of size 2. So in R^2.<br>
In general, if your function is R^n -&gt; R^m, the Jacobian will be 2D of size mxn.</p>