Gradient of a function of non-element-wise operation

username: oat
cooked: <p>In the <a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="noopener nofollow ugc">PyTorch official intro on autograd</a>, Q is a vector output of element-wise operation of a function on two vectors a and b.</p>
<p>How to calculate the gradients of a and b if the function is not element-wise operation, i.e. each element in the vector Q is obtained by different calculations on different elements of a and b, e.g. Q[0] = 3<em>a[0]**3 - b[0]**2, and Q[1] = a[1]**2 - 3</em>b[1]**3 ?</p>
<p>I did the following test with one vector input x and one vector output y. However, the gradient of x is not able to be calculated.</p>
<pre><code class="lang-auto">import torch 

x = torch.tensor([1., 2.], requires_grad=True)

y1 = 2*x[0]**2 + x[1]
y2 = 3*x[0] + 4*x[1]**3
y = torch.tensor([y1, y2])

external_grad = torch.tensor([1., 1.])
y.backward(gradient=external_grad)

x.grad
</code></pre>
<p>Error:<br>
<code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code></p>

username: AlphaBetaGamma96
cooked: <p>Hey,</p>
<p>I have a feeling this issue arises with <code>y = torch.tensor([y1, y2])</code>, as it takes 2 floats and puts them within a Tensor. I quickly ran your code but with <code>torch.stack</code> and the gradients are computed!</p>
<pre><code class="lang-auto">import torch 

x = torch.tensor([1., 2.], requires_grad=True)

y1 = 2*x[0]**2 + x[1]
y2 = 3*x[0] + 4*x[1]**3
y = torch.stack([y1, y2])

y.backward(torch.ones_like(y))

print(x.grad) #returns tensor([ 7., 49.])
</code></pre>

username: oat
cooked: <p>Thank you very much, <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a></p>
<p>May I ask why <strong><code>y_a = torch.tensor([y1, y2])</code></strong> didn’t create a torch tensor with a <strong><code>grad_fn</code></strong> attached, although it has the same <em><strong>shape</strong></em> and <em><strong>type</strong></em> as the one created by <em><strong>stacking</strong></em> y1 and y2 as you suggested, i,e. <strong><code>y_b = torch.stack([y1, y2])</code></strong> as shown below?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/8/d85fcdc3818e01a2068866afd7eb4e2efacec0b4.png" alt="Image 54" data-base62-sha1="uS8ojKJR562l2yOBSJOrqy9QBTu" width="360" height="478"></p>

username: AlphaBetaGamma96
cooked: <p>I’m not 100% sure (it’d be best to get a dev’s opinion) but this is an educated guess, <code>torch.Tensor</code> takes in a list of floats and converts it into a pytorch array (called Tensor). So, I would assume they don’t implemented a <code>grad_fn</code> for it as what would be the gradient of putting elements in an array?</p>

username: albanD
cooked: <p>Yes this is exactly the right answer!  This is a factory function to create Tensors based on numbers. So no gradient can flow back. You can use cat/stack to build a bigger Tensor based on smaller ones in a differentiable way.</p>

username: AlphaBetaGamma96
cooked: <p>Looks like I’m learning some PyTorch afterall! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"> Thanks <a class="mention" href="/u/alband">@albanD</a>!</p>