Autograd problem with tensors generated by torch.arange

Andrea_A: <p>I want to compute the gradient of a function in several points. However, If I use tensors generated with <code>torch.arange</code> the gradient is not computed (the .grad attributes are None). Instead, using classical tensors it works. Why?</p>
<p>In the following an example</p>
<pre><code class="lang-auto">    import torch
    from torch import tensor
    
    def l(w_1,w_2):
        return w_1*w_2
    
    w_1 = tensor(3., requires_grad=True)
    w_2 = tensor(5., requires_grad=True)
    l_v = l(w_1, w_2)
    l_v.backward()
    print(l_v.item(), w_1.grad, w_2.grad) # HERE WORKS OK
    #############
    for w_1_value  in torch.arange(+2,+4,0.1, requires_grad=True):
        for w_2_value  in torch.arange(-2,+4,0.1, requires_grad=True):
            print(w_1_value, w_2_value)
            l_value = l(w_1_value, w_2_value)
            l_value.backward()
            print(l_value.item(), w_1_value.grad, w_2_value.grad) # HERE I GET NONE ON GRAD VALUES
</code></pre>

ptrblck: <p>The <code>for</code> loop creates non-leaf variables with a valid <code>grad_fn</code> seen as <code>UnbindBackward0</code>.<br>
Create leaf variables inside the loop and it will work:</p>
<pre data-code-wrap="python"><code class="lang-python">        w_1_value = w_1_value.detach().requires_grad_()
        w_2_value = w_2_value.detach().requires_grad_()
</code></pre>
<p>You are also seeing a warning explaining it:</p>
<pre data-code-wrap="python"><code class="lang-python">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. 
</code></pre>

Andrea_A: <p>Thank you for your answer, but I still don’t understand.  What is UnbindBackward and why  is it a problem when connected to my custom function graph?<br>
Thanks again for your help</p>

ptrblck: <p>The <code>for</code> loop performs an <code>unbind</code> operations in the same way as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.arange(+2,+4,0.1, requires_grad=True)
a, *b = x
print(a)
# tensor(2., grad_fn=&lt;UnbindBackward0&gt;)
</code></pre>
<p>This operation is differentiable and thus the returned tensors are non-leaf tensors:</p>
<pre data-code-wrap="python"><code class="lang-python">print(a.is_leaf)
# False
</code></pre>

Andrea_A: <p>ok, thanks! In the meanwhile, I made the same question to stackoverflow, but the answer provided seems different (<a href="https://stackoverflow.com/questions/78279924/pytorch-autograd-with-tensors-generated-by-arange/78280921#78280921" class="inline-onebox" rel="noopener nofollow ugc">python - pytorch: autograd with tensors generated by arange - Stack Overflow</a>) . Is it correct for you?</p>

ptrblck: <p>The answer is pointing towards the same issue.</p>
<blockquote>
<p>When you iterate over the items in the <code>torch.arange</code> tensors, you are getting a view of the tensor, which doesn’t have the grad.</p>
</blockquote>
<p>Yes, you could also call it a <code>view</code> even if technically it’s an <code>unbind</code> operation. The root cause is still the same: the returned tensors in the <code>for</code> loop are non-leaf variables.</p>