Is there a way to check if the tensor is in the computation graph?

username: Fractale
cooked: <p>I want to test my tensor to see if I have to call <code>.detach()</code> on them. Any way of testing that?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can check if they have the <code>requires_grad</code> field to <code>True</code>. That would mean that some leaf tensor need gradients through them to compute their gradients.</p>

username: Fractale
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>requires_grad</code></p>
</blockquote>
</aside>
<p>Thank you so much! Why it’s not in the doc? <a href="https://pytorch.org/docs/stable/tensor_attributes.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Attributes — PyTorch 2.1 documentation</a></p>

username: albanD
cooked: <p>HI,</p>
<p>It is mentioned in the <a href="https://pytorch.org/docs/stable/tensors.html"><code>tensor</code></a> part.<br>
But indeed this should be added as well as <code>is_leaf</code> (that will tell you if a tensor is a leaf tensor).</p>

username: Fractale
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>is_leaf</code></p>
</blockquote>
</aside>
<p>from my understanding a leaf is a tensor witch don’t require gradiant.<br>
so <code>not requires_grad == is_leaf</code> is always true<br>
But I really not understand what is really a leaf.</p>
<p>to compute the gradiant you generate a tree? what are the node in the tree witch are not leaf?</p>

username: albanD
cooked: <p>A leaf tensor is a tensor with no parent in the graph. If it does not require grad, then it never has parents and is always a leaf, if it requires grad, it is a leaf only if it was created by the user (not the result of an op). When running the backward pass, the gradients are accumulated in all the leaf tensors that require gradients.</p>

username: pinocchio
cooked: <p><a class="mention" href="/u/alband">@albanD</a></p>
<p>related question. Why aren’t cloned vectors leaf nodes (specially if they are clones of leaf nodes):</p>
<pre><code class="lang-auto">import torch
​
w = torch.tensor([2.0], requires_grad=True)
w1 = torch.tensor([2.0], requires_grad=False)
w_clone = w.clone()
​
print(w)
print(w.is_leaf)
print(w1.is_leaf)
print(w_clone)
print(w_clone is w)
print(w_clone == w)
print(w_clone.is_leaf)
tensor([2.], requires_grad=True)
'''
output:
'''
True
True
tensor([2.], grad_fn=&lt;CloneBackward&gt;)
False
tensor([True])
False
</code></pre>
<p>what does it even mean to do a backward pass/differentiation of a cloned operation?</p>

username: albanD
cooked: <p>The backward of a clone is just a clone of the gradients.</p>
<p>All (almost) of pytorch operations are differentiable. And you can compute gradient through them.<br>
If you want to break the graph you should use <code>.detach()</code>.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>All (almost) of pytorch operations are differentiable.</p>
</blockquote>
</aside>
<p>Really? Including <code>.clone()</code>? I’ve seen it my graph and thought that was super strange (related question: <a href="https://discuss.pytorch.org/t/why-is-the-clone-operation-part-of-the-computation-graph-is-it-even-differentiable/67054" class="inline-onebox">Why is the clone operation part of the computation graph? Is it even differentiable?</a>).</p>
<pre><code class="lang-auto">#%%

## Q: 

import torch
from torchviz import make_dot

x = torch.randn(1, requires_grad=True)

temp = x**2
# y = temp.clone().detach()
y = temp.clone()
print(f'y.is_leaf= {y.is_leaf}')
print(f'y.requires_grad = {y.requires_grad}')
temp = temp+y

print(temp)

temp.backward()

print(f'x.grad = {x.grad}')
print(f'y.grad = {y.grad}')

make_dot(temp)
# make_dot(temp)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea2f7b056406b4404938e077882546c851b54c4d" title="Screen Shot 2020-01-18 at 3.36.46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_327x500.png" alt="Screen Shot 2020-01-18 at 3.36.46 PM" data-base62-sha1="xpHsbQTwXhMiOFqh5aOLSnY3flP" width="327" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_327x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_490x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_654x1000.png 2x" data-dominant-color="565959"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-01-18 at 3.36.46 PM</span><span class="informations">858×1310 74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>All (almost) of pytorch operations are differentiable.</p>
</blockquote>
</aside>
<p>I’m really curious! What are examples of an operations that are not differentiable?</p>

username: albanD
cooked: <p>Usually operations that return indices like <code>argmax</code>. Or any function that works with integer values (index is not differentiable wrt the indices for example, only the values).</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>if it requires grad, it is a leaf only if it was created by the user (not the result of an op).</p>
</blockquote>
</aside>
<p>I thought that was correct but I tried running some code and the result of exponentiation results in the dummy loss being a leaf, which I thought was super bizarre. Is there anything wrong with my code? shouldn’t <code>l_y</code> NOT be a leaf?</p>
<pre><code class="lang-auto">import torch
from torchviz import make_dot

lst = []
lst2 = lst

print('-- lst vs lst2')
print(lst is lst2)
lst.append(2)
print(f'lst={lst} vs lst2={lst2}')

x = torch.tensor([2.0],requires_grad=False)
xx = x

print('-- x vs xx')
print(f'xx == x = {xx == x}')
print(f'xx is x = {xx is x}')
xx.fill_(4.0)
print(f'x = {x}')
print(f'xx = {xx}')

x = torch.tensor([2.0],requires_grad=False)
y = x**2
l_y = (y - x + 3)**3 + 123

print('\n-- x vs y')
print(f'y = {y}')
print(f'y == x = {y==x}')
print(f'y is x = {y is x}')
print(f'y.is_leaf = {y.is_leaf}')
print(f'l_y.is_leaf = {l_y.is_leaf}')
y.fill_(3.0)
print(f'x = {x}')
print(f'y = {y}')
print(f'x.is_leaf = {x.is_leaf}')
print(f'y.is_leaf = {y.is_leaf}')
print(f'l_y.is_leaf = {l_y.is_leaf}')
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">
-- lst vs lst2
True
lst=[2] vs lst2=[2]
-- x vs xx
xx == x = tensor([True])
xx is x = True
x = tensor([4.])
xx = tensor([4.])

-- x vs y
y = tensor([4.])
y == x = tensor([False])
y is x = False
y.is_leaf = True
l_y.is_leaf = True
x = tensor([2.])
y = tensor([3.])
x.is_leaf = True
y.is_leaf = True
l_y.is_leaf = True
</code></pre>
<p>I doubt its a bug in pytorch but I can’t see what I did wrong…or is my expectation wrong?</p>

username: albanD
cooked: <p>A leaf is a Tensor with no gradient history. You need to keep in mind that Tensors that do <strong>not</strong> require gradients, since we don’t track their history, they are always leafs.<br>
So when you create <code>x = torch.tensor([2.0],requires_grad=False)</code>, it is a leaf because it has no history. But when you use it in <code>y = x**2</code>, since <code>x</code> does not require gradients, we do not track the operation, and <code>y</code> does not require gradient and has no history. So <code>y</code> is a leaf as well. Same thing for the computation of your loss.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="13" data-topic="28886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>A leaf is a Tensor with no gradient history.</p>
</blockquote>
</aside>
<p>Ah! I see. That makes so much sense.</p>
<p>I didn’t even realize I had it’s <code>requires_grad</code> flag set to false X’D</p>
<p>I confirmed what you said and its true, it was fixed after changing that flag:</p>
<pre><code class="lang-auto">print('-- what is a leaf?')
y_clone = y.clone()
print(f'y.is_leaf = {y_clone.is_leaf}')
x = torch.tensor([2.0],requires_grad=True)
lx = (x - 3)**2
print(f'x.is_leaf = {x.is_leaf}')
print(f'lx.is_leaf = {lx.is_leaf}')
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">-- is y_clone an leaf
y.is_leaf = True
x.is_leaf = True
lx.is_leaf = False

</code></pre>
<p>Thanks! Again your a savior <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>