Why is the clone operation part of the computation graph? Is it even differentiable?

username: pinocchio
cooked: <p>I saw that <code>.clone()</code> was part of the computation graph and thought it was really weird. Why is that?</p>
<p>I wanted to know</p>
<ol>
<li>what it even meant to take derivatives with respect to such an operation (I thought it shouldn’t be defined)</li>
<li>Why is it part of the computation graph if one can’t take derivatives wrt to clone</li>
</ol>
<p>Look its in the graph?!</p>
<pre><code class="lang-auto">#%%

## Q: 

import torch
from torchviz import make_dot

x = torch.randn(1, requires_grad=True)

temp = x**2
# y = temp.clone().detach()
y = temp.clone()
print(f'y.is_leaf= {y.is_leaf}')
print(f'y.requires_grad = {y.requires_grad}')
temp = temp+y

print(temp)

temp.backward()

print(f'x.grad = {x.grad}')
print(f'y.grad = {y.grad}')

make_dot(temp)
# make_dot(temp)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d.png" data-download-href="https://discuss.pytorch.org/uploads/default/ea2f7b056406b4404938e077882546c851b54c4d" title="Screen Shot 2020-01-18 at 3.36.46 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_327x500.png" alt="Screen Shot 2020-01-18 at 3.36.46 PM" data-base62-sha1="xpHsbQTwXhMiOFqh5aOLSnY3flP" width="327" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_327x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_490x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/ea2f7b056406b4404938e077882546c851b54c4d_2_654x1000.png 2x" data-dominant-color="565959"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-01-18 at 3.36.46 PM</span><span class="informations">858×1310 74 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>The clone function is like an identity. f(x) = x, So you can take its derivative.<br>
You were expecting something else?</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="67054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>The clone function is like an identity. f(x) = x, So you can take its derivative.<br>
You were expecting something else?</p>
</blockquote>
</aside>
<p>I guess because we are creating an actual tensor (i.e. we are using a clone operation) I expected the new vector to be a leaf (or something “new”). Thus it seemed odd to me that it would be considered the identity function, because those two operation don’t seem the same to me…but I guess clone is just the identity? Whats the point of clone then?</p>

username: albanD
cooked: <p>Clone is an identity with new memory. The same way you could use <code>new = t.view_as(t)</code> would be an identity with the exact same memory.<br>
The main use of clone is to be able to do inplace operations on the result without impacting the original Tensor.</p>
<p>Note that in pytorch, all functions will be differentiated and give you gradients. The only exception is <code>.detach()</code> that is defined as setting all gradients to 0 and the ops inside a <code>torch.no_grad()</code> block that are not tracked.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="67054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Note that in pytorch, all functions will be differentiated and give you gradients. The only exception is <code>.detach()</code> that is defined as setting all gradients to 0 and the ops inside a <code>torch.no_grad()</code> block that are not tracked.</p>
</blockquote>
</aside>
<p>So is <code>.detach()</code> the correct way to create new “leaf nodes”?</p>
<p>Btw, thank for the help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: albanD
cooked: <p>Yes. If you want to have a new Tensor that has no gradient history, you should use detach().<br>
Note that the result of detach() uses the same memory space as the original Tensor. So if you plan on modifying it inplace, you want to do <code>.clone().detach()</code>.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="67054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>if you plan on modifying it inplace, you want to do <code>.clone().detach()</code> .</p>
</blockquote>
</aside>
<p>I plan to create a completely separate computation graph (and don’t want the call of backwards to interfere with each other). For that I am running <code>requires_flag=True</code> immediately after <code>.detach()</code>. Is that the right thing to do?</p>
<p>e.g.</p>
<pre><code class="lang-auto">        ##
        wt_new = wt.clone().detach()
        wt_new.requires_grad = True
        l = (wt_new*x - y)**2
        print(f'l = {l}')
        l.backward()
</code></pre>

username: albanD
cooked: <p>If you want a different graph, you can use <code>.detach()</code>.<br>
In your example, since you don’t modify <code>wt_new</code> inplace, you don’t need the <code>.clone()</code>.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="8" data-topic="67054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>In your example, since you don’t modify <code>wt_new</code> inplace, you don’t need the <code>.clone()</code> .</p>
</blockquote>
</aside>
<p>However, the original <code>wt</code> is going to collect gradients with respect to the original graph. So if I call <code>.detach()</code> wouldn’t it collect the gradients for <strong>both</strong> graphs in the same tensor? That’s definitively not what I want. I want separate gradients (or at least that’s my rationale for calling <code>.clone()</code> first and then <code>.detach()</code>). What are your thoughts master albanD?</p>

username: albanD
cooked: <p>In pytorch, it is different to have different Tensor and have different memory.<br>
When you do <code>b = a.detach()</code>, a and b are two completely different Tensors that look at the same memory.<br>
Just like <code>b = a.view(-1)</code> are two different Tensors that look at the same memory.</p>
<p>The inplace version (that would modify the Tensor inplace) is <code>a.detach_()</code> or <code>b = a.detach_()</code>. If you do this, then <code>a</code> and <code>b</code> actually point to the exact same python object (check that <code>id(a) == id(b)</code>) and same Tensor.</p>

username: pinocchio
cooked: <aside class="quote no-group" data-username="albanD" data-post="10" data-topic="67054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>In pytorch, it is different to have different Tensor and have different memory.<br>
When you do <code>b = a.detach()</code> , a and b are two completely different Tensors that look at the same memory.<br>
Just like <code>b = a.view(-1)</code> are two different Tensors that look at the same memory.</p>
</blockquote>
</aside>
<p>That was really helpful. Thank you.</p>
<p>I guess to answer my own question, I do not need to call clone it seems. The new tensor tracks its own memory space for its gradients automagically (without interference, so it doesn’t collect the same gradients for both graphs in the same place):</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([2.0], requires_grad=True)
b = a.detach()
b.requires_grad = True

la = (5.0 - a)**2
la.backward()
print(f'a.grad = {a.grad}')

lb = (6.0 - b)**2
lb.backward()
print(f'b.grad = {b.grad}')
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">a.grad = tensor([-6.])
b.grad = tensor([-8.])
</code></pre>
<p>of course you guys would have thought of a good implementation of this! Not surprised! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: pinocchio
cooked: <p>so is there ever a case one needs <code>.clone()</code>?</p>
<p>I can’t think of one except making 2 variables pointing to different memories and then doing inplace ops separately for each one or something like that…which seems dangerous…</p>

username: albanD
cooked: <p>There are a few.<br>
For example if you want to save the current state of the weights of your net, you want to clone because the optimizer update works inplace and so your save will change with your network if you don’t clone.</p>

username: pinocchio
cooked: <p>My earlier post checked that two tensors really are detached by comparing gradients. Is there an internal flag to check something like this? A nicer way to check that <code>b</code> is detached?</p>

username: albanD
cooked: <p>detached from what?<br>
If you just created <code>b</code>, you can check by checking if it requires gradients or not.</p>

username: pinocchio
cooked: <p>that’s not enough because I am setting the requires gradients true myself later. See sample script (where in it I check they are actually detached and form a separate graph by computing gradients I manually know how to check, but thats harder to check in a complicated net however):</p>
<pre><code class="lang-auto">import torch

a = torch.tensor([2.0], requires_grad=True)
b = a.detach()
b.requires_grad = True

la = (5.0 - a)**2
la.backward()
print(f'a.grad = {a.grad}')

lb = (6.0 - b)**2
lb.backward()
print(f'b.grad = {b.grad}')
</code></pre>
<p>result:</p>
<pre><code class="lang-auto">a.grad = tensor([-6.])
b.grad = tensor([-8.])</code></pre>