Difference between detach().clone() and clone().detach()

isalirezag: <p>can someone explain to me the difference between <code>detach().clone()</code> and <code>clone().detach()</code> for a tensor<br>
<code>A = torch.rand(2,2)</code><br>
what is the difference between <code>A.detach().clone()</code> and  <code>A.clone().detach()</code><br>
are they equal?<br>
when i do detach it makes requres_grad false, and clone make a copy of it, but how the  two aforementioned method are different?</p>
<p>is there any of them preferred?</p>

justusschock: <p>If you first detach the tensor and then clone it, the computation path is not copied, the other way around it is copied and then abandoned. Other than that, the methods should be identical.</p>
<p>Thus I’d recommend to first detach and then copy because it does not make sense to copy anything that is never used.</p>

spacemeerkat: <p>I have a quick question about this…say you had an autoencoder like below that uses two learned parameters <code>A</code> and <code>B</code>.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a10523e756d1df77335b5e329008ee4b37affaeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/a10523e756d1df77335b5e329008ee4b37affaeb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a10523e756d1df77335b5e329008ee4b37affaeb_2_690x199.png" alt="image" data-base62-sha1="mYrSUwZT3ZszW87f1nN6P9JW6WD" width="690" height="199" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a10523e756d1df77335b5e329008ee4b37affaeb_2_690x199.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a10523e756d1df77335b5e329008ee4b37affaeb_2_1035x298.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/1/a10523e756d1df77335b5e329008ee4b37affaeb.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1134×328 16 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Say along the top decoder branch you don’t want to backprop over <code>B</code>, you only want to use it as a fixed input to <code>f(A,B)</code>…would you want to use <code>B.detach()</code> in place of <code>*?</code> or would that prevent the bottom branch from learning over <code>B</code> because you have completely detached it from the computational graph?</p>
<p>This is a circumstance where I feel like using <code>B.clone().detach()</code> would still allow learning over B in the lower branch and allow B as input to the top branch as a fixed input without <code>requires_grad</code>.</p>

justusschock: <p>I think doing <code>B.detach()</code> would be sufficient there.</p>
<pre><code class="lang-python">b_det = B.detach()
</code></pre>
<p>would create a new tensor that is detached from the previous computation graph.<br>
However, they still share the same memory and you can still use <code>B</code> for the lower part. You could maybe check some GAN examples, where this is also common, when you only want to train discriminators on the output of a generator.</p>

pinocchio: <aside class="quote no-group" data-username="justusschock" data-post="2" data-topic="34173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/48/4637_2.png" class="avatar"> justusschock:</div>
<blockquote>
<p>the other way around it is copied and then abandoned. Other than that, the methods should be identical.</p>
<p>Thus I’d recommend to first detach and then copy because it does not make sense to copy anything that is never used.</p>
</blockquote>
</aside>
<p>I am having trouble understanding this. I think the main issue I am having is because I do not understand <code>.clone()</code>'s semantics. The docs (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor — PyTorch 2.1 documentation</a>) say:</p>
<blockquote>
<p>Returns a copy of the  <code>self</code>  tensor. The copy has the same size and data type as  <code>self</code> .</p>
</blockquote>
<blockquote>
<p>NOTE: Unlike copy_(), this function is recorded in the computation graph. Gradients propagating to the cloned tensor will propagate to the original tensor.</p>
</blockquote>
<p>I don’t understand what the point of recording the clone in the graph. Like what is the difference with doing:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

a = torch.tensor([1,2,3.], requires_grad=True)
b = a # a copy of a, at least by reference in python
c = a.clone() # a copy of a by reference AND records operation in backward
d = torch.nn.Identity()(a) # same as c except it has different name for identity vs cloning
</code></pre>
<p>it seems to me that the only addition is that <code>clone()</code> records the operation in the graph, which seems really weird to. Why do we ever want to do this?</p>
<p>What is the difference between the above 4 operations? What is the use of clone? <a class="mention" href="/u/alband">@albanD</a></p>

albanD: <p>Hi,</p>
<p><code>b = a</code> in python actually gives you the same Tensor. So there is nothing to “record”.<br>
If you do a clone, you get a new Tensor (object and memory). And so if you want the backward to work, you need to have a way to propagate gradients from the new Tensor to the original one.</p>
<p><code>F.identity</code> I don’t think that exists.</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="34173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>F.identity</code> I don’t think that exists.</p>
</blockquote>
</aside>
<p>here is the corrected code:</p>
<pre><code class="lang-auto">d = torch.nn.Identity()(a)
</code></pre>
<p>I still don’t understand why <code>.clone()</code> exists. It seems useless.</p>
<p>Why would one want to record <code>.clone</code> if it’s identical to the identity?</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="34173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>b = a</code> in python actually gives you the same Tensor. So there is nothing to “record”.<br>
If you do a clone, you get a new Tensor (object and memory)</p>
</blockquote>
</aside>
<p>Ah! That’s useful, thanks! <code>a=b</code> are exactly the same while <code>c vs a</code> are not.</p>
<p>But the underlying memory the share is the same, right? So in effect it looks like they are truly the same except that python isn’t aware of it…right?</p>

albanD: <aside class="quote no-group" data-username="pinocchio" data-post="7" data-topic="34173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>I still don’t understand why <code>.clone()</code> exists. It seems useless.</p>
</blockquote>
</aside>
<p>Well identity is just a wrapper for clone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>But the underlying memory the share is the same, right? So in effect it looks like they are truly the same except that python isn’t aware of it…right?</p>
</blockquote>
<p>There are three cases here:</p>
<ul>
<li>Same Tensor object</li>
<li>Different Tensor object but looking at the same memory (a view)</li>
<li>Different Tensor object looking at different memory.</li>
</ul>
<p>For example:<br>
You get the first one when you do <code>new = base</code><br>
You can get the second one by doing <code>new = base.view_as(base)</code><br>
You can get the third one by doing <code>new = base.clone()</code></p>

pinocchio: <p>Just for me to understand what “different tensor objects mean”. Does that mean they are a different instance of the Tensor class and might also have additional meta-data that is different (for some reason)?</p>
<p>Trying to understand what different tensors really means.</p>
<p>Thanks for the help! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Yes you can see different Tensors as being different instances of the Tensor class.</p>

pinocchio: <p>Sorry for the spam Alban. I am just trying to understand the terminology clearly.</p>
<p>So “a new tensor” means a “new tensor instance/object” and not a new allocation of memory for a matrix/tensor data. Right?</p>
<p>Is “memory of a tensor” the same thing as its “view”? I think I read you saying “they have the same view” and it confused me. I always thought view was the same as shape but now I am unsure. Do you mind clarifying that for me?</p>
<p>I am referring to this:</p>
<blockquote>
<ul>
<li>Different Tensor object but looking at the same memory (a view)</li>
</ul>
</blockquote>
<p>And as always thanks so much for your help!</p>

albanD: <blockquote>
<p>So “a new tensor” means a “new tensor instance/object” and not a new allocation of memory for a matrix/tensor data. Right?</p>
</blockquote>
<p>Yes</p>
<p>You can see the Tensor as being two things (not the actual implementation):</p>
<ul>
<li>A class instance containing metadata</li>
<li>A raw pointer to the memory containing the data</li>
</ul>
<p>When I say that a Tensor is a view of another, I mean that they different instances of the Tensor class but they point to the same memory containing the data.</p>

pinocchio: <p>the real answer to this is here:</p>
<aside class="quote quote-modified" data-post="42" data-topic="16861">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/42">Clone and detach in v0.4.0</a> 
  </div>
  <blockquote>
    Sorry if this repetitive but I still don’t get it. What is wrong with doing clone first and then detach i.e. .clone().detach() ? 

Nothing. They will given an equivalent end result. 
The minor optimization of doing detach() first is that the clone operation won’t be tracked: if you do clone first, then the autograd info are created for the clone and after the detach, because they are inaccessible, they are deleted. So the end result is the same, but you do a bit more useless work. 
In any meani…
  </blockquote>
</aside>

<hr>
<p>Basically <code>.detach().clone()</code> is the one you should do and the difference is essentially negligible. Operationally they are essentially equivalent with the one I mentioned being slightly more efficient but probably negligible.</p>

pinocchio: <aside class="quote no-group" data-username="albanD" data-post="13" data-topic="34173">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>When I say that a Tensor is a view of another, I mean that they different instances of the Tensor class but they point to the same memory containing the data.</p>
</blockquote>
</aside>
<p>I see. So basically it seems that “view” means they are different tensor objects this with different meta-data (main example is different “shape”, in fact I always thought shape and view were the same) pointing to the <strong>same memory</strong>.</p>
<hr>
<p>useful link: <a href="https://pytorch.org/docs/stable/tensor_view.html" class="inline-onebox" rel="noopener nofollow ugc">Tensor Views — PyTorch 2.1 documentation</a></p>
<blockquote>
<p>TENSOR VIEWS</p>
</blockquote>
<blockquote>
<p>PyTorch allows a tensor to be a  <code>View</code>  of an existing tensor. View tensor shares the same underlying data with its base tensor. Supporting  <code>View</code>  avoids explicit data copy, thus allows us to do fast and memory efficient reshaping, slicing and element-wise operations.</p>
</blockquote>