Copy.deepcopy() vs clone()

Shisho_Sama: <p>when copying modules/tensors around, which one should I use?<br>
are they interchangable?<br>
Thanks a lot</p>

spanev: <p>Hi <a class="mention" href="/u/shisho_sama">@Shisho_Sama</a>,</p>
<p>For <code>Tensor</code>s in most cases, you should go for <code>clone</code> since this is a PyTorch operation that will be recorded by autograd.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.rand(1, requires_grad=True)
&gt;&gt;&gt; t.clone()
tensor([0.4847], grad_fn=&lt;CloneBackward&gt;) # &lt;=== as you can see here
</code></pre>
<p>When it comes to <code>Module</code>, there is no <code>clone</code> method available so you can either use <code>copy.deepcopy</code> or create a new instance of the model and just copy the parameters, as proposed in this post <a href="https://discuss.pytorch.org/t/deep-copying-pytorch-modules/13514/2" class="inline-onebox">Deep copying PyTorch modules</a>.</p>

Shisho_Sama: <p>Hi, Thanks a lot.<br>
So this means, when I do clone() on  tensors, their clones will still be on the graph and any operations on them will be reflected in the graph right? for example changing the values or attributes will also change the original tensor as well? or affect the graph computation when doing backward pass?<br>
In the case of the following two code snippets, what does happen in each case?<br>
I think, deepcopy disregards any graph related information and just copies  the data<br>
as if it is a simple object while the clone, will create a new tensor which any operations on it will be reflected in the graph and in order to prevent this I need to use detach as well. am I right?</p>
<pre><code class="lang-auto">weights_encoder = sae_model.encoder[0].weight.data.clone() 
weights_decoder = sae_model.decoder[0].weight.data.clone()
</code></pre>
<p>or</p>
<pre><code class="lang-auto">weights_encoder = copy.deepcopy(sae_model.encoder[0].weight.data) 
weights_decoder = copy.deepcopy(sae_model.decoder[0].weight.data)
</code></pre>

spanev: <p>When you use <code>.data</code>, you get a new <code>Tensor</code> with <code>requires_grad=False</code>, so cloning it won’t involve autograd. So both are equivalent, but there might be a (small) speed difference, I am not sure about that.</p>
<p>Another use case could is when you want to clone/copy a non-parameter <code>Tensor</code> without autograd. You should use <code>.detach()</code> (and <a href="https://github.com/pytorch/pytorch/issues/6990#issuecomment-384680164" rel="nofollow noopener">not data</a>) before cloning:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; t = torch.rand(1, requires_grad=True)
&gt;&gt;&gt; t.detach().clone()
tensor([0.4847])
</code></pre>

Shisho_Sama: <p>Thank you very much. I really appreciate it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

dio_din: <p>Is there any difference with <code>t.clone().detach()</code>?</p>

bobvo23: <p>Yes there is. Though both methods create same outcomes, however, t.clone().detach() is less efficient. The t.clone() with create a copy that attaches to the graph, then it will create another copy (detach()). So there will be more redundant.</p>

pinocchio: <aside class="quote no-group" data-username="spanev" data-post="2" data-topic="55022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/spanev/48/14892_2.png" class="avatar"> spanev:</div>
<blockquote>
<p>For <code>Tensor</code> s in most cases, you should go for <code>clone</code> since this is a PyTorch operation that will be recorded by autograd.</p>
</blockquote>
</aside>
<p>I never understood this. Why would one ever want to have clone be in the computation graph? It’s just the identity!</p>
<p>When I make a copy of something I usually expect a brand new object, with new memory allocation and new instance of the object class it belongs. Not just copying pointers/references around. Can you clarify?</p>

ptrblck: <p>Answered <a href="https://discuss.pytorch.org/t/confusion-about-using-clone/39673/12">here</a> and <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/21">here</a>.</p>

pinocchio: <p>Let me see if I understand (it seems the accepted answer here is outdated, <code>.data</code> is not in the library or going to be removed according to what I’ve read in other answers with from albanD).</p>
<p><code>.clone()</code> produces a new tensor instance with a new memory allocation to the tensor data. In addition it remembers the history of the original tensor and is connected to the earlier graph and appears as <code>CloenBackward</code>. The main advantage it seems is that its safer wrt in-place ops afaik.<br>
<code>deepcopy</code> make a deep copy of the original tensor meaning it creates a new tensor instance with a new memory allocation to the tensor data (it definitively does this part correctly from my tests). I assume it also does a complete copy of the history too, either pointing to the old history or create a brand new deep copy history. I’m unsure how to test this but I believe if it is to behave as a proper deep copy method then it should create a new history that is a mirror of the earlier (instead of just pointing to it).</p>
<p>Test I did wrt memory allocation:</p>
<pre><code class="lang-auto">def clone_vs_deepcopy():
    import copy
    import torch

    x = torch.tensor([1,2,3.])
    x_clone = x.clone()
    x_deep_copy = copy.deepcopy(x)
    #
    x.mul_(-1)
    print(f'x = {x}')
    print(f'x_clone = {x_clone}')
    print(f'x_deep_copy = {x_deep_copy}')
</code></pre>
<p>output</p>
<pre><code class="lang-auto">x = tensor([-1., -2., -3.])
x_clone = tensor([1., 2., 3.])
x_deep_copy = tensor([1., 2., 3.])
</code></pre>
<p>since neither changed it must be a different memory. I just realized I could have checked it with <code>id</code> or something…alas.</p>
<p>I am still seeking clarification on the history part. Is it a deep copy of that or a pointer copy if we use deep copy?</p>
<p>I know for know for clone it is a pointer copy to the original history and not a complete deep copy.</p>
<hr>
<p>related:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/62437509/what-is-the-difference-between-detach-clone-and-deepcopy-in-pytorch-tensors-in" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    <img alt="Pinocchio" src="https://www.gravatar.com/avatar/ba92064b084638cb903cbd271120be64?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/62437509/what-is-the-difference-between-detach-clone-and-deepcopy-in-pytorch-tensors-in" target="_blank" rel="nofollow noopener">What is the difference between detach, clone and deepcopy in Pytorch tensors in detail?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/3167448/pinocchio" target="_blank" rel="nofollow noopener">
    Pinocchio
  </a>
  on <a href="https://stackoverflow.com/questions/62437509/what-is-the-difference-between-detach-clone-and-deepcopy-in-pytorch-tensors-in" target="_blank" rel="nofollow noopener">08:06PM - 17 Jun 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <aside class="quote no-group" data-username="pinocchio" data-post="10" data-topic="55022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>I am still seeking clarification on the history part. Is it a deep copy of that or a pointer copy if we use deep copy?</p>
</blockquote>
</aside>
<p>The history will not be copied, as you cannot call <code>copy.deepcopy</code> on a non-leaf tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, requires_grad=True)
y = x + 1
copy.deepcopy(y)
&gt; RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
<p>While <code>y</code> will be attached to the computation graph and will have a valid <code>.grad_fn</code>, you can only copy leaves as stated in the error message.</p>
<p>If you want to keep the history, use <code>.clone()</code>, otherwise <code>.detach()</code> the tensor additionally to the <code>clone()</code> call.</p>

pinocchio: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="55022">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">&gt; RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
</code></pre>
</blockquote>
</aside>
<p>I always wondered why that error appeared!</p>
<p>Whats the choice for that semantics?</p>

ptrblck: <p>I don’t know, why the <code>deepcopy</code> isn’t supported (also wasn’t supported on <code>Variables</code>) and my best guess is that <code>clone()</code> or <code>detach().clone()</code> are a valid workaround and are also more explicit.</p>

pinocchio: <p>who would know why it’s not supported?</p>

jchiu: <p>Hi! This thread has been extremely useful but I ran into an issue with using <code>copy.deepcopy()</code> on a model  inherited from <code>pl.LightningModule</code> where <code>"_trainer"</code> attribute was not <code>None</code>. It would throw the following error:</p>
<p><code>AttributeError: 'MyModel' object has no attribute '_parameters'</code></p>
<p>I noticed that setting it to <code>None</code>, running <code>copy.deepcopy()</code> and then resetting it would solve this issue. Is this an issue caused by PyTorch Lightning?</p>

ptrblck: <p>This issue might be specific to Lightning and seems to be related to <a href="https://github.com/Lightning-AI/lightning/issues/7505#issuecomment-1339718787">this one</a>.</p>

jchiu: <p>Got it. Will follow up on this discussion in the github issue thread! Thanks for your help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>