Clone and detach in v0.4.0

nsndimt: <p>I’m currently migrating my old code from v0.3.1 to v0.4.0. During migration, I feel confused by the document about clone and detach. After searching related topics in the forum,  I find that most discussions are too old.  Specifically, I want an answer to the three following questions:</p>
<ol>
<li>the difference between tensor.clone() and tensor.detach() in v0.4.0?</li>
<li>the difference between tensor and tensor.data in v0.4.0?</li>
<li>the difference between tensor and tensor.data when calling clone() or detach() in v0.4.0?<br>
And an additional question which will help me understand how my old code work and what is those old topic talking:</li>
<li>the difference between tensor.clone(), variable.clone() and variable.detach() in v0.3.1?</li>
</ol>

richard: <p>We’ll provide a migration guide when 0.4.0 is officially released. Here are the answers to your questions:</p>
<ol>
<li>
<p><code>tensor.detach()</code> creates a tensor that shares storage with <code>tensor</code> that does not require grad. <code>tensor.clone()</code>creates a copy of tensor that imitates the original <code>tensor</code>'s <code>requires_grad</code> field.<br>
You should use <code>detach()</code> when attempting to remove a tensor from a computation graph, and <code>clone</code> as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from.</p>
</li>
<li>
<p><code>tensor.data</code> returns a new tensor that shares storage with <code>tensor</code>. However, it always has <code>requires_grad=False</code> (even if the original <code>tensor</code> had <code>requires_grad=True</code></p>
</li>
<li>
<p>You should try not to call <code>tensor.data</code> in 0.4.0. What are your use cases for <code>tensor.data</code>?</p>
</li>
<li>
<p><code>tensor.clone()</code> makes a copy of <code>tensor</code>. <code>variable.clone()</code> and <code>variable.detach()</code> in 0.3.1 act the same as <code>tensor.clone()</code> and <code>tensor.detach()</code> in 0.4.0.</p>
</li>
</ol>

nsndimt: <p>Thank you so much for such detailed explanation.<br>
In my code, I use clone to do two things:</p>
<ol>
<li>save model parameter whenever validation set performance reaches a new peak and restore them after several training rounds whose validation set performances stop improving(implement early stop). here’s my code:</li>
</ol>
<pre><code class="lang-auto">#because my parameter size is small compared with GPU memory I don’t save it to disk
best_params = dict()
if map_score &gt; best_map:
    for name, tensor in model.state_dict(keep_vars=True).items():
        #a large part of my parameter is embedding which is freeze during training so I don’t save it
        if tensor.requires_grad:
            #use tensor.data to empty its grad_fn then clone a new one to avoid being changed during training
            best_params[name] = tensor.data.clone()
            #if i understand correctly tensor.clone() will be ok too 
            #because no computation is performed with best parameters and no backward() is called. 
            #In such conditions having an none-empty grad_fn won’t cause any problem
model.load_state_dict(best_params, strict=False)
</code></pre>
<ol start="2">
<li>save model output when test model on the validation set. I divide validation task into several batches, save each batch’s ouput and concatenate them on GPU before transferring back to CPU here’s my code:</li>
</ol>
<pre><code class="lang-auto">save_res = list()
for test_data in task:
    score_res = model(‘test’, test_data)
    #use tensor.data to empty its grad_fn then clone a new one to avoid being free
    #use tensor.data is essential because requires_grad need to be False when calling .numpy()
    save_res.append(score_res.data.clone())
    #i didn't use score_res.detach() because it will be deleted later
    #i think just score_res.detach() and no del score_res result in keeping graph in GPU memory
    del score_res
save_res = torch.cat(save_res).cpu().numpy()
do_some_eval(save_res)
</code></pre>

kush8singh: <p>hi,<br>
i have read many forums trying to understand <code>.detach()</code> , and although i get a intutution of what it is , i still don’t understand it completely . I dont understand what removing a tensor from computational graph implies , for ex. consider piece of code below</p>
<pre><code class="lang-auto">x = torch.tensor(([1]))
y = x**2
z = 2*y
w= z**3
z = z.detach()

x.grad.zero_()
w.backward()
print x.grad

</code></pre>
<p>The output came out to be <code>48</code> which is the same with or without <code>z.detach()</code> . What does `z.detach() do here ? Why wasn’t it removed from the computational graph?</p>
<p>thanks</p>

Naruto-Sasuke: <p>detach() does not effect the origin graph. Only if you want to get non-leaf variable, eg, <code>z</code>,  and perform operations on it without effecting the origin gradients( Here are gradients w.r.t <code>x</code>,<code>y</code> and <code>z</code>), you need add <code>detach()</code>.</p>
<p>Eg.</p>
<pre><code class="lang-python">x = torch.tensor(([1]),requires_grad=True)
y = x**2
z = 2*y
w= z**3

# This is the subpath
# Do not use detach()
p = z
q = torch.tensor(([2]), requires_grad=True)
pq = p*q
pq.backward(retain_graph=True)

w.backward()
print x.grad
</code></pre>
<p>The gradient will be accumulated in <code>z</code>, and it gives result of ‘56’.<br>
However, when you try:</p>
<pre><code class="lang-auto">x = torch.tensor(([1]),requires_grad=True)
y = x**2
z = 2*y
w= z**3

# detach it, so the gradient w.r.t `p` does not effect `z`!
p = z.detach()
q = torch.tensor(([2]), requires_grad=True)
pq = p*q
pq.backward(retain_graph=True)

w.backward()
print x.grad
</code></pre>
<p>The gradient is still <code>48</code>. So <code>detach()</code> makes the gradient flow in the subpath no harm to the main path.</p>

kush8singh: <p>Thanks for the quick reply…that cleared things up</p>

yxchng: <p>Just curious, how to clone Variable without its history in 0.4 (meaning clone and make it a leaf variable)?</p>
<p>Is it <code>x_clone = torch.tensor(x.detach(), requires_grad=True)</code>? Or is there a better way?</p>

albanD: <p>Hi,</p>
<p><code>x_clone = x.detach().clone()</code> will do the job.</p>

acgtyrant: <p><a class="mention" href="/u/alband">@albanD</a>  Is <code>sourceTensor.clone().detach()</code> better than <code>sourceTensor.detach().clone()</code>? Why?</p>

acgtyrant: <p>I tried to copy from the tensor to a new leaf variable, like <code>b = torch.tensor(a.datach(), requires_grad=True)</code>, however, the REPL adviced me:</p>
<pre><code class="lang-auto">/home/acgtyrant/.local/bin/ipython:1: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
</code></pre>
<p>But I think <code>sourceTensor.datach().clone().requires_grad_(True)</code> will be better than <code>sourceTensor.clone().requires_grad_(True)</code> while any function like <code>clone</code> or <code>cuda</code> will create some subgraph before call <code>detach</code>. So detach first, then call some functions which do not have any side effect, then call requires_grad_(True) to create a pure leaf variable finally, like <code>sourceTensor.detach().index_select(0).clone().cuda(1).requires_grad_(True)</code>.</p>

albanD: <p>Hi,</p>
<p>The <code>requires_grad_</code> call should be last of you do that in one line to make sure you get the leaf tensor.<br>
Otherwise, the difference between all of these is going to be almost impossible to see !</p>
<p>By the way calling <code>.cuda()</code> on a cpu tensor will copy it, so no need to clone before. You can have a check and clone if it’s already on the correct device, otherwise just call <code>.cuda(1)</code> without the clone.</p>

kekmodel: <p>But why is “data” still used in the source?</p>

albanD: <p>Because we did not had time to modify all the existing code.<br>
The use of <code>.data</code> in the pytorch codebase are safe and so not critical to be changed but for consistency.</p>

pinocchio: <aside class="quote no-group" data-username="richard" data-post="2" data-topic="16861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p>You should try not to call <code>tensor.data</code> in 0.4.0.</p>
</blockquote>
</aside>
<p>Why should we not call <code>.data</code>?</p>

albanD: <p>It is going to go away soon ! It’s not even in the doc anymore !<br>
You can replace it with <code>.detach()</code> or <code>with torch.no_grad()</code> depending what you want to do.</p>

pinocchio: <aside class="quote no-group" data-username="richard" data-post="2" data-topic="16861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p>You should use <code>detach()</code> when attempting to remove a tensor from a computation graph</p>
</blockquote>
</aside>
<p>Does this mean that it also removes the flow of gradients down it’s path? i.e. removes an edge from the graph and thus a path for backprop to flow through?</p>

ziyad_hamada: <p>It is a temporary restriction to prevent people from relying on impls for large arrays on a stable compiler just yet.</p>
<p>In the rare chance that we need to give up on const generics and rip it out of the compiler, it avoids getting into a situation where users are already relying on e.g.  <code>IntoIterator for [T; 262143]</code>  and we have no way to support them.</p>
<p>When const generics are finally stabilized the restriction would be removed and the impl would apply to arrays of arbitrarily large size.</p>

pinocchio: <p>My apologies ziyad, but I am having a hard time following what you are responding to. Are you responding to my question about gradient flows?</p>

pinocchio: <aside class="quote no-group" data-username="pinocchio" data-post="16" data-topic="16861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar"> pinocchio:</div>
<blockquote>
<p>Does this mean that it also removes the flow of gradients down it’s path? i.e. removes an edge from the graph and thus a path for backprop to flow through?</p>
</blockquote>
</aside>
<p>Yes, compare printed graphs:</p>
<pre><code class="lang-auto"># http://www.bnikolic.co.uk/blog/pytorch-detach.html

import torch
from torchviz import make_dot

x=torch.ones(10, requires_grad=True)
weights = {'x':x}

y=((x**2)**3)
y=((x**2).detach()**3)
y=((x**2)**3).detach()

z=(x+3)+4
r=(y+z).sum()

make_dot(r,params=weights)
</code></pre>

pinocchio: <aside class="quote no-group" data-username="richard" data-post="2" data-topic="16861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar"> Richard Zou:</div>
<blockquote>
<p><code>tensor.detach()</code> creates a tensor that shares storage with <code>tensor</code> that does not require grad.</p>
</blockquote>
</aside>
<p>I am trying to understand what “shares storage means”. Does that mean if I detach a tensor from it’s original but then <code>opt.step()</code> to modify the original, then <strong>Both</strong> would change?</p>
<p>i.e.</p>
<pre><code class="lang-auto">a = torch.tensor([1,2,3.], requires_grad=True)
b = a.detach()
opt = optim.Adam(a,lr=0.01)
a.backward()
opt.step() # changes both because they share storage?
</code></pre>
<p>So both change because they share storage? That seems an odd semantics to have or I am missing the point of <code>.detach()</code>.</p>
<hr>
<p>here is the migration guide Richard mentioned:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/" target="_blank" rel="noopener nofollow ugc">PyTorch 0.4.0 Migration Guide</a></h3>

  <p>Welcome to the migration guide for PyTorch 0.4.0. In this release we introduced many exciting new features and critical bug fixes, with the goal of providing users a better and cleaner interface. In this guide, we will cover the most important...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<hr>
<p>Related useful links:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/55266154/pytorch-preferred-way-to-copy-a-tensor" class="inline-onebox" rel="noopener nofollow ugc">python - PyTorch preferred way to copy a tensor - Stack Overflow</a></li>
<li><a href="https://discuss.pytorch.org/t/difference-between-detach-clone-and-clone-detach/34173" class="inline-onebox">Difference between detach().clone() and clone().detach()</a></li>
</ul>