Exclude a variable from a change through a computation

JanoschMenke: <p>Hi this is a beginner question and probably really quickly answered. But I could nto find it online, or probably to know the proper vocabulary to find what I am looking for.</p>
<p>I am trying to save the gradients into the variable “grads”, as in the code below.</p>
<pre><code class="lang-auto">grads=[0,0]
with torch.no_grad():
        for idx,p in enumerate(model.parameters()):
           
            #save current gradients
            grads[idx]=p.grad
            p.sub_(lr * p.grad)
            p.grad.zero_()
</code></pre>
<p>However, <code>p.grad.zero_</code> always not only sets the <code>p.grad</code> but also <code>grads</code> to zero.</p>
<p>I understand that it has todo with the way these two variables are connected through a graph. But how do I stop <code>p.grad.zero_</code> form setting also <code>grads</code> to zero.</p>
<p>I tried using <code>detach</code>, but that did not work.</p>
<pre><code class="lang-auto">grads=[0,0]
with torch.no_grad():
        for idx,p in enumerate(model.parameters()):
           
            #save current gradients
            grads[idx]=p.grad.detach()
            p.sub_(lr * p.grad)
            p.grad.zero_()
</code></pre>

JuanFMontesinos: <p>Both variables are pointing to the same memory address, you have to do a deep copy of p.grad in order not to be modified. You probably can do that by using the clone()  tensor method.</p>

JanoschMenke: <p>Hi, thanks for the quick reply. It worked! yey.</p>
<p>However, I dont understand why.</p>
<p>In this post<br>
the difference between clone and detach is explained as follows:</p><aside class="quote quote-modified" data-post="2" data-topic="16861">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/richard/48/1861_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/clone-and-detach-in-v0-4-0/16861/2">Clone and detach in v0.4.0</a> 
  </div>
  <blockquote>
    We’ll provide a migration guide when 0.4.0 is officially released. Here are the answers to your questions: 


tensor.detach() creates a tensor that shares storage with tensor that does not require grad. tensor.clone()creates a copy of tensor that imitates the original tensor's requires_grad field. 
You should use detach() when attempting to remove a tensor from a computation graph, and clone as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from. …
  </blockquote>
</aside>

<blockquote>
<p>You should use  <code>detach()</code>  when attempting to remove a tensor from a computation graph, and  <code>clone</code>  as a way to copy the tensor while still keeping the copy as a part of the computation graph it came from.</p>
</blockquote>
<p>Maybe I am confusing the computatoinal graph and the memory adress. In my case I would like to remoe the <code>grads</code> object form my graph or not?</p>

JuanFMontesinos: <p>Hi,<br>
As far as I understand you want to save gradients.<br>
That’s not really related to computational graphs or pytorch stuff rather than python’s.</p>
<p>When I told you to use clone on grads because gradients are tensors, thus, you can use tensor’s methods. The reason why is clone does a kind of deep copy of the tensor (You can google python’s deepcopy and shallow copy).</p>
<p>In short, when you append a tensor into a list, that element of the list keeps pointing to the original tensor, thus, if the original tensor is modified, the one in the list is modified too.<br>
(Example here copied from <a href="https://stackoverflow.com/questions/17873384/how-to-deep-copy-a-list" rel="nofollow noopener">https://stackoverflow.com/questions/17873384/how-to-deep-copy-a-list</a>)</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = [[1, 2, 3], [4, 5, 6]]
&gt;&gt;&gt; b = list(a)
&gt;&gt;&gt; a
[[1, 2, 3], [4, 5, 6]]
&gt;&gt;&gt; b
[[1, 2, 3], [4, 5, 6]]
&gt;&gt;&gt; a[0][1] = 10
&gt;&gt;&gt; a
[[1, 10, 3], [4, 5, 6]]
&gt;&gt;&gt; b   # b changes too -&gt; Not a deepcopy.
[[1, 10, 3], [4, 5, 6]]
</code></pre>
<p>About the computational graph it’s a really long story. In short, when you detach a tensor you are breaking the computational graph, whereas when you clone a tensor you are creating a “new branch” in the graph.</p>
<p>Feel free to keep asking</p>

JanoschMenke: <p>Oh okay thanks for the explanation. I just recently switched to python and did not know about that functionality of python and thought it was something unquie to the comp. graph of pytorch.</p>
<p>I will read into  it and might get back if more questions arise.</p>