UserWarning pops up only in Debug mode? The .grad attribute of a Tensor that is not a leaf Tensor is being accessed

username: EricTian
cooked: <p>Hi there, I’m learning pytorch (with VS Code) and facing an odd warning.</p>
<p>If I run my codes without breakpoints, it works just fine. However, when I set some breakpoints and run in debug mode, <strong>SOMETIMES</strong> this userwaring pops up:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61f355cd730847c047c10da38f31c0bc9dc1147b.png" data-download-href="https://discuss.pytorch.org/uploads/default/61f355cd730847c047c10da38f31c0bc9dc1147b" title="warning"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61f355cd730847c047c10da38f31c0bc9dc1147b.png" alt="warning" data-base62-sha1="dYvEMFyp0drw9nKr9vBw4AbjaUH" width="690" height="149" data-dominant-color="292C2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">warning</span><span class="informations">1169×254 14.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<em>C:\ProgramData\Anaconda3\envs\Python_37_envs\lib\site-packages\torch\tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won’t be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See <a href="http://github.com/pytorch/pytorch/pull/30531" rel="noopener nofollow ugc">github.com/pytorch/pytorch/pull/30531</a> for more informations.<br>
warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "</em></p>
<p>Even if I keep the same code and same breakpoints, the warning pops up occasionally(20% chance, give or take, and it pops up <strong>seconds after the breakpoints pause</strong>). I found one similar post</p><aside class="quote quote-modified" data-post="1" data-topic="82313">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/q/919ad9/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/grad-attribute-of-a-non-leaf-tensor-being-accessed/82313">.grad attribute of a non-leaf tensor being accessed</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi there, im a newbie at pytorch. 
I am running into the warning: “UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won’t be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.” 
I am not sure which tensor in my procedure is raising the warning, or what exactly i…
  </blockquote>
</aside>
<p>
but it does not answer my question since there is no .to() operation in my codes.</p>
<p>Any tips on how to solve it? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Does it happen when you try to access the <code>.grad</code> field of a given Tensor? If so, how was this Tensor created?</p>

username: mathematics
cooked: <p>As from tutorial , I was using <code>b.backward()</code> function, it throws same err when printing <code>b.grad</code><br>
My code is:</p>
<pre><code class="lang-auto">a = torch.randn(2,2)
a = ((a*3)/ (a-1))
a.requires_grad_(True)
b = (a * a).sum()
b.backward()
print(b.grad)
</code></pre>
<p>Throws error</p>
<pre><code class="lang-auto">None
/usr/local/lib/python3.6/dist-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
</code></pre>
<p>help, Thank you for response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> .</p>

username: albanD
cooked: <p>Hi,</p>
<p>As you can see in the warning message, it is because you call <code>.grad</code> on a Tensor that is not a leaf (meaning that it has some gradient history). And because it is not a leaf, the backward call won’t populate it’s .grad field so your code should most likely not be doing that.</p>
<p>In your example, <code>a</code> is a leaf and so will have a .grad field.<br>
If you need the gradient for <code>b</code>, you need to set <code>b.retain_grad()</code> as mentioned in the warning.</p>

username: mathematics
cooked: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  ,<br>
As you told i tried and may be it worked, but</p>
<pre><code class="lang-auto">a = torch.randn(2,2)
a = ((a*3)/ (a-1))
a.requires_grad_(True)
b = (a * a).sum()
b.backward()
# print(b.grad)
b.retain_grad()
print(b.grad)
</code></pre>
<p>it prints nothing.</p>
<p>But when i used <code>b.retain_grad()</code> before <code>b.backward()</code>  it seems working now , although i am thinking what it is printing</p>
<pre><code class="lang-auto">
b.retain_grad()
b.backward()
# print(b.grad)
print(b.grad)
</code></pre>
<p>prints <code>tensor(1.)</code> may be this is right.</p>

username: albanD
cooked: <p>You can check the doc, <code>.retain_grad()</code> means that it will save the .grad field when doing backward. So you have to call it before the .backward() for it to be useful.</p>
<blockquote>
<p>prints <code>tensor(1.)</code> may be this is right.</p>
</blockquote>
<p>Yes this is expected.<br>
<code>b</code> is the output so its gradient is the default backward value we use for scalar: 1.</p>

username: mathematics
cooked: <p>oh , I am slowly getting over it.<br>
Thanks for kind answer.</p>

username: EricTian
cooked: <aside class="quote no-group" data-username="mathematics" data-post="5" data-topic="83203">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mathematics/48/40082_2.png" class="avatar"> mathematics:</div>
<blockquote>
<p><code>backward</code></p>
</blockquote>
</aside>
<p>Hi, thanks for the reply.<br>
I got this warning when I was trying to debug this project:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/naplab/Conv-TasNet">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/naplab/Conv-TasNet" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/4e3a950e6e1a35e5344b1406a1c566a7adcf55de636e8d3b5597141da8d956e5/naplab/Conv-TasNet" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/naplab/Conv-TasNet" target="_blank" rel="noopener nofollow ugc">GitHub - naplab/Conv-TasNet</a></h3>

  <p>Contribute to naplab/Conv-TasNet development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I just added some codes which prints out the size of the intermediates (in models.py) and I did not change the work flow of main.</p>
<p>x = torch.rand(2, 32000)<br>
nnet = TasNet()<br>
x = nnet(x)<br>
s1 = x[0]<br>
print(s1.shape)</p>
<p>It just creats a random x and iputs to the conv_tasnet then prints the output. I think it does not try to access the .grad field of any given Tensor cause I did not find the .backward operation. I don’t know how this warning is triggered.</p>

username: Hady_Pranoto
cooked: <p>The .grad attribute of a Tensor that is not a leaf Tensor is being accessed.<br>
Its .grad attribute won’t be populated during autograd.backward().<br>
If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor.</p>
<p>If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.</p>
<p>my code working well, after many iteration and epoch, but suddenlly i got above warnning, i think the code worked,</p>
<p>Im predict that problem because this</p>
<p>"If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. "</p>
<p>why this happen? why suddenlly access ?  because the code working well after many epoch and iteration?</p>
<p>what wrong and i should do?</p>

username: hpf
cooked: <p>Hi, <strong>albanD</strong>,<br>
I also have a problem with gradient back propagation.<br>
My example is as follows（Correct code）:</p>
<pre><code class="lang-auto">w = torch.tensor([[[1.,2.,3.],[4.,5.,6.],[7.,8.,9.],[10.,11.,12.]]], requires_grad=True)
x = torch.tensor([4.], requires_grad=True) 

a = torch.add(w, x)     # a = w + x
b = torch.add(w, 1)     # b = w + 1
y = torch.mul(a, b)     # y = a * b

y.backward(torch.ones_like(w))   
print(w.grad)   
</code></pre>
<p>The right result is：</p>
<pre><code class="lang-auto">tensor([[[ 7.,  9., 11.],
         [13., 15., 17.],
         [19., 21., 23.],
         [25., 27., 29.]]])
</code></pre>
<p>But，When I want to change the shape of the variable <strong>W</strong> from [1,4,3] to [4,3], a warning occurs and the output is <strong>none</strong>。<br>
As shown below：</p>
<pre><code class="lang-auto">w = torch.tensor([[[1.,2.,3.],[4.,5.,6.],[7.,8.,9.],[10.,11.,12.]]], requires_grad=True)
w=w.squeeze()
x = torch.tensor([4.], requires_grad=True) 

a = torch.add(w, x)     # a = w + x
b = torch.add(w, 1)     # b = w + 1
y = torch.mul(a, b)     # y = a * b

y.backward(torch.ones_like(w))   
print(w.grad)  
</code></pre>
<p>The warning:</p>
<pre><code class="lang-auto">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. 
Its .grad attribute won't be populated during autograd.backward(). 
If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. 
If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.
</code></pre>
<p>My question is <strong>Can’t I change the shape of the tensor in the process of gradient back propagation? No matter it’s view, reshape, or squeeze, etc</strong><br>
Waiting for your reply</p>

username: albanD
cooked: <p>Hi,</p>
<p>The only problem here is that you change the python variable “w” to point to the new squeezed Tensor. So it doesn’t point to the Tensor that you created with requires_grad=True anymore and so the Tensor it now points to has a .grad field that is not populated.<br>
You just need to do <code>w_new = w.squeeze()</code> and use <code>w_new</code> in your computations and the <code>w.grad</code> will still be populated as you expect.</p>

username: hpf
cooked: <p>Thank you for your guidance,It works.I may have understood what you mean,<br>
In the process of back propagation, the state of the original node should not be changed…<br>
<code>w=w.squeeze()</code>will change w’s  <code>requires_grad=True</code> become to <code>grad_fn=&lt;SqueezeBackward0&gt;</code>.I’ll remember to look at the state of the variable first.<br>
Finally, thank you again,have a nice day.</p>