Difference between .grad and ._grad

username: xuehy
cooked: <p>I notice that the Variable class has two similar members <code>.grad</code> and <code>._grad</code>. What is the difference between them? In the pytorch a3c implementation <a href="https://github.com/ikostrikov/pytorch-a3c" rel="nofollow noopener">pytorch-a3c</a>, there is a piece of code:</p>
<pre><code>def ensure_shared_grads(model, shared_model):
  for param, shared_param in zip(model.parameters(), shared_model.parameters()):
      if shared_param.grad is not None:
          return
      shared_param._grad = param.grad  
</code></pre>
<p>Why update member <code>._grad</code> of the shared_model?</p>

username: alexis-jacq
cooked: <p>Variable.grad is not writable, while <code>Variable._grad</code> is writable. If you want to force the value of the gradient to equal another value, you must modify <code>._grad</code> and not <code>.grad</code> otherwise, you will have the exception:</p>
<pre><code class="lang-auto">AttributeError: attribute 'grad' of 'torch._C._VariableBase' objects is not writable
</code></pre>

username: PatrickChen83
cooked: <p>I just checked this using a simple script</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.optim import SGD

x = Variable(torch.zeros(3, 4), requires_grad=True)

a = (3 * x).sum()

oa = SGD([x], momentum=0.0, lr=1.0)

for i in range(100):
    oa.zero_grad()
    a.backward()
    print('-' * 40)
    print(x.grad)
    x._grad = Variable(torch.ones(x.grad.size()))
    print(x.grad)
    oa.step()
    print(x)

</code></pre>
<p>However this behavior seems undocumented…</p>

username: hughperkins
cooked: <blockquote>
<p>However this solution seems undocumented</p>
</blockquote>
<p>I sense a solution for this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> .</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3" title="Screen Shot 2017-07-14 at 1.03.09 AM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3_2_690x119.png" width="690" height="119" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3_2_690x119.png, https://discuss.pytorch.org/uploads/default/optimized/1X/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3_2_1035x178.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/31f135d29cbf8c1bdca0ed4994cc761a2fe843b3_2_1380x238.png 2x" data-dominant-color="EDF1F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-14 at 1.03.09 AM.png</span><span class="informations">2106×366 56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As a bonus, you then get the snazzy ‘contributor’ annotation after your name, in anything you post into the Issues page:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/6b42f32ca2a25a825b024f68e952866cd9dd7683.png" data-download-href="https://discuss.pytorch.org/uploads/default/6b42f32ca2a25a825b024f68e952866cd9dd7683" title="Screen Shot 2017-07-14 at 1.04.34 AM.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/6b42f32ca2a25a825b024f68e952866cd9dd7683_2_690x124.png" width="690" height="124" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/6b42f32ca2a25a825b024f68e952866cd9dd7683_2_690x124.png, https://discuss.pytorch.org/uploads/default/optimized/1X/6b42f32ca2a25a825b024f68e952866cd9dd7683_2_1035x186.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/1X/6b42f32ca2a25a825b024f68e952866cd9dd7683_2_1380x248.png 2x" data-dominant-color="E9EDF4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2017-07-14 at 1.04.34 AM.png</span><span class="informations">1636×296 56.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: smth
cooked: <p>it’s undocumented because it’s not meant to be used by the user. Generally all stuff starting with <code>_</code> is treated as internal stuff.</p>

username: sbb
cooked: <p>If we want to manually set the gradient of a variable, would you recommend using ._grad ?</p>

username: jdhao
cooked: <p>I think soumith is pretty clear about this:</p>
<blockquote>
<p>It is not meant to be used by the user.</p>
</blockquote>

username: sbb
cooked: <p>So if you wanted to manually set the gradient of a variable, what would be the correct command?</p>

username: alexis-jacq
cooked: <p>A gradient is defined wrt some operations, it has a precise mathematical signification.</p>
<p>There is no point to manually modify it. You may want to modify the usage of the gradient while, for example, setting the gradient descent (in that case you can do something inheriting from <code>torch.optim.Optimizer</code>)</p>

username: xuehy
cooked: <p>what about the case in my example on the top?</p>

username: smth
cooked: <p>if you want to modify the gradient, use hooks. this tutorial in introduces hooks: <a href="http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks">http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html#forward-and-backward-function-hooks</a></p>

username: Yummy_Chen
cooked: <p>so, our user should forget the existence of ‘._grad’, just use ‘.grad’ ?<br>
thanks!</p>

username: Yummy_Chen
cooked: <p>what do you mean?<br>
if you do not specify the parameter “<em>retain_graph</em>” to True, isn’t the loss only can be backward one time?</p>