'NoneType' object has no attribute 'zero_'

Audrey: <p>Hello, I got an error when writing a simple linear regression model using autograd. My model is<br>
<strong>y = w1 * x1 + w2*x2 + b</strong> and I need to learn parameters <strong>w1</strong>, <strong>w2</strong> and <strong>b</strong>. My code is as below:</p>
<pre><code class="lang-python">x1 = [0.0, 2.0]
y1 = 1.0
x2 = [1.0, 3.0]
y2 = 1.0
x3 = [1.0, 0.0]
y3 = 0.0
x4 = [2.0, 1.0]
y4 = 0.0
X = np.array([x1, x2, x3, x4])
Y = np.array([y1, y2, y3, y4])

feature = torch.from_numpy(X)
label = torch.from_numpy(Y).view(-1, 1)

epochs = 100
lr = 0.01
weight = torch.randn((2, 1), requires_grad=True, dtype=torch.float64)
bias = torch.randn(1, requires_grad=True, dtype=torch.float64)

for i in range(epochs):
    predict = torch.mm(feature, weight) + bias.item()
    loss = torch.sum(predict - label, dim=0)
    loss.backward()
    weight = weight - weight.grad*lr
    bias = bias - bias*lr
    weight.grad.zero_()
    bias.grad.zero_()
</code></pre>
<p>Error occurs at line <em>weight.grad.zero_()</em> :</p>
<pre><code class="lang-auto">AttributeError: 'NoneType' object has no attribute 'zero_'
</code></pre>
<p>I want to know how to fix it? Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

albanD: <p>Hi,</p>
<p>When you do <code>weight = weight - weight.grad*lr</code>, <code>weight</code> now points to a brand new Tensor and so the gradient informations from the original weight Tensor are gone. You can check that after this line, <code>weight.grad is None</code>.</p>
<p>The other problem you’re going to encounter is that <code>weight = weight - XXX</code>, this will be tracked by the autgrad which you most likely don’t want.</p>
<p>To fix these, you can</p>
<ul>
<li>Change weight inplace, to avoid the first problem above</li>
<li>Disable autograd while you update your weights to avoid the second one.</li>
</ul>
<p>Here is the new code update:</p>
<pre><code class="lang-auto">for i in range(epochs):
    predict = torch.mm(feature, weight) + bias.item()
    loss = torch.sum(predict - label, dim=0)
    loss.backward()
    # Disable the autograd
    with torch.no_grad():
        # Inplace changes
        weight.sub_(weight.grad*lr)
        bias.sub_(bias.grad*lr) # A .grad is missing in your code here I think ;)
        # Do the reset in no grad mode as well in case you do second order
        # derivatives later (meaning that weight.grad will requires_grad)
        weight.grad.zero_()
        bias.grad.zero_()
</code></pre>

Audrey: <p>Thank you! But I still get the same error following your code:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/audrey/PycharmProjects/PyTorchLearning/main.py", line 31, in &lt;module&gt;
    bias.grad.data.zero_()
AttributeError: 'NoneType' object has no attribute 'data'
</code></pre>
<p>Then I changed <code>bias.item()</code> into <code>bias</code></p>
<pre><code class="lang-python">predict = torch.mm(feature, weight) + bias
</code></pre>
<p>No error appeared anymore! I wonder why <code>bias.item()</code> doesn’t work.<br>
Anyway, my code can run successfully, thanks!</p>

Audrey: <p>I think I find the reason why <code>bias.item()</code> doesn’t work. Using scaler makes tensor <code>bias</code> not be tracked.</p>

albanD: <p>Exactely: <code>.item()</code> gives you a python number that means that you don’t have Tensors anymore (and so no autograd anymore).</p>
<p>Also given you error message, you seem to be using <code>.data</code>. You should not use it and using <code>with torch.no_grad()</code> instead as in my example.</p>

mathematics: <p>I tried to use with torch.no_grad() to resolve this error</p>
<pre><code class="lang-auto">def weights_init_normal(m):
    classname = m.__class__.__name__
    lr = 0.01
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        torch.nn.init.normal_(m.weight, mean=0.0, std=0.02)
        with torch.no_grad():
            m.bias.sub_(m.bias.grad*lr)
            m.bias.grad.zero_()
</code></pre>
<p>got alot of error.<br>
<code>AttributeError: 'NoneType' object has no attribute 'sub_'</code></p>
<p>I found out my bias is NoneType.</p>
<p>Tried several techniques , nothing worked .</p>
<p>I wanted to initialize the weights of bias to zero.</p>
<p>Please help!!!</p>

albanD: <p>Hi,</p>
<p>By default, the <code>.grad</code> attribute is <code>None</code>. Which means “full of zeros”.</p>

mathematics: <p>Oh ,That means, It is ok to not to touch None type bias. However, my model was not converging, may be my error. Thanks for knowledge <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Aanchal: <p>Hi, Instead of using this,<br>
with torch.no_grad():<br>
w = w - w.grad * 1e-5<br>
b = b- b.grad * 1e-5<br>
use the mentioned code:<br>
with torch.no_grad():<br>
w -= w.grad * 1e-5<br>
b -= b.grad * 1e-5<br>
while using w = w - w.grad * 1e-5, the updated values of the weight is now stored at some other memory location, that is a new tensor, and so the gradient informations from the original weight Tensor is gone, and hence we are trying to calculating w.grad it is not able to find any value as it has shifted to new location.</p>