Grad is None confusion in the "what is torch.nn" tutorial

hunan-rostomyan: <p>I’m following along <a href="https://pytorch.org/tutorials/beginner/nn_tutorial.html" rel="noopener nofollow ugc">What is torch.nn <em>really</em>?</a> tutorial. There’s a setup part in the beginning where they define:</p>
<pre data-code-wrap="py"><code class="lang-py">weights = torch.randn(784, 10) / math.sqrt(784)
weights.requires_grad_()
</code></pre>
<p>I thought I might as well just rewrite that as:</p>
<pre data-code-wrap="python"><code class="lang-python">weights = torch.randn(784, 10, requires_grad=True) / math.sqrt(784)
assert weights.requires_grad == True
</code></pre>
<p>Everything that followed was fine until this loop:</p>
<pre data-code-wrap="python"><code class="lang-python">lr = 0.5  # learning rate
epochs = 2  # how many epochs to train for

for epoch in range(epochs):
    for i in range((n - 1) // bs + 1):
        start_i = i * bs
        end_i = start_i + bs
        xb = x_train[start_i:end_i]
        yb = y_train[start_i:end_i]
        pred = model(xb)
        loss = loss_func(pred, yb)

        loss.backward()
        with torch.no_grad():
            weights -= weights.grad * lr
            bias -= bias.grad * lr
            weights.grad.zero_()
            bias.grad.zero_()
</code></pre>
<p>where I got:</p>
<pre data-code-wrap="txt"><code class="lang-txt">TypeError                                 Traceback (most recent call last)
&lt;ipython-input-55-24454d388081&gt; in &lt;cell line: 0&gt;()
     21         with torch.no_grad():
     22             print((weights.requires_grad, bias.requires_grad))
---&gt; 23             weights -= weights.grad * lr
     24             bias -= bias.grad * lr
     25             weights.grad.zero_()

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
</code></pre>
<p>Turns out that when I use the original definition:</p>
<pre data-code-wrap="py"><code class="lang-py">weights = torch.randn(784, 10) / math.sqrt(784)
weights.requires_grad_()
</code></pre>
<p>this error is avoided, but I can’t figure out <strong>why</strong>? I know it’s got something to do with this division by sqrt of 784 because this also works:</p>
<pre data-code-wrap="py"><code class="lang-py">weights = torch.randn(784, 10, requires_grad=True) # No division
</code></pre>
<p>Thanks in advance for any feedback.</p>

ptrblck: <p>A division is a differentiable operation and this <code>weights</code> is not a leaf tensor anymore. If you try to access its <code>grad</code> attribute you will see a warning explaining you are trying to access the gradient of a non-leaf.</p>

hunan-rostomyan: <p>Thanks Piotr, that’s helpful. I see that <code>weights</code> variable is referring to a non-leaf node, but when we eventually compute a loss, shouldn’t that automatically result in a gradient calculation for every node in the underlying computational graph?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/6787a13af70e2222fe63cbb1d90b8171b982fea6.png" data-download-href="https://discuss.pytorch.org/uploads/default/6787a13af70e2222fe63cbb1d90b8171b982fea6" title="Screenshot 2025-02-23 at 10.40.14 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6787a13af70e2222fe63cbb1d90b8171b982fea6_2_690x396.png" alt="Screenshot 2025-02-23 at 10.40.14 PM" data-base62-sha1="eLRLqz3GTTZLBHGUnEqgeJuWtSK" width="690" height="396" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6787a13af70e2222fe63cbb1d90b8171b982fea6_2_690x396.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6787a13af70e2222fe63cbb1d90b8171b982fea6_2_1035x594.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/7/6787a13af70e2222fe63cbb1d90b8171b982fea6_2_1380x792.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-02-23 at 10.40.14 PM</span><span class="informations">1604×922 46.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Two things I tried that didn’t work:</p>
<pre data-code-wrap="py"><code class="lang-py"># Attempt 1
weights = torch.div(torch.randn(784, 10), math.sqrt(784), requires_grad=True)
#=&gt; TypeError: div() received an invalid combination of arguments ...
</code></pre>
<p>and</p>
<pre data-code-wrap="py"><code class="lang-py"># Attempt 2
weights = torch.randn(784, 10, requires_grad=True)
weights.div_(math.sqrt(784))
#=&gt; RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>I feel like since the node is differentiable, there should be a gradient available on it regardless of whether it’s a leaf or not?</p>

ptrblck: <p>I’m not on my workstation but a warning should have been raised explaining how to access gradients from non-leaf tensors via <code>.retain_grad</code>. Search for this warning and don’t ignore it.</p>