Grad is None even when requires_grad=True

li012589: <p>I run into this wired behavior of autograd when try to initialize weights.</p>
<p>Here is a minimal case:</p>
<pre><code class="lang-python">import torch

print("Trial 1: with python float")
w = torch.randn(3,5,requires_grad = True) * 0.01

x = torch.randn(5,4,requires_grad = True)

y = torch.matmul(w,x).sum(1)

y.backward(torch.ones(3))

print("w.requires_grad:",w.requires_grad)
print("x.requires_grad:",x.requires_grad)

print("w.grad",w.grad)
print("x.grad",x.grad)

print("Trial 2: with on-the-go torch scalar")
w = torch.randn(3,5,requires_grad = True) * torch.tensor(0.01,requires_grad=True)

x = torch.randn(5,4,requires_grad = True)

y = torch.matmul(w,x).sum(1)

y.backward(torch.ones(3))

print("w.requires_grad:",w.requires_grad)
print("x.requires_grad:",x.requires_grad)

print("w.grad",w.grad)
print("x.grad",x.grad)

print("Trial 3: with named torch scalar")
t = torch.tensor(0.01,requires_grad=True)
w = torch.randn(3,5,requires_grad = True) * t

x = torch.randn(5,4,requires_grad = True)

y = torch.matmul(w,x).sum(1)

y.backward(torch.ones(3))

print("w.requires_grad:",w.requires_grad)
print("x.requires_grad:",x.requires_grad)

print("w.grad",w.grad)
print("x.grad",x.grad)
</code></pre>
<p>The output should be</p>
<pre><code class="lang-bash">Trial 1: with python float
w.requires_grad: True
x.requires_grad: True
w.grad None
x.grad tensor([[-0.0267, -0.0267, -0.0267, -0.0267],
        [ 0.0040,  0.0040,  0.0040,  0.0040],
        [-0.0034, -0.0034, -0.0034, -0.0034],
        [-0.0010, -0.0010, -0.0010, -0.0010],
        [ 0.0215,  0.0215,  0.0215,  0.0215]])
Trial 2: with on-the-go torch scalar
w.requires_grad: True
x.requires_grad: True
w.grad None
x.grad tensor([[-0.0028, -0.0028, -0.0028, -0.0028],
        [ 0.0130,  0.0130,  0.0130,  0.0130],
        [-0.0027, -0.0027, -0.0027, -0.0027],
        [ 0.0054,  0.0054,  0.0054,  0.0054],
        [ 0.0133,  0.0133,  0.0133,  0.0133]])
Trial 3: with named torch scalar
w.requires_grad: True
x.requires_grad: True
w.grad None
x.grad tensor([[ 0.0227,  0.0227,  0.0227,  0.0227],
        [ 0.0101,  0.0101,  0.0101,  0.0101],
        [-0.0200, -0.0200, -0.0200, -0.0200],
        [-0.0052, -0.0052, -0.0052, -0.0052],
        [-0.0031, -0.0031, -0.0031, -0.0031]])

</code></pre>
<p>You can see that even with tensors’ requires_grad being True, their grad still is None. Is this a supposed behavior?</p>
<p>I know that adding <code>w.requires_grad_()</code> can solve this problem, but shouldn’t autograd at least change the tensor’s requires_grad to false?</p>

InnovArul: <p>I think, this touches upon the concept of leaf variables and intermediate variables.<br>
As far as I could see, in all three cases, <code>w</code> is an intermediate variable and the gradients will be accumulated in <code>torch.randn(..., requires_grad=True)</code> (which is one of the roots of the computation tree) instance. All the intermediate variables’ gradient (including <code>w</code>) is removed during the <code>backward()</code> call. If you want to retain those gradients, call <code>w.retain_grad()</code> before calling <code>backward()</code>.</p>

li012589: <p>Thanks for the reply!</p>
<p>This clear things up, now I understand why w doesn’t get a grad. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=6" title=":grinning:" class="emoji" alt=":grinning:"></p>

Rohit_Deshpande: <p>I’m getting None for grad after specifying retain_grad().<br>
Can anyone help?</p>
<pre><code class="lang-auto">import torch
import numpy as np

# Temp, rain, hum
inputs = np.array([[73, 67, 43], 
                   [91, 88, 64], 
                   [87, 134, 58], 
                   [102, 43, 37], 
                   [69, 96, 70]], dtype='float32')

# apples and oranges
targets = np.array([[56, 70], 
                    [81, 101], 
                    [119, 133], 
                    [22, 37], 
                    [103, 119]], dtype='float32')

inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)
print(inputs)
print(targets)

# weights and Biases
w = torch.randn(2,3, requires_grad=True)
b = torch.randn(2, requires_grad=True)
print(w)
print(b)


#  Define model
def model(x):
  return x @ w.t() + b

predit = model(inputs)

def mse(t1, t2):
  diff = t1 - t2
  return torch.sum(diff * diff)/ diff.numel()

loss = mse(predit, targets)
loss


# Compute gradient
w.retain_grad()
b.retain_grad()
loss.backward()

print(w)
print(w.grad)
</code></pre>

albanD: <p>Hi,</p>
<p>Running this code does print a non-None gradient. You can check it in this colab notebook: <a href="https://colab.research.google.com/drive/1dXUTa5bx2pfewb52KIGpftXu66k7qeTj?usp=sharing">https://colab.research.google.com/drive/1dXUTa5bx2pfewb52KIGpftXu66k7qeTj?usp=sharing</a><br>
How did you install pytorch? Which version is it? Do you do anything else?</p>

Rohit_Deshpande: <p><a class="mention" href="/u/alband">@albanD</a> Thanks, I was also able to print grad I’m running torch =&gt;1.6.0 and i’m executing in colab. I reinstalled the pytorch then I was able to execute</p>

Sharifi: <p>I cannot understand that why gradients of w should be removed during the backward()?<br>
we want to update the <strong>w</strong> (<em>our model’s parameter</em>) and we need it’s gradients.<br>
Am I wrong?</p>

Sharifi: <p>I also had a similar problem. Although all the variables had <code>require_grad = True</code>, the gradients was <code>None</code></p>
<p>After scrutinizing my code, I realized that in one of my modules, there was <code>with torch.no_grad:</code> in forward method that disabled the gradient computation. Therefore, I removed it and the problem was solved.</p>

Yangmin: <p>I am also curious. Is there a answer for this?</p>

jiuluan_lv: <p>For Tensors that have requires_grad which is True, they will be leaf Tensors if they were created by the user. This means that they are not the result of an operation and so grad_fn is None.</p>
<p>In this line:</p>
<pre data-code-wrap="python"><code class="lang-python">w = torch.randn(3,5,requires_grad = True) * 0.01
</code></pre>
<p>We could also wirte this which is the same as above:</p>
<pre data-code-wrap="python"><code class="lang-python">temp = torch.randn(3,5,requires_grad = True)
w =  temp * 0.01
</code></pre>
<p>Use method <code>is_leaf</code> to print whether the variable is a leaf variable or not</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e76fe5324694b130bfb8e1305845ded99903cc38.png" alt="image" data-base62-sha1="x1o1SgcPhp4MdRoxmIi1hD8qyJW" width="408" height="122"></p>
<p>So,that is the reason why <code>w</code> does not have grad</p>