Grad is NoneType

username: credoxfromcqut
cooked: <p>hello every one<br>
there is a quastion when i use torch to optimize my simple model<br>
it works when they are in   cpu(no .cuda())<br>
when it moves to cuda  and it fails to compute<br>
i hope someone help me to solve this<br>
thanks in advance</p>
<p>N, D_in, H, D_out = 64, 1000, 100, 10</p>
<h1></h1>
<p>X = torch.randn(N,D_in).cuda()<br>
Y = torch.randn(N,D_out).cuda()<br>
W1 = torch.randn(D_in,H,requires_grad=True).cuda()<br>
W2 = torch.randn(H, D_out,requires_grad=True).cuda()</p>
<p>learning_rate = 1e-6</p>
<p>for t in range(500):<br>
<span class="hashtag">#forward</span> propagation<br>
h = X.mm(W1) <span class="hashtag">#N</span> * H<br>
h_relu = h.clamp(min=0)<br>
y_pred = h_relu.mm(W2) <span class="hashtag">#N</span> * D_out</p>
<pre><code>#loss function
loss = (y_pred-Y).pow(2).sum()
#print('y',Y)
#print('y pred',y_pred)
print(loss.item())
#backword propagation
loss.backward()
with torch.no_grad():
    
#update weights
    W1 -= learning_rate * W1.grad
    W2 -= learning_rate * W2.grad
    W1.grad.zero_()
    W2.grad.zero_()
</code></pre>
<p>===========================================</p>
<p>26         W1 -= learning_rate * W1.grad<br>
27         W2 -= learning_rate * W2.grad<br>
28         W1.grad.zero_()</p>
<p>TypeError: unsupported operand type(s) for *: ‘float’ and ‘NoneType’</p>

username: albanD
cooked: <p>Hi,</p>
<p>When you do <code> W1 = torch.randn(D_in,H,requires_grad=True).cuda()</code> what is returned and stored as <code>W1</code> is not a leaf anymore: it is the result of the differentiable op <code>.cuda()</code>.<br>
You should do <code>W1 = torch.randn(D_in,H, device="cuda", requires_grad=True)</code> to make sure <code>W1</code> is a leaf and thus will get a .grad field.</p>

username: credoxfromcqut
cooked: <aside class="quote no-group quote-modified" data-username="albanD" data-post="2" data-topic="89013">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>device=“cuda”,</p>
</blockquote>
</aside>
<p>God ! it really works and thanks for your flash reply .very thankful<br>
best wishes</p>

username: credoxfromcqut
cooked: <p>hey guys would you mind telling me some detail about the " differentiable op .cuda()` ." or some links helping understanding to find out the principle?<br>
that will be grateful</p>

username: albanD
cooked: <p>It is the same as if you’re doing:</p>
<pre><code class="lang-auto">a = torch.rand(10, requires_grad=True)
b = a + 1

b.sum().backward()
</code></pre>
<p>Then <code>b.grad</code> will be None.<br>
The same thing happens if you replace the <code>+ 1</code> op by <code>.cuda()</code>.<br>
It is handled like any other differentiable operation on Tensor.</p>

username: credoxfromcqut
cooked: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="89013">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>When you do <code> W1 = torch.randn(D_in,H,requires_grad=True).cuda()</code> what is returned and stored as <code>W1</code> is not a leaf anymore: it is the result of the differentiable op <code>.cuda()</code> .<br>
You should do <code>W1 = torch.randn(D_in,H, device="cuda", requires_grad=True)</code> to make sure <code>W1</code> is a leaf and thus will get a .grad field.</p>
</blockquote>
</aside>
<p>wow  interesting and now i’ve got it. thanks for your reply and your reply is  also easy to understand .<br>
thank you again !!</p>