Loss.grad() is constant 1

StellaJourney_Huang: <p>Hi,</p>
<p>I am using torch.nn.BCELoss() and F.binary_cross_entropy_with_logits, but loss.grad() is always constant 1.</p>
<p>Do you know why this happens and how to solve it?<br>
Thanks!</p>
<pre><code>        optimizer.zero_grad()
        A_pred, z = model(features, adj_norm)

        weight_tensor = torch.tensor(weight_tensor, dtype=torch.float).cuda()

        criterion = torch.nn.BCELoss(weight=weight_tensor)
        log_lik = norm*criterion(A_pred.view(-1), adj_label.to_dense().view(-1))
        loss = log_lik 
        loss.retain_grad()
        loss.backward()
        print(loss.grad)
        optimizer.step()</code></pre>

gphilip: <aside class="quote no-group" data-username="StellaJourney_Huang" data-post="1" data-topic="130964">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stellajourney_huang/48/41877_2.png" class="avatar"> StellaJourney_Huang:</div>
<blockquote>
<p>but loss.grad() is always constant 1</p>
</blockquote>
</aside>
<p>Why is this a problem?</p>

StellaJourney_Huang: <p>I tried four different models, but all of them are not learning… then I checked loss gradient and see it is constant 1. I guess it needs to change at least if the model is training properly.</p>

gphilip: <p>Why do you call <code>loss.retain_grad()</code> ?</p>

StellaJourney_Huang: <p>without retain_grad, loss gradient will be None. So I add retain_grad(), then gradient shows 1 always.</p>

gphilip: <p>
From what I understand, <code>loss.retain_grad()</code> initializes <code>loss.grad</code> to <code>1</code>
.  
This is why you get <code>1</code> as the value of <code>loss.grad</code>.
 This understanding was wrong. See my comment below.</p>
<p>This has no relation to whether your network is learning or not, since (as far as I understand) calling <code>loss.backward()</code> has no effect on <code>loss.grad</code>.</p>
<p>A better way to figure out if your network is learning is to print out one of the values in <code>loss</code> and see 
how
 if it changes over iterations.</p>

ptrblck: <aside class="quote no-group" data-username="gphilip" data-post="6" data-topic="130964">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/858c86/48.png" class="avatar"> gphilip:</div>
<blockquote>
<p>since (as far as I understand) calling <code>loss.backward()</code> has no effect on <code>loss.grad</code>.</p>
</blockquote>
</aside>
<p>For the sake of completeness:<br>
The <code>backward()</code> operation will use the default gradient value of <code>1</code> for scalar loss values (for other shapes, you need to manually pass the gradient to <code>backward</code>).<br>
Here is a small example showing that <code>loss.grad</code> is defined by the passed gradient:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Linear(1, 1)
x = torch.randn(1, 1)

# default gradient
out = model(x)
loss = out**2
loss.retain_grad()

loss.backward()
print(loss.grad)
&gt; tensor([[1.]])

# pass gradient manually
out = model(x)
loss = out**2
loss.retain_grad()

loss.backward(torch.tensor(10.).view_as(loss))
print(loss.grad)
&gt; tensor([[10.]])
</code></pre>
<p>Besides that, I agree with <a class="mention" href="/u/gphilip">@gphilip</a> that checking the <code>loss.grad</code> value is not particularly useful to check if the model is learning or not.<br>
As a debugging step I would also suggest to check the <code>.grad</code> attribute of all parameters of the model and make sure they show valid (or the expected) values.</p>

gphilip: <p>To complete my understanding, based on what <a class="mention" href="/u/ptrblck">@ptrblck</a> said: <code>loss.retain_grad()</code> initializes  <code>loss.grad</code> to <code>None</code>, not <code>1</code> like I said above. It is the subsequent call to <code>loss.backward()</code> that sets <code>grad</code> to <code>1</code>. This is analogous to what happens when we specify <code>requires_grad=True</code> : the <code>grad</code> value is initially set to <code>None</code>, and is set to numerical values after a call to <code>backward()</code>.</p>
<p>Here is a mini-tutorial in code that I wrote to get my head around these concepts:</p>
<pre><code class="lang-auto">import torch
from torch import nn

model = nn.Linear(1, 1)

print("#"*7)
print("Case 1")
print("#"*7)

# Case 1: No tensor has the grad attribute, none of the learnt gradients 
# is saved

# This is a leaf, but has no grad attribute 
x = torch.randn(1, 1) 

print(x)
print(x.is_leaf) # True
print(x.grad) # None

# Forward pass, compute loss
out = model(x)
loss = out**2

# The following will print None, since we haven't computed any gradients 
# yet. (Also because neither variable has a grad attribute!)
print(loss.grad) # None
print(x.grad) # None


# This computes the gradients of loss w.r.t the leaf nodes 
# (which is just x). Since loss is a 1-element tensor, this method 
# implicitly sets loss.grad to torch.ones_like(loss).
# But this setting is temporary, and is not saved because loss is 
# not a leaf. So loss.grad remains at None after this call.
loss.backward()

# The following will print None since, while we did compute the 
# gradients of loss w.r.t x, neither loss nor x has a grad attribute
print(loss.grad) # None
print(x.grad) # None

print("")
print("*"*7)
print("Case 2")
print("*"*7)
# Case 2: The one leaf tensor has the grad attribute, so its
# learnt gradient is saved

# This is a leaf, and has a grad attribute because we told so.
# This grad attribute is initialized to None.
y = torch.randn(1, 1, requires_grad=True)

print(y)
print(y.is_leaf) # True
print(y.grad) # None

# Forward pass, compute loss
out = model(y)
loss = out**2

# The following will print None, since we haven't computed any gradients 
# yet
print(loss.grad) # None
print(y.grad) # None

# This computes the gradients of loss w.r.t the leaf nodes 
# (which is just y). Since loss is a 1-element tensor, this method 
# implicitly sets loss.grad to torch.ones_like(loss).
# But this setting is temporary, and is not saved because loss is not 
# a leaf. So loss.grad remains at None after this call.
loss.backward()

# The following will print None since, while we did compute the gradients
# of loss w.r.t y,
# loss has no grad attribute
print(loss.grad) # None

# This will print a [1, 1] float tensor, which is the gradient of 
# loss w.r.t y 
print(y.grad) 

print("")
print("*"*7)
print("Case 3")
print("*"*7)
# Case 3: Both the leaf tensor and the loss (which is not a leaf) have 
# grad attributes, 
# so their learnt gradients are saved

# This is a leaf, and has a grad attribute because we told so.
# This grad attribute is initialized to None.
z = torch.randn(1, 1, requires_grad=True)

print(z)
print(z.is_leaf) # True
print(z.grad) # None

# Forward pass, compute loss
out = model(z)
loss = out**2

# Enable the grad attribute for the non-leaf tensor loss.
loss.retain_grad()

# The following will print None, since we haven't computed any gradients 
# yet
print(loss.grad) # None
print(z.grad) # None

# This computes the gradients of loss w.r.t the leaf nodes 
# (which is just z). Since loss is a 1-element tensor, this method 
# implicitly sets loss.grad to torch.ones_like(loss).
# Since we enabled the grad attribute of loss, this setting is saved even
# after the call returns.
loss.backward()

# The following will print a [1, 1] float tensor whose element is 1.0. 
# This is the value which was implicitly set by the call to loss.backward()
print(loss.grad) # [[1.0]]

# This will print a [1, 1] float tensor, which is the gradient of loss 
# w.r.t z 
print(z.grad) 

# Case 4: Same as Case 3, except we explicitly set the grad attribute 
# of loss to a large value.
# Observe that the grad of the leaf changes proportionately.
print("")
print("*"*7)
print("Case 4")
print("*"*7)

w = torch.randn(1, 1, requires_grad=True)

print(w)
print(w.is_leaf)
print(w.grad)


out = model(w)
loss = out**2

loss.retain_grad()


print(loss.grad) 
print(w.grad) 


loss.backward(torch.tensor(1e5).view_as(loss))


print(loss.grad) 
print(w.grad) 


</code></pre>

BruceDai003: <p>I guess you probably didn’t understand what Loss.grad() means here. Because you are doing backprop from Loss all the way backwardly to the leaf nodes in the graph, the starting point is Loss, so Loss.grad() is the grad of loss w.r.t loss itself, so the result will be 1, besides the grad passed to loss which is a scalar defaults to 1 also. Meaning 1 * 1 = 1, so you are getting 1 in the end.<br>
For other nodes, e.g., in the below equation:<br>
y = 2 * x;<br>
if you are calculating x.grad(), which is the grad of loss w.r.t x as in a computation graph, so let’s denote it as dL/dx.<br>
dL/dx = dL/dy * dy/dx = dL/dy * 2;<br>
So if you are calculating Loss.grad(). Which would be:<br>
Loss = Loss. And dL/dL = 1.<br>
So you are getting:<br>
dL/dL = 1 * 1 = 1<br>
As already mentioned by <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/gphilip">@gphilip</a> , if you are trying to debug if your model is learning or not, you probably should check the loss value itself.</p>

StellaJourney_Huang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/gphilip">@gphilip</a> <a class="mention" href="/u/brucedai003">@BruceDai003</a>  Thanks for you reply.</p>
<p>I followed your suggestion, but unfortunately it is still not training…I checked loss and weights that have different value in each epoch, but the model is not learning within 50 epoches. Besides, I also tried different model, architecture and hyperparameters. The code is as following link, can you please help to check the code? Appreciate your help.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/yellow07200/Test">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/yellow07200/Test" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/f9e2a66f26f5fc39292612066126804aff29a1d7943de3fccf7509f3179c0271/yellow07200/Test" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/yellow07200/Test" target="_blank" rel="noopener nofollow ugc">GitHub - yellow07200/Test</a></h3>

  <p>Contribute to yellow07200/Test development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>