All Gradients are Zeros

username: alexj
cooked: <p>I have a rather complext network, way too complex to post. After I call <code>loss.backward()</code>, all my gradients become <code>0.0</code>, including the last one before the loss function. What could be the reason? Thank you</p>

username: Brock_Brown
cooked: <p>Without seeing the details, my guess is that the loss gradient is zero at most points. <a href="https://stackoverflow.com/a/76440593/2121074" rel="noopener nofollow ugc">I posted an answer on Stack Overflow</a> answering a question similar to this. Maybe you have a chunk of your math done with discrete type parts? Check out how you’re calculating loss.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/alexj">@alexj</a><br>
Following might be some of the reasons why the grads might become 0</p>
<ol>
<li>The input tensor consists of zeros</li>
</ol>
<pre><code class="lang-auto">import os
import numpy as np

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms

class linearRegressionOLS(nn.Module):
    def __init__(self):
        super(linearRegressionOLS,self).__init__()
        self.linearModel=nn.Linear(10,1)
        
    def forward(self,x):
        x = self.linearModel(x)
        return x


learning_rate = 0.001
num_epochs=2

model=linearRegressionOLS()
criterion=nn.MSELoss()
optimizer=torch.optim.SGD(model.parameters(),lr=learning_rate)

X=np.zeros((100,10)).astype(np.float32)
Y=np.random.randint(2,size=(100)).reshape(100,1).astype(np.float32)

inputVal=Variable(torch.from_numpy(X))
outputVal=Variable(torch.from_numpy(Y))

for epoch in range(num_epochs):
    # In a gradient descent step, the following will now be performing the gradient descent now
    optimizer.zero_grad()
    # We will now setup the model
    dataOutput = model(inputVal)
    # We will now define the loss metric
    loss = criterion(dataOutput, outputVal)
    # We will perform the backward propagation
    loss.backward()
    print(model.linearModel.weight.grad)
    optimizer.step()
    if epoch % 1 == 0:
        print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, num_epochs, loss))        

# Final weights of the linear Regression Model
coeff=model.linearModel.weight.data.numpy()
print("the final coeff are {}".format(coeff))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3af55454c0fd09001dce8edba016143a7b7050b.png" data-download-href="https://discuss.pytorch.org/uploads/default/b3af55454c0fd09001dce8edba016143a7b7050b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3af55454c0fd09001dce8edba016143a7b7050b.png" alt="image" data-base62-sha1="pDz7ehSXLF31TihRoWfX5ZdFWz1" width="690" height="105" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">819×125 5.42 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="2">
<li>By mistake you are equating the output of the model with itself for calculating loss</li>
</ol>
<pre><code class="lang-auto">    loss = criterion(dataOutput, dataOutput)
</code></pre>