The gradients of the weights of my model approach to zero

Saidur_Pavel: <p>Hi, I am new to pytorch. The weights of my NN gradients become close to zero at the beginning, and my loss function does not change much. Any help or suggestions will be much appreciated.</p>
<p><strong>Problem description:</strong> My dataset contains two variables, namely ‘posterior’ and ‘theta’. The variable ‘posterior’ is the input of the NN, which is basically a probability distribution. As an example, For a particular observation, at the beginning, the ‘posterior’ is taken from a uniform distribution over 1801 grid points.  The final output will also like a probability distribution, where 9 of the grid points will be highly probable (close to 1) and remaining close to zero.</p>
<p>The NN will take the posterior input and will give a matrix A at the output. Then, based on the input  ‘theta’, a variable X, and then another variable y= AX will be computed and then, by doing some calculation, an updated posterior will be returned. The posterior will be updated for N samples of X.  BCELoss function is used to compute the loss between the updated posterior and the label.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d08a0389041ce619170241a52a672878326c8e2.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d08a0389041ce619170241a52a672878326c8e2" title="Screen Shot 2022-10-01 at 1.17.09 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d08a0389041ce619170241a52a672878326c8e2_2_690x302.png" alt="Screen Shot 2022-10-01 at 1.17.09 AM" data-base62-sha1="8HVBBtBV5yrOW0DQNM4X1xTpCVQ" width="690" height="302" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d08a0389041ce619170241a52a672878326c8e2_2_690x302.png, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d08a0389041ce619170241a52a672878326c8e2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d08a0389041ce619170241a52a672878326c8e2.png 2x" data-dominant-color="333C44"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-10-01 at 1.17.09 AM</span><span class="informations">758×332 99.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre data-code-wrap="def"><code class="lang-def">    
    class model(nn.Module):
        def __init__(self):
            super().__init__()
            
            
            self.input = nn.Linear(1801,2048)
            self.fc1 = nn.Linear(2048,2048)
            self.fc2 = nn.Linear(2048,2048)
            self.out = nn.Linear(2048,10*100)
            
        def forward(self,x):
            x = F.relu(self.input(x))
            
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))

            
            x = self.out(x)
            
            return x
            
    net = model()
    #lossfun = nn.BCEWithLogitsLoss()
    lossfun = nn.BCELoss()
    optimizer = torch.optim.Adam(net.parameters(),lr = .001)

            
    
    return net,lossfun,optimizer

def trainModel(train_loader,numepochs,samples,device):
    
    net,lossfun,optimizer = createModel()
    net.to(device)
    losses = np.zeros(numepochs)
    
    for epochi in range(numepochs):
        
        batchLoss = []
        
        for theta, posterior, label in train_loader:
            theta = theta.to(device)
            posterior = posterior.to(device)
            label = label.to(device)
            
            for samp in range(samples): # Iterate through the samples in X
            
            # Computing A matrix as the NN output and do some postprocessing
                A_tmp = net(posterior)
                A_tmp = A_tmp.cpu()
                A_shaped = A_tmp.view(-1,10,100)
                
                A_r =A_shaped[:,:,:50]
                A_i = A_shaped[:,:,50:]
                A = A_r+1j*A_i
                
                
                # Initializing a sample for X
                X = torch.zeros((theta.shape[0],50,1),dtype = torch.cfloat)
                
                # Computing X from theta for every data in a batchj
                for i in range(theta.shape[0]):
                    theta_loop = theta[i,:]
                    received_loop = ap.arr_received_tensor(50, theta_loop, 1, 20)
                    X[i,:,:] = received_loop # Array received signal
                    
                #Computing y
                y = A@X  
                
                
                posterior = torch.zeros((theta.shape[0],1801))
                

              # Updating posterior
                for i in range(y.shape[0]):
                    y_loop = y[i,:,:] # Grab i th y data from a batch of 32
                    Ry_loop = y_loop@y_loop.conj().T
                    posterior[i,:] = ap.op_angle_tensor(-90, 90, .1, Ry_loop, 9)[2]

                    
                print(f'Sample No: {samp}\n\n')
                    


            loss = lossfun(posterior,label)
 
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            
            batchLoss.append(loss.item())
            
        losses[epochi] = np.mean(batchLoss)
        print(f'Loss in epoch {epochi} is {losses[epochi]}')
        
    
    return losses,net ```</code></pre>

srishti-git1110: <p>Could you try using <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html" rel="noopener nofollow ugc">this</a> learning rate scheduler &amp; see if it helps?</p>
<p>Also, please try to post the code by enclosing it between 3 backticks ```.</p>

krypticmouse: <p>Could be dying relu problem can you check using <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.leaky_relu.html#torch.nn.functional.leaky_relu" rel="noopener nofollow ugc">leaky_relu</a> or any other relu variant?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/saidur_pavel">@Saidur_Pavel</a>,</p>
<p>Surely the issue is the fact your <code>posterior</code> tensor seems to be the phase of an imaginary number which will hold a degeneracy and hence has an undefined gradient.</p>
<p>Can you share what <code>ap</code> is? As well as <code>ap.op_angle_tensor</code>?</p>
<p>Also, as <a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> has already said copy and paste your code and wrap it with three backticks ```, so your code snippet is more readable!</p>

Saidur_Pavel: <p>Thanks for your suggestion. I tried learning rate scheduler, but still it seems no improvement</p>

Saidur_Pavel: <p>Thanks for your suggestion. I tried using leakyrelu instead of relu, but still the issue remains. I will also try other variants to see if it solve the issue.</p>

Saidur_Pavel: <p>Hi, Thanks for your comment. I did not fully get your comment, " <code>posterior</code> tensor seems to be the phase of an imaginary number". To me, it seems ‘posterior’ tensor is real valued.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fc191fbd07cfb5ed24f6d6ec624e278d5048489.png" data-download-href="https://discuss.pytorch.org/uploads/default/8fc191fbd07cfb5ed24f6d6ec624e278d5048489" title="Screen Shot 2022-10-01 at 10.03.14 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc191fbd07cfb5ed24f6d6ec624e278d5048489_2_690x218.png" alt="Screen Shot 2022-10-01 at 10.03.14 AM" data-base62-sha1="kvJ02nzYsigSTYWgVORrjuuYjbj" width="690" height="218" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8fc191fbd07cfb5ed24f6d6ec624e278d5048489_2_690x218.png, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fc191fbd07cfb5ed24f6d6ec624e278d5048489.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8fc191fbd07cfb5ed24f6d6ec624e278d5048489.png 2x" data-dominant-color="2F3941"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-10-01 at 10.03.14 AM</span><span class="informations">962×304 84.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also ‘ap’ is another python file, where I wrote the function op_angle_tesnor. I included the fucntion here for convenience.</p>
<pre><code class="lang-auto">def steer_tensor(n,d,phi):
    
    # Provide array manifold 
    # n = numer of Antenna
    # d = antenna separation 
    # phi = doa's with size n x 1
    
    pi = torch.tensor(math.pi)
    fac = pi/180
    out = torch.exp(-1j*2*pi*torch.arange(n).reshape(n,1)*d*torch.sin(fac*phi));
    return out

def op_angle_tensor(low,high,interval,R,n):
    
    # This function generate spectrum and extracts estimated angles
    
    
    # low = lower range of theta
    # high = higher tange of theta
    # interval = interval for discretizing grid
    # R = covariance matrix
    # n = number of sources
    
    theta_r = torch.round(torch.arange(low,high+interval,.1),decimals = 1)
    theta_r = theta_r.reshape(1,theta_r.shape[0])
    p = torch.zeros(theta_r.shape)
    n_antenna = R.shape[0]
    
    for index,ii in enumerate(theta_r[0]):
        aa = steer_tensor(n_antenna,.5,ii)
        p[:,index] = torch.abs(1/(aa.conj().T@torch.inverse(R)@aa))
    
    p = p/torch.sum(p) # This will act as updated posterior
    sorted_ind = torch.argsort(p)
    theta_o = theta_r[:,sorted_ind[:,-n:]]
    
    
    return theta_r[0],np.sort(theta_o)[0,0],p[0] ```</code></pre>

srishti-git1110: <p>I see.<br>
What <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> has suggested makes sense to me.<br>
Could you please share what they asked for?<br>
Probably the error could be solved then.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/saidur_pavel">@Saidur_Pavel</a>,</p>
<p>Yeah I should’ve written complex instead of imaginary but my point still stands. The posterior tensor seems to be a real component of a complex value. The initial point I made about degeneracy is that it seems you’re trying to differentiate the phase of a complex number which isn’t uniquely definied.</p>
<p>However, after looking at your <code>ap.op_angle_tensor</code> why are you calling <code>np.sort(theta_o)</code> instead of <code>torch.sort(theta_o)</code>? That’ll surely break your graph which also make explain your gradients being zero?</p>

parthshah231: <p>Could you also try reducing your <code>initial_lr</code> if you are using say <code>1e-3</code> try <code>1e-4</code> maybe?</p>

AlphaBetaGamma96: <p><a class="mention" href="/u/saidur_pavel">@Saidur_Pavel</a> in fact, given the gradient is near zero you could try starting with a larger learning rate to see if you’re suffering from the vanishing gradient problem. But do check why you’re using <code>np.sort</code> instead of a torch equivalent operation, as that’ll break your graph and hence your gradient.</p>