Gradient not flowing throughout the entire process but loss is reducing

seankala: <p>Hi. I’m not exactly sure if this might be the appropriate place to post this but figured I could use any help I can get.</p>
<p>I’m currently trying to train a neural network model but have noticed that it’s not training properly. I noticed this because I’m currently aiming to train it to perform binary classification, and after seeing that it only predicts 1 during test time I tried to train it using data with labels that are only 0. However, it still managed to predict mainly 1’s (95% of the time in fact).</p>
<p>After seeing this, I plotted the average gradients per layer for each epoch, and the plot looks something like this:</p>
<p><a href="https://i.stack.imgur.com/7SGl0.png" rel="nofollow noopener"><img src="https://i.stack.imgur.com/7SGl0.png" alt="enter image description here" width="640" height="480"></a></p>
<p>The overall code framework  is something as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self, num, dim):
        self.features_linear = nn.Linear(in_features=300, out_features=100)
        self.embedding = nn.Embedding(num_embeddings=num, embedding_dim=10)
        self.output_linear = nn.Linear(in_features=(100 + num), out_features=1)
        self.dropout = nn.Dropout(p=0.8)

    def forward(self, a, b):
        a2 = F.relu(self.features_linear(a))
        b_emb = self.embedding(b)
        emb_sum = torch.sum(b_emb, dim=1)

        input_rep = torch.cat((a2, emb_sum), dim=1)
        input_rep = self.dropout(input_rep)
        output = self.output_linear(input_rep)

        return output


model = Model(16, 100)

if torch.cuda.is_available():
    model = model.to('cuda')

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(params=model.parameters())

for epoch in range(num_epochs):
    for batch in training_loader:
        optimizer.zero_grad()

        a, b, label = batch

        if torch.cuda.is_available():
            a = a.to('cuda')
            b = b.to('cuda')
            label = label.to('cuda')

        output = model(a, b)

        loss = criterion(output, label)
        epoch_loss += loss.item()
        loss.backward()
        optimizer.step()
</code></pre>
<p>What’s especially perplexing is that the loss value decreases rather ideally (I don’t have a plot for that ATM).</p>
<p>Would anyone have any idea on what I might be doing wrong? Any tips or thoughts are appreciated. Thanks.</p>

pfloat: <p>Hello,<br>
are your input (images?) rescaled between 0 and 1 ?<br>
I saw that your model output is 1. Do you mean to use sigmoid activation ? if so, then it is not included in <code>BCELoss</code>.</p>
<p>If that is the problem, some parts of this discussion may help:</p><aside class="quote" data-post="1" data-topic="33586">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shani_gamrian/48/1299_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bceloss-vs-bcewithlogitsloss/33586">BCELoss vs BCEWithLogitsLoss</a> 
  </div>
  <blockquote>
    What is the advantage of using binary_cross_entropy_with_logits (aka BCE with sigmoid) over the regular binary_cross_entropy? I have a multi-binary classification problem and I’m trying to decide which one to choose.
  </blockquote>
</aside>


seankala: <p>Hi <a class="mention" href="/u/pfloat">@pfloat</a> sorry for the late reply. I am using the sigmoid activation function right now. Perhaps I could change it to ReLU and see how that works.</p>