Why does the loss stay constant?

otis.hongpu: <p>Hi there. I was wondering why does the loss in my self-build LeNet5 stay constant?<br>
Below is the code.</p>
<pre><code class="lang-auto"># LeNet5
class LeNet5(nn.Module):
    
    def __init__(self):
        super(LeNet5, self).__init__()
        self.model = nn.Sequential(
        nn.Conv2d(3, 6, kernel_size=3, stride=1),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.ReLU(),
        nn.Conv2d(6, 16, kernel_size=3, stride=1),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.ReLU(),
        
        nn.Flatten(),
        # MLP
        nn.Linear(400, 120, bias=True),
        nn.ReLU(),
        nn.Linear(120, 84, bias=True),
        nn.ReLU(),
        nn.Dropout(keep_prob),
        nn.Linear(84, 2),
        )
    def forward(self, x):
        x = self.model(x)
        x_softmax = F.softmax(x, dim=1)
        return x_softmax

# instance
model = LeNet5()
#  optimizer &amp; loss function
optimizer = optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9)
loss_function = nn.CrossEntropyLoss()

# 
def train(
    model,
    data_loader,
    optimizer,
    loss_function,
    epoch,
    log_interval=200    # print out info every 200 batches
):
    model.train()
    for batch_idx, (x, label) in enumerate(data_loader):
        
        optimizer.zero_grad()
        output = model(x)
        loss = loss_function(output, label)
        loss.backward()
        optimizer.step()
        if batch_idx % log_interval == 0:
            print("Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}".format(
                epoch, batch_idx*len(x), len(data_loader.dataset),
                100. * batch_idx/len(data_loader), loss.data.item()
                # "batch_idx/len(data_loader)" = 当前的batch index除以batch总数
            ))
# 
def validate(
    model, 
    data_loader,
    loss_function,
    epoch,
    loss_vector,
    accuracy_vector
):
    '''applied at train dataset as well as validate dataset'''
    model.eval()
    loss, correct = 0, 0
    for x, label in data_loader: # retrieve tensor dataset and label
        output = model(x)
        loss += loss_function(output, label).data.item()   # "loss_function" is pre-defined, which should be applied in this section
        pred = output.data.max(1)[1]
        correct += pred.eq(label.data).cpu().sum()
    
    # save cal
    loss /= len(data_loader)
#     print(data_loader.dataset)
    loss_vector.append(loss) # for plotting
    accuracy = 100. * correct.to(torch.float32) / len(data_loader.dataset)
    accuracy_vector.append(accuracy)
    print('Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        loss, correct, len(data_loader.dataset), accuracy
    ))
# 
def main(model, train_loader, validate_loader, optimizer, loss_function, num_epochs):
    lossv, accv = [], []
    train_lossv, train_accv = [], []
    for epoch in range(1, num_epochs+1):
        train(model, train_loader, optimizer, loss_function, epoch)
        print("Train set:")
        validate(model, train_loader, loss_function, epoch, train_lossv, train_accv)
        print("Validate set:")
        validate(model, validate_loader, loss_function, epoch, lossv, accv)
    return {'validate':[lossv, accv], 'train':[train_lossv, train_accv]}
</code></pre>
<p>I’ll really appreciate it if someone could give me a hint!</p>

KFrank: <p>Hi Hongpu!</p>
<aside class="quote no-group quote-modified" data-username="otis.hongpu" data-post="1" data-topic="169275" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/otis.hongpu/48/55434_2.png" class="avatar"> otis.hongpu:</div>
<blockquote>
<pre><code class="lang-auto">    def forward(self, x):
        x = self.model(x)
        x_softmax = F.softmax(x, dim=1)
        return x_softmax
...
loss_function = nn.CrossEntropyLoss()
</code></pre>
</blockquote>
</aside>
<p>Using <code>softmax()</code> with <code>CrossEntropyLoss</code> will significantly degrade your<br>
training (although it is unlikely to cause your loss to remain exactly constant).<br>
Try removing the <code>softmax()</code> and see if that helps.</p>
<p><code>CrossEntropyLoss</code> expects its <code>input</code> (the predictions you pass it) to be<br>
“unnormalized” <em>log-probabilities</em> (for example, the output of your final<br>
<code>Linear</code> layer).  Passing such through <code>softmax()</code> converts them to actual<br>
probabilities – not what <code>CrossEntropyLoss</code> wants.</p>
<p>Also, you don’t show us what your <code>learning_rate</code> is.  If it happened to be<br>
zero, your loss would remain constant.</p>
<p>If these ideas don’t help, try passing a single sample through your network<br>
and check that the output does depend on your input.  If that works, perform<br>
a single backward pass and check that you get non-zero gradients.  If that<br>
also works, perform a single <code>optimizer.step()</code> and check that your model<br>
parameters do get changed.  If they do, check that the updated parameters<br>
produce a different output for the original input.  If they do, check that the<br>
different output does, in fact, lead do a different loss.</p>
<p>(As an aside, it looks like your model predicts two values – your final<br>
<code>Linear</code> layer has <code>out_features = 2</code>.  It would likely be marginally more<br>
efficient to treat you problem as binary-classification problem, rather than<br>
a two-class multi-class problem, have your final layer be <code>Linear (84, 1)</code>,<br>
and use <code>BCEWithLogitsLoss</code> as your loss criterion.  Note, this isn’t the<br>
cause of our problem – it’s just a suggestion for a slight improvement.)</p>
<p>Good luck.</p>
<p>K. Frank</p>