Validation loss extremely large but good results

lorenzo_fabbri: <p>I’m training a simple self-attention model and I’m obtaining some good results on the validation set (in terms of accuracy, MCC, recall and precision). I’ve done this doing a train/test split several times. The <em>only</em> problem is that the validation loss is extremely large compared to the training loss. I’m attaching an example but they more or less all look the same:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4ef2156ad3da67fd8acd79c2c6e951dcbec63871.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ef2156ad3da67fd8acd79c2c6e951dcbec63871" title="Screenshot%20from%202019-09-14%2018-00-50"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ef2156ad3da67fd8acd79c2c6e951dcbec63871_2_690x250.png" alt="Screenshot%20from%202019-09-14%2018-00-50" data-base62-sha1="bgnU54XoskIBrmtRtcdl6hsSDbX" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ef2156ad3da67fd8acd79c2c6e951dcbec63871_2_690x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4ef2156ad3da67fd8acd79c2c6e951dcbec63871_2_1035x375.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4ef2156ad3da67fd8acd79c2c6e951dcbec63871.png 2x" data-dominant-color="F5F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-14%2018-00-50</span><span class="informations">1184×430 70.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I train it for like 500 epochs, the validation loss keep decreasing nicely (while the training loss oscillates more) but it’s always much larger. Has anyone seen something similar before?</p>
<p>I’m also attaching the training and validation loop:</p>
<pre><code>training_loss = []
val_loss = []

for epoch in range(1, num_epochs+1):
    #print(f'EPOCH: {epoch}...')

    model.train()

    avg_loss = 0.0
    for idx, batch in enumerate(train_loader):
        smiles, labels = batch[0].to(device), batch[1].to(device)

        # Fit
        optimizer.zero_grad()

        out = model(smiles)

        loss = criterion(out, labels)
        avg_loss =+ loss.item() * smiles.size(0)
        loss.backward()

        optimizer.step()

    training_loss.append(avg_loss / len(train_loader))

    # Validation
    model.eval()
    with torch.no_grad():

        avg_loss = 0.0
        y_pred = []
        y_val = []
        for idx, batch in enumerate(val_loader):
            smiles, labels = batch[0].to(device), batch[1].to(device)

            out = model(smiles)
            y_val.extend(list(labels.detach().cpu().numpy()))
            y_pred.extend(list(torch.argmax(out, dim=1).detach().cpu().numpy()))

            loss = criterion(out, labels)
            avg_loss =+ loss.item() * smiles.size(0)

    val_loss.append(avg_loss / len(val_loader))
</code></pre>

lorenzo_fabbri: <p>I believe I found the error: I did <code>=+</code> rather than <code>+=</code> when computing the average loss. Now I get:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/2a59387fe7c99d4de5aadeae25b5a1c0747b35df.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a59387fe7c99d4de5aadeae25b5a1c0747b35df" title="Screenshot%20from%202019-09-14%2018-37-54"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2a59387fe7c99d4de5aadeae25b5a1c0747b35df_2_690x250.png" alt="Screenshot%20from%202019-09-14%2018-37-54" data-base62-sha1="62DcUrbAixVOF8G62IwJamaOcKX" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/2a59387fe7c99d4de5aadeae25b5a1c0747b35df_2_690x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/2a59387fe7c99d4de5aadeae25b5a1c0747b35df_2_1035x375.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/2a59387fe7c99d4de5aadeae25b5a1c0747b35df.png 2x" data-dominant-color="F9FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-09-14%2018-37-54</span><span class="informations">1175×426 38.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Nonetheless, they are now both very large! The scores are always the same, obviously. So now the question is: does it matter if the loss is <em>high</em> (I’m using cross-entropy)?</p>

ptrblck: <p>It seems you are multiplying by the batch size, which would thus accumulate the loss of all samples in <code>avg_loss</code>.<br>
If that’s the case, you should divide by the number of samples to get the average loss, bot the length of the <code>DataLoader</code>, which will return the number of batches.</p>