Why does validation loss increase while training loss decreases?

Asya: <p>my validation loss increases while training loss decreases, and accuracy increases for both validation and training</p>
<p>I’m trying to find a reason why could that be and how to fix it. Is it overfitting or can it be something else?</p>
<ul>
<li>
<p><code>validation loss:</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/944a45d64889fc9adae0d128692f610d9ef470ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/944a45d64889fc9adae0d128692f610d9ef470ad" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/944a45d64889fc9adae0d128692f610d9ef470ad_2_240x175.png" alt="image" data-base62-sha1="l9PNfyhvftmKg5sC3G0Q12Tudch" width="240" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/944a45d64889fc9adae0d128692f610d9ef470ad_2_240x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/944a45d64889fc9adae0d128692f610d9ef470ad_2_360x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/944a45d64889fc9adae0d128692f610d9ef470ad_2_480x350.png 2x" data-dominant-color="F8FBFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">872×634 66.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p><code>train loss:</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1bb6d7441d5658aac1b273827b693bb1c6437970.png" data-download-href="https://discuss.pytorch.org/uploads/default/1bb6d7441d5658aac1b273827b693bb1c6437970" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bb6d7441d5658aac1b273827b693bb1c6437970_2_241x175.png" alt="image" data-base62-sha1="3XaCQdvKEy2hoAqsykkFiwqUoBW" width="241" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bb6d7441d5658aac1b273827b693bb1c6437970_2_241x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bb6d7441d5658aac1b273827b693bb1c6437970_2_361x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1bb6d7441d5658aac1b273827b693bb1c6437970_2_482x350.png 2x" data-dominant-color="FCFDFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">852×618 40 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p><code>validation accuracy:</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/f/1f94074288992601df73b522403c13c544b3f166.png" data-download-href="https://discuss.pytorch.org/uploads/default/1f94074288992601df73b522403c13c544b3f166" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f94074288992601df73b522403c13c544b3f166_2_237x175.png" alt="image" data-base62-sha1="4vlWXbppTxTRb1z9iIMhQKDEcSO" width="237" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f94074288992601df73b522403c13c544b3f166_2_237x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f94074288992601df73b522403c13c544b3f166_2_355x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/f/1f94074288992601df73b522403c13c544b3f166_2_474x350.png 2x" data-dominant-color="FDFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">852×628 29.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p><code>train accuracy:</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75_2_238x175.png" alt="image" data-base62-sha1="u4p7QfI3B06PLv9rEMXB9DDm2Al" width="238" height="175" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75_2_238x175.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75_2_357x262.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c0b32ecf7ebf8c55e8dea7182e90d1fe537c75_2_476x350.png 2x" data-dominant-color="FEFEFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">866×634 24.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ul>

lxm-001: <p>it seems like the overfitting problem,and you need to check the function of validation func.</p>

Asya: <p>this is my function, is there anything I can fix here?</p>
<pre><code class="lang-auto">def train_model(model, criterion, optimizer, num_epochs):

    best_acc = 0.0

    for epoch in range(num_epochs):
        print("Epoch {}/{}".format(epoch, num_epochs))
        print('-' * 10)

        loss_train = 0
        loss_val = 0
        acc_train = 0
        acc_val = 0
        min_valid_loss = np.inf

        # model.train(True)

        for batch, (inputs, labels) in enumerate(train_data):

            inputs, labels = inputs.cuda(), labels.cuda()

            # Clear the gradients
            optimizer.zero_grad()
            # Forward Pass
            outputs = model(inputs)
            # Find the Loss
            loss = criterion(outputs, labels)
            # Calculate gradients
            loss.backward()
            # Update Weights
            optimizer.step()
            # Calculate Loss
            loss_train += float(loss)

            _, preds = torch.max(outputs.data, 1)

            acc_train += int(torch.sum(preds == labels.data)) / len(preds)

        avg_loss = loss_train / len(train_data)
        avg_acc = acc_train / len(train_data)


        for batch, (inputs, labels) in enumerate(val_data):

            inputs, labels = inputs.cuda(), labels.cuda()

            # Forward Pass
            outputs = model(inputs)
            # Find the Loss
            loss = criterion(outputs, labels)
            # Calculate Loss
            loss_val += loss.item()

            _, preds = torch.max(outputs.data, 1)

            acc_val += int(torch.sum(preds == labels.data)) / len(preds)

            del inputs, labels, outputs, preds
            torch.cuda.empty_cache()

        avg_loss_val = loss_val / len(val_data)
        avg_acc_val = acc_val / len(val_data)

        print()
        print("Epoch {} result: ".format(epoch))
        print("Avg loss (train): {:.4f}".format(avg_loss))
        print("Avg acc (train): {:.4f}".format(avg_acc))
        print("Avg loss (val): {:.4f}".format(avg_loss_val))
        print("Avg acc (val): {:.4f}".format(avg_acc_val))
        print('-' * 10)
        print()

        if avg_acc_val &gt; best_acc:
            best_acc = avg_acc_val
            best_model_wts = copy.deepcopy(model.state_dict())


    print("Best acc: ", best_acc)

    model.load_state_dict(best_model_wts)
    return model
</code></pre>

lxm-001: <p>it has some problems ,first , it seems you not change to eval model ,try to use model.eval(), and you not stop the backward ,try to use " with torch.no_grad()"</p>

lxm-001: <p>and you even not clear the loss in the last time in the eval model,maybe it is the issue of your problem</p>

Asya: <p>where should I change for <code>modal.eval()</code> mode? for training or for validation?</p>
<p>and I did cleaning like that before</p>
<pre><code class="lang-auto">    del inputs, labels, outputs, preds
    torch.cuda.empty_cache()
</code></pre>
<p>after training part, but it didn’t help</p>

Asya: <p>so I tried different ways with switching eval mode and added <code>with torch.no_grad()</code> but it still didn’t improve anything <img src="https://discuss.pytorch.org/images/emoji/apple/crying_cat_face.png?v=10" title=":crying_cat_face:" class="emoji" alt=":crying_cat_face:"></p>

Gorgen: <p>hello. sorry to disturb you. I meet the same issue. Could you please give me some advice how to solve it? Thanks, Best wishes</p>

J_Johnson: <ol>
<li>
<p>Try adding dropout layers with p=0.25 to 0.5.</p>
</li>
<li>
<p>Add augmentations to the data(this will be specific to the dataset you’re working with).</p>
</li>
<li>
<p>Increase the size of your training dataset.</p>
</li>
<li>
<p>Alternatively, you can try a high learning rate and batchsize(See super convergence). <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.OneCycleLR.html" class="inline-onebox" rel="noopener nofollow ugc">OneCycleLR — PyTorch 1.11.0 documentation</a></p>
</li>
</ol>