Validation Loss Lower than Train Loss

littlefly_1115: <p>Hi, I’m now working on a MLP regression model, and here’s my training code:</p>
<pre><code class="lang-auto">def train_per_epoch(network, training_set, batch_size, optimizer, loss_function, device):
    cum_loss = 0  # creating list to hold loss per batch
    train_loader = DataLoader(training_set, batch_size)  # defining dataloader
    network.train()  # defining model state
    # print('training...')
    for i, (data, targets) in enumerate(train_loader):
        data = data.to(device)  # sending data to device
        targets = targets.to(device)
        optimizer.zero_grad()   # zeroing optimizer gradients
        predictions = network(data)  # classifying instances
        loss = loss_function(predictions, targets)   # computing loss/how wrong our predictions are
        loss.backward()  # computing gradients/the direction that fits our objective
        optimizer.step() # optimizing weights/slightly adjusting parameters
        cum_loss = cum_loss+loss.item()  # adding loss per batch to list

    return cum_loss
</code></pre>
<p>Validation code</p>
<pre><code class="lang-auto">def validate(network, validation_set, batch_size, loss_function, device):
    cum_loss = 0  # creating a list to hold loss per batch
    network.eval()  # defining model state
    val_loader = DataLoader(validation_set, batch_size)  # defining dataloader
    # print('validating...')  # preventing gradient calculations since we will not be optimizing
    with no_grad():
        for i, (data, targets) in enumerate(val_loader):
            data= data.to(device)  # sending data and targets to device
            targets = targets.to(device)
            predictions = network(data)  # making predictions
            loss = loss_function(predictions, targets)  # computing loss
            cum_loss = cum_loss+loss.item()

    return cum_loss
</code></pre>
<p>Complete training code</p>
<pre><code class="lang-auto">def train_m2(epoch, network, train_data, valid_data, 
             batch_size, optimizer, loss_function, device, patience):
    train_losses, valid_losses = [], []
    start = time.time()
    det = False
    trigger = 0
    for i in range(epoch):
        train_loss = train_per_epoch(network, train_data, batch_size, optimizer, loss_function, device)
        valid_loss = validate(network, valid_data, batch_size, loss_function, device)
        train_losses.append(train_loss)
        valid_losses.append(valid_loss)

        # early stopping
        # If validation loss is consecutively lower than 1e-3 more than the patience, 
        # it stops training.
        if i &gt; 0 and valid_losses[-1] &lt;= 1e-3:
            trigger += 1
            det = True
            if trigger &gt;= patience:
                break
        elif det and valid_losses[-1] - valid_losses[-2] &lt;= 1e-3:
            trigger = 0
            det = False
    end = time.time()
    return train_losses, valid_losses, end-start
</code></pre>
<p>But my training curve shows like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56767c0ecf18d0754d1eea738d931b6b40fdfcad.png" alt="36_loss" data-base62-sha1="ckSUHvMFeSTbPCD2rM4AruLHsjb" width="480" height="360"></p>
<p>I have no idea why validation loss is lower than the training one. Can anyone tell me why this would happen? Thanks!</p>

Soumya_Kundu: <p>What is your data split like? Maybe your validation data just happens to be “simpler”.</p>

ptrblck: <p>This can often be the case if <code>dropout</code> layers are used as they are using models with less capacity during the training which could yield a higher loss.</p>