Training loss is greater then val loss

2hyes: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e88945d55a3245fd8639c21beaeafa0a172fd0b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/e88945d55a3245fd8639c21beaeafa0a172fd0b4" title="KakaoTalk_20200609_095045404"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e88945d55a3245fd8639c21beaeafa0a172fd0b4_2_690x427.png" alt="KakaoTalk_20200609_095045404" data-base62-sha1="xb6StQLNbfOc9xfR7xgcHGY57Lu" width="690" height="427" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e88945d55a3245fd8639c21beaeafa0a172fd0b4_2_690x427.png, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e88945d55a3245fd8639c21beaeafa0a172fd0b4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/8/e88945d55a3245fd8639c21beaeafa0a172fd0b4.png 2x" data-dominant-color="929191"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">KakaoTalk_20200609_095045404</span><span class="informations">726×450 50 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi. My model is autoencoder. In the training phase, I draw plot points in every epoch. but I think training loss should be less then validation loss. but no matter what hyperparameter I use, training loss is greater.<br>
Is it underfitting? or Is it okay?</p>
<p>my code here:</p>
<pre><code class="lang-auto">from livelossplot import PlotLosses

def train(model, patch_train_loader, patch_val_loader, EPOCHS):
  loss_func = nn.MSELoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=1e-5)
  liveloss = PlotLosses()
  lr2_tr_loss = []
  lr2_val_loss = []
  for epoch in range(EPOCHS):
      model_losses, valid_losses = [], []
      running_loss = 0.0
      logs = {}
      prefix = ''
      model.train()
      # with train data
      for idx, (data,target) in enumerate(patch_train_loader):
          data = torch.autograd.Variable(data).to(device = device, dtype = torch.float)
          optimizer.zero_grad()
          pred = model(data)
          loss = loss_func(pred, data)

          # Backpropagation
          loss.backward()
          # update
          optimizer.step()
          model_losses.append(loss.cpu().data.item())
          logs[prefix + 'MSE loss'] = loss.item()  
          model_losses.append(loss.cpu().data.item())
          running_loss += loss.item()
          model_losses.append(loss.item())
          print(idx,"complete")
 
      ## with validation data(only nodefect)
      with torch.no_grad():
        for idx, (data,target) in enumerate(patch_val_loader):
            model.eval()
            data = torch.autograd.Variable(data).to(device = device, dtype = torch.float)
            pred = model(data)
            loss = loss_func(pred, data)
            valid_losses.append(loss.item())
            prefix = 'val_'
            logs[prefix + 'MSE loss'] = loss.item()
           
      lr2_tr_loss.append(np.mean(model_losses))
      lr2_val_loss.append(np.mean(valid_losses))
      liveloss.update(logs)
      liveloss.draw()
      print ("Epoch:", epoch+1, " Training Loss: ", np.mean(model_losses), " Valid Loss: ", np.mean(valid_losses))
      
      if epoch == EPOCHS -1:
        torch.save(model.state_dict(), path)
        return lr2_tr_loss, lr2_val_loss
</code></pre>
<p>Thank you.</p>

ptrblck: <p>It depends on your model and your training loss might be higher than the validation loss, e.g. if you are using dropout during training. This will decrease the capacity of your training model and might thus yield a higher loss.</p>
<p>Also, it might be possible that your data split is not completely random (or you were unlucky, if your dataset is small) so that easier samples are in the validation set.<br>
Of course this effect should disappear if your dataset is sufficiently large and if you split randomly.</p>