The validation set loss obtained from the loaded model weights is not the same as the validation loss from the training process

Graviton: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/513450d4f848f6b57cb0eb8f16c1466276f497d4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/513450d4f848f6b57cb0eb8f16c1466276f497d4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/513450d4f848f6b57cb0eb8f16c1466276f497d4_2_690x68.jpeg" alt="image" data-base62-sha1="bAmLbBpsh4etdoARa9wQiQmj0NK" width="690" height="68" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/513450d4f848f6b57cb0eb8f16c1466276f497d4_2_690x68.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/513450d4f848f6b57cb0eb8f16c1466276f497d4_2_1035x102.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/513450d4f848f6b57cb0eb8f16c1466276f497d4_2_1380x136.jpeg 2x" data-dominant-color="383737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2338×232 161 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As shown above, the validation loss obtained from the loaded weights and the training loss obtained from the training process.A simplified version of the source code is shown below</p>
<pre data-code-wrap="python"><code class="lang-python">mean_valid_loss = []
valid_y = [None]*3
for step,(X_batch, y_batch, dtw) in enumerate(val_dataloader):
    loss, y_hat = engine.valid(X_batch,y_batch,dtw)
    mean_valid_loss.append(loss)
    if step==0:
        valid_y[0] = y_batch
        valid_y[1] = y_hat[0]
        valid_y[2] = y_hat[1]
    else:
        valid_y[0] = np.concatenate([valid_y[0], y_batch],axis=0)
        valid_y[1] = np.concatenate([valid_y[1], y_hat[0]],axis=0)
        valid_y[2] = np.concatenate([valid_y[2], y_hat[1]],axis=0)
mean_valid_loss = np.mean(mean_valid_loss)
valid_coverage, valid_width = compute_coverage(valid_y[0], valid_y[2], valid_y[1])

  
if mean_valid_loss &lt;= min_valid_loss:
    overfitting_epoch = 0
    min_valid_loss = mean_valid_loss
    torch.save(deepcopy(engine.model), save_dir + 'best_model.pth') 
else:
    overfitting_epoch+=1
if overfitting_epoch&gt;=early_stopping:
    break
logger.info("EPOCH: {:3d} TRAIN LOSS: {:.4f} TRAIN UPPER LOSS: {:.4f} TRAIN LOWER LOSS: {:.4f} VALID LOSS: {:.4f} MIN VALID LOSS: {:.4f}".format(epoch, mean_train_loss,mean_train_upper_loss, mean_train_lower_loss,mean_valid_loss, min_valid_loss))
 
# check_point = torch.load(save_dir + 'best_model.pth')
engine.model = torch.load(save_dir + 'best_model.pth')

mean_valid_loss = []
valid_y = [None]*3
for step,(X_batch, y_batch, dtw) in enumerate(val_dataloader):
  loss, y_hat = engine.valid(X_batch,y_batch,dtw)
  mean_valid_loss.append(loss)
  if step==0:
      valid_y[0] = y_batch
      valid_y[1] = y_hat[0]
      valid_y[2] = y_hat[1]
  else:
      valid_y[0] = np.concatenate([valid_y[0], y_batch],axis=0)
      valid_y[1] = np.concatenate([valid_y[1], y_hat[0]],axis=0)
      valid_y[2] = np.concatenate([valid_y[2], y_hat[1]],axis=0)
mean_valid_loss = np.mean(mean_valid_loss)
print("After loading model valid loss:",mean_valid_loss)
</code></pre>
<p><code>model.eval()</code> and <code>torch.no_grad()</code> have been added to <code>engine.valid()</code> and <code>engine.test()</code></p>