Validation loss is way Lower than the training loss

ays: <p>Hi,<br>
im trading my model, and the validation loss seems to be way smaller than the training loss. So at test time the model does not perform well. im not sure why that is. Please any suggestions?</p>
<p>Below is the image of both losses. The top is the training loss while the bottom is the validation loss.<br>
Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9eb81b4af9002774789da7379f9d98f1c181494.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b9eb81b4af9002774789da7379f9d98f1c181494" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9eb81b4af9002774789da7379f9d98f1c181494_2_690x284.jpeg" alt="image" data-base62-sha1="qwITY6liMtXYau4Kh3ajMb9rNsM" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9eb81b4af9002774789da7379f9d98f1c181494_2_690x284.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9eb81b4af9002774789da7379f9d98f1c181494_2_1035x426.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9eb81b4af9002774789da7379f9d98f1c181494_2_1380x568.jpeg 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2842×1172 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Ays!</p>
<aside class="quote no-group" data-username="ays" data-post="1" data-topic="189511" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ays/48/56534_2.png" class="avatar"> ays:</div>
<blockquote>
<p>im not sure why that is. Please any suggestions?</p>
</blockquote>
</aside>
<p>There could be any number of reasons.</p>
<p>Perhaps you’re computing your loss with <code>sum()</code> (instead of <code>mean()</code>) and<br>
your validation batch size is smaller than your training batch size so summing<br>
over fewer batch elements gives you a smaller loss.</p>
<p>Perhaps you’re using <code>model.train()</code> and <code>model.eval()</code> and, for example,<br>
turning on such things as <code>Dropout</code> layers makes the performance of your<br>
model worse in training mode.</p>
<p>Perhaps the character of your evaluation dataset is different than that of<br>
your training set.  If your evaluation dataset happened to contain a lot more<br>
“easy” data samples than your training dataset, those easy samples could<br>
lead to a lower loss.</p>
<p>Or perhaps you just have a bug in your code somewhere.</p>
<p>What happens if you pretend that your training dataset is your validation<br>
dataset and use it to compute your “validation” loss.  What size loss values<br>
do you get then?</p>
<p>Best.</p>
<p>K. Frank</p>

J_Johnson: <p>You want loss to be lower. That’s a sign the model is learning. So it might not be a bad thing.</p>
<p>Here are some additional points you can check, in addition to what <a class="mention" href="/u/kfrank">@KFrank</a> mentioned.</p>
<ol>
<li>Are you using dropout layers? If so, perhaps try lowering your p value to something in the range of 0.1 to 0.3.</li>
<li>Are you using data augmentations in your preprocessing? Perhaps try lowering their thresholds.</li>
<li>Does your training and validation sets have an equal representation of classes? If not, you might try adding positional weights to your loss function.</li>
</ol>