Higher accuracy in the validation dataset

username: azer
cooked: <p>Hi all,</p>
<p>I am training resnet18 for  image classification and I got good results but accuracy of the validation dataset is higher than the training dataset.</p>
<p>| train loss: 0.0059 | train acc 0.9621 | val loss: 0.007485 | val_acc:  0.9771</p>
<p>For the train dataset I used the weighted cross entropy  as loss function since my dataset was imbalanced.</p>
<p>Any help is appreciated</p>

username: KFrank
cooked: <p>Hi Azeai!</p>
<aside class="quote no-group" data-username="azer" data-post="1" data-topic="132563" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azer/48/43253_2.png" class="avatar"> azer:</div>
<blockquote>
<p>accuracy of the validation dataset is higher than the training dataset.</p>
</blockquote>
</aside>
<p>There are a number of possible reasons for this:</p>
<p>(First, a quick note:  One would normally expect your model to perform<br>
at least a little better on the training data.)</p>
<p>The character of your training and validation datasets could be<br>
different.  For example, maybe your validation images were collected<br>
under better conditions and are less noisy.  Then they might be easier<br>
to classify so you would get a higher accuracy.  Or maybe certain<br>
classes are easier to classify than others and your validation dataset<br>
has more samples of the easier classes in it.  There’s nothing<br>
fundamentally wrong with using distinctly different datasets for training<br>
and validation, but the accuracies won’t be directly comparable.</p>
<p>It’s often the case that you have one large dataset that you randomly<br>
split into training and validation (and possibly test) datasets.  Now<br>
your training and validation datasets will have the same character.<br>
However, by happenstance, it could be that your validation dataset<br>
nonetheless has more easier samples in it, so you get a higher<br>
validation accuracy.  You could test this by retraining your model<br>
and performing the training / validation comparison multiple times<br>
with different random splits for the training and validation datasets.<br>
Even if for some specific runs you have (by happenstance) a higher<br>
validation accuracy, you should find, on average, that your validation<br>
accuracy is lower than your training accuracy.</p>
<p>If you use your validation dataset to <em>select</em> your final model, you<br>
could bias your model to perform better on the validation dataset.<br>
To be concrete, let’s say that your training protocol is to train for 100<br>
epochs, and you keep as your final model the model from each of<br>
the last 25 training epochs that has the highest validation-dataset<br>
accuracy (or perhaps the lowest validation-dataset loss).  This is not<br>
an unreasonable thing to do – the quality of a model can jump around<br>
during training, and you’re trying to pick the best one – but the chosen<br>
model will be biased to perform especially well on the validation dataset,<br>
even if it doesn’t perform as well (and it likely won’t) on an independent<br>
test dataset.</p>
<p>If your model has separate “training” and “evaluation” modes (for<br>
example, because it uses things like batch normalization or dropout),<br>
your training-mode predictions and evaluation-mode predictions will<br>
be different (even for the same input sample).  If you want your training<br>
and validation accuracies (and losses) to be directly comparable, you<br>
can’t just collect your training accuracy during your training iterations;<br>
you have to switch from training mode to evaluation mode when you<br>
compute the training accuracy that you want to compare with your<br>
validation accuracy.</p>
<p>Of course, you could have a bug somewhere that throws off one of<br>
the accuracy calculations.  There is no magic way to defend against<br>
this – you just have to double-check your code and do some spot<br>
checks of various results.</p>
<aside class="quote no-group">
<blockquote>
<p>For the train dataset I used the weighted cross entropy  as loss function since my dataset was imbalanced.</p>
</blockquote>
</aside>
<p>This will not affect whether your training and validation accuracies<br>
are comparable – your loss function doesn’t (directly) affect your<br>
accuracy calculation.  However, if you want your training and<br>
validation <em>losses</em> to be comparable, you should use the same<br>
loss function – including any class weights – for both.  (If your<br>
training and validation datasets have the same character, they<br>
will, in particular, have the same class imbalance, and it would<br>
not be inappropriate to use class weights for your validation loss<br>
calculation.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Asu_Guvenli
cooked: <p>This is also the case in transfer learning tutorial. Can this be because we are augmenting our data? Meaning in the training set there are pictures that are cropped and so on. Hence it is harder to guess what they actually are.</p>