Validation loss goes high with validation accuracy

username: luanpham
cooked: <p>Hi everyone, I met this situation and get the confusion. I have some questions to ask for your help?</p>
<ol>
<li>
<p>The validation loss is increasing when validation accuracy is increasing, too. So, the best model should be the lowest loss or highest accuracy?</p>
</li>
<li>
<p>If we choose the highest accuracy as the best model, then if we look at the losses, easy to see the overfitting scenarios (low training loss and high validation loss).</p>
</li>
<li>
<p>If we choose the lowest loss as the best model. Why sometimes the loss is high with the accuracy, it seems like have true predict but low confidence (aka. probability, I use cross-entropy loss). Then, is the high confidence is always good?</p>
</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a.png" data-download-href="https://discuss.pytorch.org/uploads/default/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a_2_586x500.png" alt="image" data-base62-sha1="iwG1tgnb3c58NqWEYtiLSGd03vc" width="586" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a_2_586x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/81d8c56a4d9a7adbd91f9485e77aee7afd1f5e2a.png 2x" data-dominant-color="FBF9FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">803×684 46.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: beaupreda
cooked: <p>Hi,</p>
<aside class="quote no-group" data-username="luanpham" data-post="1" data-topic="60412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luanpham/48/17475_2.png" class="avatar"> luanpham:</div>
<blockquote>
<p>The validation loss is increasing when validation accuracy is increasing, too. So, the best model should be the lowest loss or highest accuracy?</p>
</blockquote>
</aside>
<p>From little experience I have, I tend to take the best model as the one with the highest accuracy, if there is no overfitting (not your case).</p>
<aside class="quote no-group" data-username="luanpham" data-post="1" data-topic="60412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luanpham/48/17475_2.png" class="avatar"> luanpham:</div>
<blockquote>
<p>If we choose the highest accuracy as the best model, then if we look at the losses, easy to see the overfitting scenarios (low training loss and high validation loss).</p>
</blockquote>
</aside>
<p>Yes, you are right.</p>
<aside class="quote no-group" data-username="luanpham" data-post="1" data-topic="60412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/luanpham/48/17475_2.png" class="avatar"> luanpham:</div>
<blockquote>
<p>If we choose the lowest loss as the best model. Why sometimes the loss is high with the accuracy, it seems like have true predict but low confidence (aka. probability, I use cross-entropy loss). Then, is the high confidence is always good?</p>
</blockquote>
</aside>
<p>Yes, a model with high confidence in its predictions is preferable. It is also why the loss is often a better metric to look at when you want to determine if the model is overfitting (e.g. the accuracy does not differentiate from a prediction at probability 0.5 or 0.9, both can predict the correct class, but your loss does).</p>
<p>In your situation, I would try to modify the architecture of the model to get rid of the overfitting that you have.</p>
<p>Good luck!</p>

username: silvester
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry for reviving an old topic. I am training a model which has a high validation accuracy, but the training and the validation losses are high which show low confidence in the predictions. I wonder if there is a way to get a lower training loss as I am using Adam with the learning rate lr=0.003.</p>