Weights of cross entropy loss for validation/dev set

username: apganesh
cooked: <p>Hi All,<br>
I’m trying Deep learning network in pytorch for image classification and my dataset is class imbalanced. Hence I’ve applied the class weights while calculating the cross entropy loss during training.<br>
However, am having following doubt,<br>
Do we apply the class weights to the loss function for validation/dev set?<br>
If so, would it not mislead us from the actual target?<br>
Or<br>
We have to apply the weights to loss function only during training and not during validation?<br>
Pls clarify</p>

username: ptrblck
cooked: <p>I would assume the weights are only used during training to add a penalty for misclassified minority classes. I’m not sure if e.g. early stopping should also be used with a weighted validation loss, so maybe <a class="mention" href="/u/rasbt">@rasbt</a> could give a hint on this topic, as he’s really experienced in this domain.</p>

username: rasbt
cooked: <p>I recommend using a balanced dataset for evaluation because on an imbalanced dataset, e.g., accuracy is misleading. The loss function would only be applied during training then (I assume you want to measure sth like “accuracy” on the validation/test set).</p>
<p>Furthermore, to get more insights into the classification performance with and without loss weighting during training, i recommend looking at the confusion matrices on the validation/test sets when you train with and without weighting.</p>
<aside class="quote no-group" data-username="apganesh" data-post="1" data-topic="46132">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f4b2a3/48.png" class="avatar"> apganesh:</div>
<blockquote>
<p>Do we apply the class weights to the loss function for validation/dev set?<br>
If so, would it not mislead us from the actual target?</p>
</blockquote>
</aside>
<p>It depends on what you actual target is – that’s something you need to decide depending on the goal for which you train the model.</p>
<p>Lastly, a metric that might be helpful here would be the “average-per-class accuracy” – I’ve summarized it here: <a href="http://rasbt.github.io/mlxtend/user_guide/evaluate/scoring/" class="inline-onebox" rel="noopener nofollow ugc">scoring: computing various performance metrics - mlxtend</a></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/652041c9326d20fb1b5f24c5ceb81947b7590cf2.png" alt="29%20PM" data-base62-sha1="eqBkKQEEABZ59BYqIXrhRrLk8O6" width="639" height="425"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c9da974cb3bdc45e43765708cd5262d99536f35f.png" data-download-href="https://discuss.pytorch.org/uploads/default/c9da974cb3bdc45e43765708cd5262d99536f35f" title="35%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c9da974cb3bdc45e43765708cd5262d99536f35f_2_629x500.png" alt="35%20PM" data-base62-sha1="sNGiDGRIGCeYqiCexRlwCxGBJsb" width="629" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c9da974cb3bdc45e43765708cd5262d99536f35f_2_629x500.png, https://discuss.pytorch.org/uploads/default/original/2X/c/c9da974cb3bdc45e43765708cd5262d99536f35f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/c9da974cb3bdc45e43765708cd5262d99536f35f.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">35%20PM</span><span class="informations">640×508 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<blockquote>
<p>I’m not sure if e.g. early stopping should also be used with a weighted validation loss, so maybe</p>
</blockquote>
<p>I’d say the concept of early stopping is not necessarily related to the class imbalanced problem. It can be used if you notice an increasing degree of overfitting (measured on the validation set) when you continue training, but this can happen with/without class imbalance. I don’t think there’s a direct relationship between these two issues.</p>

username: apganesh
cooked: <p>Thanks for the detailed info on validation part. The classification of model performance improved after applying the class weights during training.</p>

username: Bhagyashree_Gaikwad
cooked: <p>My Train and Validation set comes from the exact same imbalanced class distribution. I am using weighted CrossEntropy loss function to calculate training loss. As it would be unfair to use weighted loss function for Validation, I interpret from above discussion, not to calculate validation loss and rely on Validation accuracy only. What criterion should I use for early stopping in this case? Or the plain CrossEntropy loss without weights can be reasonable for validation. Please let me know your suggestion. Thank you.</p>