How can overfitting be detected using weighted binary cross entropy?

Maryam_Khaliji: <p>A balanced dataset allows us to detect overfitting or under fitting by comparing train and validation error. But is this comparison meaningful for imbalanced dataset when we use class-weight in binary cross entropy loss calculation?<br>
How we can find the best model?</p>

ptrblck: <p>Weighting a loss allows you to use the actual loss value as a proxy for the metric you care about.<br>
E.g. assume you are dealing with an imbalanced dataset containing 99% <code>class0</code> and only 1% <code>class1</code> samples. If your model does not learn the underling distribution but only returns <code>class0</code> as the predicted class, the loss would have a low value and the accuracy would already be at 99%, which might look great but it quite useless. This case is also known as the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy paradox</a>.<br>
To avoid this, you would try to force the model to learn the actual class distribution by using a weighted loss and might want to check another metric (e.g. the F1-score instead of the accuracy). The loss still acts as a proxy for your metric, so I would assume you would also use it during the validation phase.<br>
Let’s see if other users have other opinions about it.</p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="Maryam_Khaliji" data-post="1" data-topic="169317" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryam_khaliji/48/45054_2.png" class="avatar"> Maryam_Khaliji:</div>
<blockquote>
<p>But is this comparison meaningful for imbalanced dataset when we use class-weight in binary cross entropy loss calculation?</p>
</blockquote>
</aside>
<p>Yes, provided that what you call your “error” is the loss criterion you use<br>
for training (in your case weighted binary cross entropy), or, if your error<br>
is not your actual training loss, then at least – to use <a class="mention" href="/u/ptrblck">@ptrblck</a>’s language –<br>
your loss is a good “proxy for the metric you care about.”</p>
<p>The short story is that provided your training and validation datasets are<br>
similar in character and you use the same loss criterion for both, including<br>
class weights, if you have them, then divergence between your validation<br>
and training losses indicates overfitting.</p>
<p>In a narrow technical sense you can still talk about overfitting even if your<br>
training loss isn’t a good match for the “metric you care about” (although in<br>
such a case you won’t be training your model to do what you really want<br>
it to do) – it’s still logically consistent to say that you are overfitting your<br>
training set with respect to the loss criterion you are training with.</p>
<p>What matters is that your training and validation data sets have the same<br>
character and, in the case of unbalanced data, have the same relative<br>
class weights.  The cleanest way to do this – if you can, which you generally<br>
should be able to – is to have your training and validation data sets come<br>
from the same larger data set.</p>
<p>Let’s say you have 10,000 individual data samples.  If you randomly split<br>
this data set into a training set of, say, 8,000 samples and a validation set<br>
of 2,000 samples, the two datasets will be statistically identical.  In particular,<br>
they will have (statistically*) the same relative class weights,</p>
<p>Train your training set with a loss criterion of weighted binary cross entropy<br>
and also track the same weighted binary cross entropy on your validation<br>
set.  If your validation-set loss starts going up, even as your training-set loss<br>
keeps going down, overfitting has set in, and further training is actually<br>
making your model worse, rather than better.  (Note, if the “metric you care<br>
about” doesn’t track your training loss, then your model may or may not<br>
perform well on the “metric you care about,” but that’s a separate issue<br>
from overfitting.)</p>
<p>*)  Let’s say that your whole dataset of 10,000 contains 1% positive<br>
samples – that is 100 positive samples – and you randomly split it 80/20.<br>
Then your validation set of 2,000 samples will contain <em>about</em> 20 positive<br>
samples.  But this number could easily range from something like 15 to<br>
25 positive samples, which could be enough of a difference to somewhat<br>
affect your validation-loss computation.  If you want to be fancier, you<br>
could separately randomly split your positive and negative samples 80/20<br>
so that your validation set contains exactly 20 positive samples (and your<br>
training set, 80).  The smaller your dataset and the greater your class<br>
imbalance, the more this nuance will matter.)</p>
<p>Best.</p>
<p>K. Frank</p>

Maryam_Khaliji: <p>The plot shows the loss function values. Which epoch shows the best model or, in other words, when should I stop learning?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17b646f5ca87b1805561ce4915ab7358e168991d.png" alt="image" data-base62-sha1="3nLvybNIWJYMZF7OkFYPOdNg10F" width="387" height="376"></p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="Maryam_Khaliji" data-post="4" data-topic="169317" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryam_khaliji/48/45054_2.png" class="avatar"> Maryam_Khaliji:</div>
<blockquote>
<p>Which epoch shows the best model</p>
</blockquote>
</aside>
<p>Your loss plot is somewhat noisy, so it’s a bit of a judgment call.  To my eye<br>
I would say:</p>
<p>It looks to me that your training loss is going down systematically throughout<br>
your entire plot.  Your validation loss appears to be going down up to about<br>
iteration 12.  Mentally trying to average some of the noise away, I would say<br>
that your validation starts drifting up after iteration 12, but you could argue<br>
that there’s a plateau from iteration 12 up to about iteration 25 or 30.  Your<br>
“best” model almost certainly occurs between iterations 12 and 30.</p>
<p>A legitimate rule of thumb for picking a good candidate for your best model<br>
is to choose the model with the lowest validation loss – in your case<br>
seemingly that at iteration 12.  This won’t necessarily be your best model,<br>
but based just on the information in your loss plot, it’s probably the best<br>
choice you can make.</p>
<p>Having said that, depending on the complexity of your model (and the size<br>
of your training set), 50 epochs of training is not a lot.  If you can afford the<br>
training time, it would be prudent to continue training significantly longer,<br>
perhaps for thousands of epochs.  In all probability, your training loss will<br>
continue to trend down and then flatten out, while your validation loss<br>
continues to increase as you continue to overfit.  But it’s possible that your<br>
training is temporarily “stuck” and as you continue training it will find a better<br>
“direction” to explore, causing your validation loss to start going down again.</p>
<p>(Also try experimenting with your training hyperparameters, such as learning<br>
rate and momentum, and your choice of optimizer.)</p>
<p>Best.</p>
<p>K. Frank</p>

ConvolutionalAtom: <p>Frank already said a rule of thumb. But, for some common cases (e.g. Cifar 10-100 dataset), validation error (accuracy error, not loss) can keep decreasing even though validation loss start decreasing after some point. So, if your measure of performance is based on accuracy and validation error keeps decreasing (which it probably is), you might keep training until 30 epoch in your case.</p>
<p>There was even a theoretical framework to understand this behaviour called IB (Information Bottleneck) Principle, which was later disputed to be invalid e.g. in presence of ReLU activation</p>
<p>One thing to certainly emphasize: choose your validation metric carefully and based on your goal.</p>

Maryam_Khaliji: <p>​​Thanks for your detailed responses.</p>
<p>Sorry, but I need two more tips from you:</p>
<p>I should report the average results under multiple runs and their standard deviation,so  I need a metric to get the best model in per execution.</p>
<ol>
<li>The best model (iteration) in several papers I read about this dataset is the one with the highest AUC, without considering train/validation loss. Could it be true in machine learning concepts?</li>
<li>The second challenge is that I believe AUC is not enough to find the best model, and we also need <strong>macro F1</strong> and <strong>G-mean</strong>. Is it possible to combine them to get the best model. As an example, set a threshold for the F1-score and the G-mean, then the best model is the one with the highest AUC and a higher value than the thresholds for the F1-score and the G-mean? Can I apply it?</li>
</ol>
<p>Best Regards,</p>
<p>Maryam</p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="Maryam_Khaliji" data-post="7" data-topic="169317" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryam_khaliji/48/45054_2.png" class="avatar"> Maryam_Khaliji:</div>
<blockquote>
<ol>
<li>The best model (iteration) in several papers I read about this dataset is the one with the highest AUC, without considering train/validation loss. Could it be true in machine learning concepts?</li>
</ol>
</blockquote>
</aside>
<p>It would be reasonable to use AUC to select your “best” model.  The idea<br>
would be that AUC is “the metric you care about” and that binary cross<br>
entropy is the proxy for that metric that you use for training.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>The second challenge is that I believe AUC is not enough to find the best model, and we also need <strong>macro F1</strong> and <strong>G-mean</strong>. Is it possible to combine them to get the best model.</li>
</ol>
</blockquote>
</aside>
<p>Again, this would also be reasonable.  Now you would be using some<br>
combination of AUC, F1, and G-mean as “the metric you care about”<br>
and using it to select your “best” model.</p>
<p>In either case, I would probably still use a divergence between binary<br>
cross entropy as computed for your training and validation sets as an<br>
indication that overfitting has set in (implying that further training will<br>
likely not be helpful).  But you can use the divergence between other<br>
metrics as an indication of overfitting.</p>
<p>Note, if you use some metric computed for your validation set to stop<br>
training (because of overfittiing) and especially if you use some metric<br>
computed for your validation set to select your “best” model, you should<br>
<em>not</em> use your validation set to report final performance numbers.  Instead<br>
you should use a separate “test set.”  In a common approach, your would<br>
take your entire dataset and randomly split it into three sets – a training<br>
set for training, a validation set to monitor overfitting and possibly select<br>
your “best” model, and an entirely separate test set that you don’t look at<br>
during the training and model selection process, but use solely to compute<br>
your final performance metrics.</p>
<p>Best.</p>
<p>K. Frank</p>

Maryam_Khaliji: <p>Thank you very much. It was very helpful.</p>

Joao_Mesquita: <p>How did you end up mixing all those metrics (AUC, F1, and G-mean)</p>