What is the best loss function for a Binary Segmentation problem with a class imbalance

Mahammad_Nabizade: <p>I tried FocalLoss, TverskyLoss, Focal_Tversky_Loss, DiceLoss, BCEWithLogitsLoss , none of them work well for now. P.S I also have data problem, the number of images are few (around 4k for training, 1k for validation) and maybe that is the source of issue.</p>
<p>What are some best case practices for binary segmentation with class imbalance (foreground pixels are few related to background pixels)</p>

KFrank: <p>Hi Mahammad!</p>
<aside class="quote no-group" data-username="Mahammad_Nabizade" data-post="1" data-topic="204661" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahammad_nabizade/48/69699_2.png" class="avatar"> Mahammad_Nabizade:</div>
<blockquote>
<p>I tried FocalLoss, TverskyLoss, Focal_Tversky_Loss, DiceLoss, BCEWithLogitsLoss , none of them work well for now.</p>
</blockquote>
</aside>
<p>In what sense did they not “work well?”</p>
<aside class="quote no-group">
<blockquote>
<p>What are some best case practices for binary segmentation with class imbalance</p>
</blockquote>
</aside>
<p><a href="https://discuss.pytorch.org/t/bcewithlogitsloss-validation-loss-did-not-crease-after-applying-pos-weight/204088/4">This post</a> suggests using <code>BCEWithLogitsLoss</code> together with its<br>
<code>pos_weight</code> constructor argument, possibly augmented with something<br>
like (a differentiable version of) the Dice coefficient (if you can show that<br>
it actually improves training and performance).</p>
<p>Have you tried that yet?</p>
<p>Best.</p>
<p>K. Frank</p>

Mahammad_Nabizade: <p>Hello, thanks a lot for the answer.</p>
<p>What I meant was, they all overfit, after a certain epoch (usually 6-7th) the validation loss increase while the training loss decreases. I removed <strong>early stopping</strong> to see if there will be any progress but unfortunately the scenario lasted forever. I think my data is problematic (5k training images, 1k validation and 1k for test), and I don’t have enough data, because the images I use for training are from the same scene, same place. Although objects are moving and changing the location I think it still affects the training. I tried <code>BCEWithLogitsLoss</code> with different <code>pos_weight</code>, values (starting from 100 to 600) but they all overfit so early (after 20th epoch max).</p>
<p>I also tried different drop out rates to combat overfitting unfortunately that did not help too</p>
<p>I will try to combine <code>Dice coefficient</code> with <code>pos_weight</code> as you advised and see what happens.</p>
<p>Thanks again for your time.</p>

KFrank: <p>Hi Mahammad!</p>
<aside class="quote no-group" data-username="Mahammad_Nabizade" data-post="3" data-topic="204661" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahammad_nabizade/48/69699_2.png" class="avatar"> Mahammad_Nabizade:</div>
<blockquote>
<p>What I meant was, they all overfit, after a certain epoch (usually 6-7th) the validation loss increase while the training loss decreases.</p>
</blockquote>
</aside>
<p>Yes, that does sound like overfitting (although earlier in your training than<br>
I would have expected).</p>
<aside class="quote no-group">
<blockquote>
<p>(5k training images, 1k validation and 1k for test)</p>
</blockquote>
</aside>
<p>In general, that would sound like a reasonable amount of data (but not<br>
a whole lot).</p>
<aside class="quote no-group">
<blockquote>
<p>and I don’t have enough data, because the images I use for training are from the same scene, same place.</p>
</blockquote>
</aside>
<p>Yes, if your data samples are not independent of one another (another<br>
a way of saying it, if they are highly correlated), it counts as having less<br>
data.  So you may have, effectively, many fewer than 5k training images.<br>
Depending on how “correlated” your training images are, this could be a<br>
significant problem.</p>
<p>You don’t say so explicitly, but you make it sound like your validation and<br>
test images might not be from the same scene as your training images.<br>
Even if they’re from the same scene, they might be from different time<br>
periods than the training images.</p>
<p>Either way, I would suggest (if your not doing this already) that you combine<br>
your current training, validation, and test datasets into a single 7k dataset,<br>
shuffle it, and then randomly divide it into new training, validation, and test<br>
datasets.  This will ensure that these datasets all have the same character<br>
statistically.</p>
<p>This may or may not be appropriate for your use case, but I would now<br>
expect you to be able to train using your new 5k training dataset without<br>
overfitting setting in before you can perform reasonably well on your<br>
validation dataset.  Even if your training dataset is effectively “smaller”<br>
than 5k because the images are “correlated,” your validation images will<br>
be equally well “correlated” with your training images, making it easier for<br>
your model to perform well on your validation dataset.</p>
<aside class="quote no-group">
<blockquote>
<p>I tried <code>BCEWithLogitsLoss</code> with different <code>pos_weight</code>, values (starting from 100 to 600) but they all overfit so early</p>
</blockquote>
</aside>
<p>The issue of data imbalance and overfitting are pretty much independent<br>
of one another.  Using <code>pos_weight</code> makes sense for the problem you’ve<br>
described (so keep using it), but it’s unlikely to help with overfitting.</p>
<aside class="quote no-group">
<blockquote>
<p>I also tried different drop out rates to combat overfitting unfortunately that did not help too</p>
</blockquote>
</aside>
<p>Dropout is a reasonable approach to address overfitting, so consider<br>
adding back to your model as you continue your experiments.  You might<br>
also try data augmentation (but if your images are already highly correlated,<br>
this might not help a lot) and weight decay.</p>
<p>If, after shuffling your whole combined 7k dataset and trying the above,<br>
you’re still having trouble with overfitting, you will probably have to reduce<br>
the “capacity” of your model by having fewer and / or “narrower” (that is,<br>
smaller values for <code>out_channels</code> and <code>out_features</code>) layers.</p>
<aside class="quote no-group">
<blockquote>
<p>I will try to combine <code>Dice coefficient</code></p>
</blockquote>
</aside>
<p>At this point, I wouldn’t worry about Dice coefficient.  Get your overfitting<br>
problem solved first, and only then – if you clearly are having problems<br>
with data imbalance – add the complication of incorporating something<br>
like Dice coefficient.</p>
<p>Best.</p>
<p>K. Frank</p>

Mahammad_Nabizade: <p>Thanks for the quick replies, let me give more info about the dataset.</p>
<p>The training data are always the same scene (same place) with different objects and lightning conditions sometimes. Anyway, I consider this as ‘highly correlated’ as you say.</p>
<p>The validation and test set have 3 different scenes each, but the number of images are less.</p>
<p>I’m afraid if I shuffle them I can have data leak. I manually combined 1 validation and 1 training scene with training data, it did not work out either.</p>
<p>Do you think <code>SMOTE</code> technique can help ? As I know it is used for tabular data but I saw an implementation for binary segmentation too.</p>
<p>In conclusion, I think the only solution would be try to find more image data which is not highly correlated.</p>
<p>About making the model narrower, does it mean to simplify the model ? I thought the opposite, and tried a deeper feature extractor <code>(ResNet101 instead of ResNet50)</code> but it overfitted even faster</p>

KFrank: <p>Hi Mahammad!</p>
<aside class="quote no-group" data-username="Mahammad_Nabizade" data-post="5" data-topic="204661" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahammad_nabizade/48/69699_2.png" class="avatar"> Mahammad_Nabizade:</div>
<blockquote>
<p>The training data are always the same scene (same place)</p>
</blockquote>
</aside>
<p>Okay, as I understand it, you want to train on data from scene A.</p>
<aside class="quote no-group">
<blockquote>
<p>The validation and test set have 3 different scenes each</p>
</blockquote>
</aside>
<p>But you want your model to work on data from scenes B, C, D and E, F, G.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m afraid if I shuffle them I can have data leak.</p>
</blockquote>
</aside>
<p>Yes, given your use case, this could be considered a kind of data leak.</p>
<p>This is an issue all the time in the real world.  Let’s say you train a<br>
self-driving-vehicle model on dirt road A, city street B, and highway C.<br>
You really can’t expect such a model to work on a bunch of other roads.</p>
<p>But if you train on <em>a lot</em> of different dirt roads and city streets and highways,<br>
your model might work on various roads it wasn’t trained on.</p>
<aside class="quote no-group">
<blockquote>
<p>Do you think <code>SMOTE</code> technique can help ?</p>
</blockquote>
</aside>
<p>Clear your mind of thoughts of imbalanced data.</p>
<p>Your problem is that your training data is not sufficiently representative<br>
of the data you want to apply your model to (hence leading to overfitting<br>
issues).  (Note, techniques that reduce overfitting can help your model<br>
“generalize” to data that differs somewhat in character from the data it<br>
was trained on.  But you can only push this so far – at some point you<br>
have to train on data that is reasonably representative of the data you<br>
actually want to apply it to.)</p>
<p>Issues of data imbalance are independent of your main issue, and my<br>
working assumption is that <code>pos_weight</code> will be enough to address the<br>
data imbalance you have.</p>
<aside class="quote no-group">
<blockquote>
<p>In conclusion, I think the only solution would be try to find more image data which is not highly correlated.</p>
</blockquote>
</aside>
<p>Yes.  Going back to your notion of “scenes,” let’s say that your goal is to<br>
have your model work on data from scenes B through G.  But when you<br>
train on data from scene A, your model turns out to learn about the <em>specific<br>
details</em> of scene A, rather than the general character of your actual use<br>
case.  (This effect is indeed what we call overfitting.)</p>
<p>It’s fair to consider training on data from scenes B through G a data leak.</p>
<p>So what you need to do is train on data from other scenes, say H, I, J, K, …,<br>
that are <em>representative of</em> (but not the same as) the scenes you want to<br>
apply your model to.</p>
<p>Your not “cheating” (data leak) because your model isn’t learning any details<br>
of scenes B through G.  Instead, you’re not learning the specific details of<br>
scene A (because your model is being forced to learn the shared general<br>
character of a bunch of different scenes).  As long as the “details” of your<br>
training scenes are <em>representative of</em> (but not the same as) the "details* of<br>
the scenes to which you will apply your model, you should be able to train<br>
your model to work for your use case.</p>
<aside class="quote no-group">
<blockquote>
<p>About making the model narrower, does it mean to simplify the model ?</p>
</blockquote>
</aside>
<p>Yes – basically fewer parameters.</p>
<aside class="quote no-group">
<blockquote>
<p>I thought the opposite, and tried a deeper feature extractor <code>(ResNet101 instead of ResNet50)</code> but it overfitted even faster</p>
</blockquote>
</aside>
<p>This was to be expected.  The greater the “capacity” of your model (more<br>
parameters, roughly speaking), the more capacity it will have to learn the<br>
irrelevant details of your training data and overfit.</p>
<p>As a rule of thumb, if you can’t fit (including overfit) your training data,<br>
you might try a model with greater capacity, but if you overfit too easily,<br>
you might try a model with lower capacity.</p>
<p>One last note:  You can sometimes address overfitting by fine-tuning a<br>
pre-trained version of your model, rather than training the model from<br>
scratch.  But this depends on having an appropriately pre-trained model.</p>
<p>Best.</p>
<p>K. Frank</p>

Mahammad_Nabizade: <p>Thank you sir, I learnt a lot from your answers.</p>