Segmentation loss and metrics help ignore background

vg224: <p>Hello, I am training a model for multiclass semantic segmentation. I have 4 classes, background and 3 relevant classes. I only care about the 3 relevant classes. My data is imbalanced with a lot of background pixels. Calculating class weights it’s like (1.5, 40, 50, 30). I am sort of confused with the loss and metrics.</p>
<p>For loss, I am choosing between nn.CrossEntropyLoss with using class weights and Dice Loss from <a href="https://github.com/qubvel/segmentation_models.pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - qubvel/segmentation_models.pytorch: Segmentation models with pretrained backbones. PyTorch.</a>. I’ve read that Dice Loss should automatically consider class imbalance, so there is no need to use class weights. There is a option to select classes to consider for DiceLoss. However, if I select classes (1,2,3) and ignore the background, I assume it will never label anything as the background. Are these the best losses to consider and am I using them correctly? Are there any other loss functions I should consider?</p>
<p>For metrics, I use Torchmetrics Jaccard/iou and Dice score <a href="https://torchmetrics.readthedocs.io/en/stable/classification/jaccard_index.html" class="inline-onebox" rel="noopener nofollow ugc">Jaccard Index — PyTorch-Metrics 0.11.4 documentation</a>. For this I am not sure whether I should include background in the iou calculation. I am wondering what is the correct/official way to calculate these metrics? If I set ignore_index to 0 (my background class), it affects the other iou scores.</p>

KFrank: <p>Hi vg!</p>
<aside class="quote no-group" data-username="vg224" data-post="1" data-topic="177519" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vg224/48/59270_2.png" class="avatar"> vg224:</div>
<blockquote>
<p>For loss, I am choosing between nn.CrossEntropyLoss with using class weights and Dice Loss</p>
</blockquote>
</aside>
<p>The lore I subscribe to concerning cross entropy vs. the Dice score<br>
runs as follows:</p>
<p>Cross entropy with class weights works well with unbalanced data<br>
unless you have quite a large number of classes.  From recollection,<br>
the papers advocating using the Dice score instead of cross entropy<br>
predated the use of class weights (or at least made their comparisons<br>
with cross entropy without class weights).</p>
<p>My intuition is that cross entropy’s logarithmic divergence for wrong<br>
predictions is very helpful for training, and this is part of why it works<br>
so well (and works better than the Dice score).</p>
<p>There are some claims that the Dice score is the better choice when<br>
you have a large number of classes, and I have no reason to think<br>
that this isn’t true.</p>
<p>Because you are working with only four classes, I would suggest that<br>
you use cross entropy (with class weights), and only move beyond<br>
that if you have evidence that that isn’t working well.  If you do decide<br>
to use the Dice score, I would suggest that you use it <em>in addition</em> to<br>
cross entropy (to preserve the benefits of that logarithmic divergence),<br>
rather than using it by itself.  If you have enough computational power<br>
to experiment, I would only use the Dice score by itself if you can show<br>
that it works better than cross entropy for your use case.</p>
<p>Best.</p>
<p>K. Frank</p>

vg224: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="177519">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you do decide<br>
to use the Dice score, I would suggest that you use it <em>in addition</em> to<br>
cross entropy</p>
</blockquote>
</aside>
<p>Thanks for the response!</p>
<p>When you say in addition, do you mean my loss should be Loss = CrossEntropyLoss + DiceLoss?</p>
<p>Also, for my second question about the metrics in my initial post, how would you recommend I set up my metrics? Should I ignore_index = 0 to ignore the background (like iou_per_class = multiclass_jaccard_index(output, target, num_classes=n_classes, average=None, ignore_index=0)? I assume I should not do that as it messes up the other iou calculations but I want to check.</p>

KFrank: <p>Hi vg!</p>
<aside class="quote no-group" data-username="vg224" data-post="3" data-topic="177519" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vg224/48/59270_2.png" class="avatar"> vg224:</div>
<blockquote>
<p>When you say in addition, do you mean my loss should be Loss = CrossEntropyLoss + DiceLoss?</p>
</blockquote>
</aside>
<p>Yes, although I would put in a relative weighting factor for the two losses:</p>
<pre data-code-wrap="python"><code class="lang-python">Loss = CrossEntropyLoss + alpha * DiceLoss
</code></pre>
<p>Small values of <code>alpha</code> mean that you would mostly be using cross entropy<br>
with a little bit of Dice loss mixed in, while large values mean that you would<br>
mostly be using Dice loss.  If you do this, I would start with <code>alpha = 0.0</code>,<br>
and then slowly increase it to see whether an admixture of Dice loss actually<br>
helps your training and, especially, your validation / test results.</p>
<aside class="quote no-group">
<blockquote>
<p>Also, for my second question about the metrics in my initial post</p>
</blockquote>
</aside>
<p>I don’t really have much to say about this.</p>
<p>In general, it will depend on your use case and on how you intend to use<br>
your inference results.  Is it more important that you distinguish foreground<br>
from background, while correctly labelling the foreground classes is only<br>
a secondary concern?  Or vice versa?  Would mislabelling a background<br>
pixel be more or less bad if it is near the border of a foreground region than<br>
if it were out in sea of pure background?</p>
<p>It’s certainly okay to compute some “standard” metrics, but you should make<br>
a point of including some metrics that directly probe the intended use of<br>
your model.</p>
<p>Best.</p>
<p>K. Frank</p>