Computing sample weights for multi class image segmentation

simines: <p>To use weighted random sampler in PyTorch we need to compute samples weight. In case of image classification where we have one label for each class, we can compute samples weight using sklearn library. However, for segmentation, for each image we have multiple classes. I was wondering how we can compute the samples weight in this case?</p>

KFrank: <p>Hi Carol!</p>
<aside class="quote no-group" data-username="simines" data-post="1" data-topic="159893" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simines/48/58738_2.png" class="avatar"> simines:</div>
<blockquote>
<p>However, for segmentation, for each image we have multiple classes. I was wondering how we can compute the samples weight in this case?</p>
</blockquote>
</aside>
<p>I will answer in the context of <em>semantic segmentation</em> where you classify<br>
each pixel in an image (rather than assigning the entire image to a single<br>
class).</p>
<p>To first approximation, you won’t want (or be able) to use sample weights<br>
for semantic segmentation.  Instead you will want to use class weights<br>
(e.g., as supported by the <code>CrossEntropyLoss</code>’s <code>weight</code> constructor<br>
argument).</p>
<p>Consider an example such as this:</p>
<p>You have images consisting of mostly grey  background pixels (class-0).<br>
They contain a smattering of red (class-1), green (class-2), and blue<br>
(class-3) foreground pixels and you want to perform four-class semantic<br>
segmentation.  Let’s say that the (smallish number of) foreground pixels<br>
in some images are mostly red, in others, green or blue, and in others<br>
mixed.</p>
<p>Let’s assume that no typical image type is uncommon in your data set,<br>
so that in this sense your data set is balanced.  But from a classification<br>
perspective, your data set is highly unbalanced because the large majority<br>
of your pixels are background pixels, so class-0 is highly over-represented.</p>
<p>This is a problem because your network could simply learn to always<br>
predict class-0 for a pixel and it would almost always be right.</p>
<p>But you can’t fix this problem with sample weights (and a weighted<br>
random sampler) because <em>all</em> of your samples (images) have class-0<br>
pixels highly over-represented.</p>
<p>Instead, use class weights in your loss function to underweight class-0<br>
pixels.  Now if your network tries to predict class-0 for all pixels, the<br>
large number of correct class-0 predictions will be underweighted, so<br>
the network will have to learn to predict class-1, class-2, and class-3<br>
pixels correctly to do well with the loss function.</p>
<p>(As a counter-example, let’s say most of your images are about half<br>
background and half foreground.   Let’s further say that most of your<br>
images have red foreground pixels and only a few have green or blue<br>
foreground pixels.  Now it would make sense to use sample weights<br>
where you increase the probability of sampling green and blue images<br>
relative to red so that a batch, on average, consists of a third each of<br>
red, green, and blue images.)</p>
<p>Best.</p>
<p>K. Frank</p>

simines: <p>Thank you very much for the explanation.</p>