Class imbalance with image segmentation

username: Xiaoyu_Song
cooked: <p>Hi, I am trying to deal with the class imbalance problem in image segmentation.<br>
And as my target mask size is 128x128. If I have 3300 training image, my total number of target data (if concatenated ) will be 422400 x 128.<br>
I tried to get the total_mask which has the size <code>torch.Size([422400, 128])</code><br>
And I try to build the sampler with this code below.</p>
<pre><code class="lang-auto">unique_color, count = np.unique(mask_total.cpu(), return_counts = True)
weight = 1. / count
samples_weight = weight[mask_total]
</code></pre>
<p>It give me this error</p>
<pre><code class="lang-auto">IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
</code></pre>
<p>since my mask_total is 422400 x 128.<br>
So if my images are 128 x128, to get the <code>samples_weight</code>,  what should I implement?<br>
Thank you a lot.</p>

username: ptrblck
cooked: <p>Could you explain your use case a bit more and how you would like to sample the masks?<br>
Usually you would create sample weights, which would assign a certain weight to each sample in the <code>Dataset</code> (image and mask pair). In your description it looks like you would like to sample pixels from your mask? Is that correct or do I misunderstand your use case?</p>
<p>Also, for reference: <a href="https://discuss.pytorch.org/t/some-problems-with-weightedrandomsampler/23242/23?u=ptrblck">this answer</a> might belong into this thread.</p>

username: An18
cooked: <p>I’m writing a multilabel classifier where the targets are a multi-hot encoded vector for 13 classes like below</p>
<pre><code class="lang-auto">0              "I'm a category 7 sample"  ...  [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]
1              "I'm a category 8 sample"  ...  [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]
2        "I'm a category 1 AND 6 sample"  ...  [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]
</code></pre>
<p>The Dataset is heavily imbalanced, with the following distribution for 20,000 samples:</p>
<pre><code class="lang-auto">{'Cat 1': 450,
 'Cat 2': 364,
 'Cat 3': 37,
 'Cat 4': 334,
 'Cat 5': 630,
 'Cat 6': 1096,
 'Cat 7': 918,
 'Cat 8': 3324,
 'Cat 9': 2053,
 'Cat 10': 532,
 'Cat 11': 1110,
 'Cat 12': 101,
 'Cat 13': 776}
</code></pre>
<p>There’s roughly 10,000 unlabelled samples in this dataset.</p>
<p>To adress the imbalance in training, I am trying to use WeightedRandomSampler, based on this answer (<a href="https://discuss.pytorch.org/t/some-problems-with-weightedrandomsampler/23242/2" class="inline-onebox">Some problems with WeightedRandomSampler</a>) by “ptrblck”</p>
<p>Here’s the relevant code:</p>
<pre><code class="lang-auto"># dist.values = [450, 364, 37, 334, 630, 1096, 918, 3324, 2053, 532, 1110, 101, 776, 9967]

weights = [1./v for v in dist.values()]
weights = torch.tensor(weights, dtype=torch.float)

# weights: tensor([0.0022, 0.0027, 0.0270, 0.0030, 0.0016, 0.0009, 0.0011, 0.0003, 0.0005,
#        0.0019, 0.0009, 0.0099, 0.0013, 0.0001])

y = torch.tensor(train_dataset['list'], dtype=torch.long)
# y:
# [   [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0],
#     [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
#     ...
#     [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],
#     [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
#     [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0],]

samples_weights = weights[y]

# for 17,000 training samples
# samples_weights torch.Size([17000, 13]):
[
[0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0027, 0.0022, 0.0022, 0.0022,
        0.0022, 0.0022, 0.0022, 0.0022],
[0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022,
        0.0022, 0.0022, 0.0022, 0.0022],
...
[0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022,
        0.0022, 0.0027, 0.0022, 0.0022],
[0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0027, 0.0022, 0.0022,
        0.0022, 0.0022, 0.0022, 0.0022],
[0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022, 0.0022,
        0.0022, 0.0022, 0.0022, 0.0022],]

</code></pre>
<p>Since my targets are multi-hot encoded, samples_weights will either be valued as weights[0] or weights[1], which is obviously wrong.</p>
<p>Is there any way I can make this work with the multi-hot targets?</p>
<p>Shouldn’t samples_weight be the inplace multiplication between the weights vector and the targets for that sample?<br>
For example, the samples_weight for sample [0] above would be:</p>
<pre><code class="lang-auto">[0, 0, 0, 0, 0, 0.0009, 0, 0, 0, 0, 0, 0, 0]

</code></pre>
<p>This is all new to me so I appreciate any insight.</p>

username: shartzog
cooked: <p>If I’m understanding what you’re after, you really need to do</p>
<pre><code class="lang-auto">samples_weights = y*weights.T
</code></pre>
<p>but to do that, you’re going to have to either add a 14th element representing “unlabeled” for every sample in y, OR cut out the 14th element of weights that represents the unlabeled category.</p>
<p>Also, welcome to the forum.  Hope this helps and good luck with your project!  <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: An18
cooked: <p>Thank you!</p>
<p>Good eye on the 14th “class”! (I left it by mistake because I wanted to count the unlabelled for this post.)</p>
<p>I’ll re-run the training and see if this works.</p>