Class imbalance in one-hot encoding

Ali_Aghababaei: <p>Hi everyone,</p>
<p>I am working on a one-hot encoded dataset (including about 2000 images of the retina with each of which having one or more labels (multi-label classification). There are 28 labels in total, and as I said are represented in a one-hot coding structure.</p>
<p>For instance : the target [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] means that the retinal image has class 0 (diabetic retinopathy) and class 16 (macular degeneration) labels.</p>
<p>Importantly, the classes are imbalanced. To my knowledge, there are two ways to deal with this problem: 1) Calculating the loss function with respect to the class weights (as discussed in <a href="https://discuss.pytorch.org/t/multi-label-multi-class-class-imbalance/37573/2" class="inline-onebox">Multi-Label, Multi-Class class imbalance - #2 by ptrblck</a>, I used BCELOSSWithlogits, setting the “reduction” parameter as “none”, and multiplying the loss with the class weights.<br>
2) Oversampling using WeightedRandomSampler<br>
Actually here is my problem. As the targets are in a one-hot coded format, how can we calculate the sample_weights tensor? I have read the post <a href="https://discuss.pytorch.org/t/class-imbalance-with-image-segmentation/38484/3" class="inline-onebox">Class imbalance with image segmentation - #3 by An18</a> completely but still am not sure whether the right solution was provided there or not (in reply to An18, it was said that sample weight could be calculated as “samples_weights = y*weights.T”. Also, if I calculate the sample weights in this way and create my_sampler as:<br>
my_sampler = torch.utils.data.WeightedRandomSampler (sample_weights, num_samples= len (sample_weights), replacement= True)</p>
<p>and then passing it to the sampler parameter of the DataLoader, an error raises:</p>
<p>“invalid multinomial distribution (sum of probabilities &lt;= 0)”</p>
<p>Would calculating the sample_weights in the following way be the right solution?</p>
<p>sample_weights = torch.mean (torch.mul (targets, class_weights.T), 1)</p>
<p>I am new to PyTorch and deeply appreciate your solutions.</p>
<p>Here is the relevant part of my code:</p>
<pre><code class="lang-auto">class_counts = torch.tensor ([376, 100, 317, 138, 101, 73, 186, 14, 47, 15, 37, 282, 28, 6, 16, 65, 58, 5, 17, 11, 14, 43, 32, 15, 22, 11, 6, 34])

class_weights = torch.tensor (1./ class_counts, dtype = torch.float)

sample_weights = torch.mean (torch.mul (targets, class_weights.T), 1)

my_sampler = torch.utils.data.WeightedRandomSampler (sample_weights, num_samples= len (sample_weights), replacement= True)

train_loader = DataLoader (train_dataset, batch_size = 128, sampler= my_sampler)

next (iter (train_loader2))

</code></pre>
<p>The last line is where the error raises if I do not calculate the mean of “torch.mul (targets, class_weights.T), 1”.</p>

ptrblck: <p>I don’t think using the <code>WeightedRandomSampler</code> would work out of the box for a multi-label classification and you might want to take a look at <a href="https://discuss.pytorch.org/t/unbalanced-data-in-multiple-object-detection/88104/2">this post</a> for some suggestions.</p>

Ali_Aghababaei: <p>Thanks a lot for your response. I will read the suggested articles in the post you’ve mentioned. So in your opinion, do you think that a more easy way to deal with data imbalance in multi-label classification is to calculate the loss with respect to the class weights (in the way you have previously mentioned here: <a href="https://discuss.pytorch.org/t/multi-label-multi-class-class-imbalance/37573/2" class="inline-onebox">Multi-Label, Multi-Class class imbalance - #2 by ptrblck</a>?</p>
<p>Also, is there any point in the post <a href="https://discuss.pytorch.org/t/class-imbalance-with-image-segmentation/38484/3" class="inline-onebox">Class imbalance with image segmentation - #3 by An18</a> and the reply to it that could be useful in my case?</p>
<p>One more question is that I cannot use sklearn.utils.class_weight.compute_class_weight to calculate the class weights since the targets are multi-label and in a one-hot coded format, can I?</p>
<p>I really appreciate your consideration.</p>

ptrblck: <aside class="quote no-group" data-username="Ali_Aghababaei" data-post="3" data-topic="130215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>So in your opinion, do you think that a more easy way to deal with data imbalance in multi-label classification is to calculate the loss with respect to the class weights</p>
</blockquote>
</aside>
<p>Yes, I believe it could be easier to experiment with a weighted loss.</p>
<aside class="quote no-group quote-modified" data-username="Ali_Aghababaei" data-post="3" data-topic="130215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>Also, is there any point in the post…</p>
</blockquote>
</aside>
<p>I don’t think that multiplying the weights directly will yield balanced samples (this should also be explained in the linked papers).</p>
<aside class="quote no-group" data-username="Ali_Aghababaei" data-post="3" data-topic="130215">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ali_aghababaei/48/61342_2.png" class="avatar"> Ali_Aghababaei:</div>
<blockquote>
<p>One more question is that I cannot use sklearn.utils.class_weight.compute_class_weight to calculate the class weights since the targets are multi-label and in a one-hot coded format, can I?</p>
</blockquote>
</aside>
<p>Yes, I don’t think it’s possible using the standard “counting” method.</p>

Ali_Aghababaei: <p>I am deeply grateful for your clear responses.</p>
<p>Wish you the best</p>