Multi-Label, Multi-Class class imbalance

MultiSeedLoaf: <p>Hi, I have implemented a network for multi-label, multi-class classification, this has been done using BCEWithLogits outputting to 6 sigmoid units. However, I have a class imbalance and was wondering if there were a way to weight such classes in the multi-label sense.</p>
<p>I have labels in the following one-hot encoded format: [0,1,0,1,0,0], refers to class 1 and class 3 are present.</p>
<p>The classes have the weighting {0: 0.16 , 1: 0.16 , 2: 0.25 , 3: 0.25 , 4: 0.083, 5: 0.083}.</p>
<p>Any ideas would be great, thank you :).</p>

ptrblck: <p>I think the easiest approach would be to specify <code>reduction='none'</code> in your criterion and then multiply each output with your weights:</p>
<pre><code class="lang-python">target = torch.tensor([[0,1,0,1,0,0]], dtype=torch.float32)
output = torch.randn(1, 6, requires_grad=True)
weights = torch.tensor([0.16, 0.16, 0.25, 0.25, 0.083, 0.083])

criterion = nn.BCEWithLogitsLoss(reduction='none')
loss = criterion(output, target)
loss = (loss * weights).mean()
loss.backward()
</code></pre>
<p>Would that work for you?</p>

MultiSeedLoaf: <p>That is brilliant thank you!</p>

raouf_ks: <p>arn’t we accentuating the loss function for the class that has a lot of examples and over-penalizing   the class which has few examples ?<br>
Over-penalizing : it s penalized by default because it has few examples and we can t make our model smooth for it</p>

ptrblck: <p>Generally yes, but as I understood from the original post, the target would be just one sample for a multi-label classification task (using 6 classes) with the given weights.</p>

Saewon_Yang: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I don’t get the idea of this post. why do we have to make reduction=‘none’ with multi-label problem? to my knowledge we have to multiply by the weight according to the distribution of the dataset. but how do we get the weights? whats the difference between multiplying weights to ‘none’ and multiplying weight to ‘mean’. thanks!</p>

ptrblck: <aside class="quote no-group" data-username="Saewon_Yang" data-post="6" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saewon_yang/48/10332_2.png" class="avatar"> Saewon_Yang:</div>
<blockquote>
<p>why do we have to make reduction=‘none’ with multi-label problem?</p>
</blockquote>
</aside>
<p>You don’t have to use <code>reduction=none</code>, but question in this topic was how to multiply each class prediction with a certain weight, so this is a valid approach.</p>
<aside class="quote no-group" data-username="Saewon_Yang" data-post="6" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saewon_yang/48/10332_2.png" class="avatar"> Saewon_Yang:</div>
<blockquote>
<p>but how do we get the weights?</p>
</blockquote>
</aside>
<p>I’m not sure how <a class="mention" href="/u/multiseedloaf">@MultiSeedLoaf</a> got the weights, so we might wait for his answer.<br>
A simple approach would be to use the inverse of the class count, but I’m not sure if that’s the best approach for a multi-label classification.</p>
<aside class="quote no-group" data-username="Saewon_Yang" data-post="6" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/saewon_yang/48/10332_2.png" class="avatar"> Saewon_Yang:</div>
<blockquote>
<p>whats the difference between multiplying weights to ‘none’ and multiplying weight to ‘mean’.</p>
</blockquote>
</aside>
<p>If you use <code>reduction='mean'</code>, you’ll get a single scalar, so that you cannot multiply the loss with the class weights anymore.</p>

Rowen_Zhao: <p>Hello, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I’m just wondering if I could use “pos_weight” in BCEWithLogitsLoss for imbalanced dataset. Do they (pos_weight and the approach you provided) have the same effect?</p>

JavisPeng: <p>I have tested the three methods, It’s strange that they got the same result</p>
<pre><code class="lang-python">from torch import nn  
import torch  
  
class WeightedMultilabel(nn.Module):  
    def __init__(self, weights: torch.Tensor):  
        super(WeightedMultilabel, self).__init__()  
        self.cerition = nn.BCEWithLogitsLoss(reduction='none')  
        self.weights = weights  
  
    def forward(self, outputs, targets):  
        loss = self.cerition(outputs, targets)  
        return (loss * self.weights).mean()  
  
x=torch.randn(3,4)  
y=torch.randn(3,4)  
  
w=torch.tensor([10,2,15,20],dtype=torch.float)  
  
criterion_BCE=nn.BCEWithLogitsLoss(w)  
criterion_mult=WeightedMultilabel(w)  
criterion_mult2=nn.MultiLabelSoftMarginLoss(w)  
  
loss1=criterion_BCE(x,y)  
loss2=criterion_mult(x,y)  
loss3=criterion_mult2(x,y)  
  
print(loss1)  
print(loss2)  
print(loss3)  
  
# tensor(7.8804)  
# tensor(7.8804)  
# tensor(7.8804)
</code></pre>

sougata_saha: <p>If you consider each label to be a node in the final layer of the network,  “pos_weight” will help in tacking the class imbalance in each node individually (check the formula in <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="nofollow noopener">documentation</a>). For example in a multi-label binary classification with 4 labels, it will help in assigning weights to the positive class for each label individually. The weighting system mentioned here will help in assigning importance to each of the individual losses calculated for each of the 4 labels. This will not necessarily tackle the class imbalance problem, but I think should scale the gradients from a node.</p>

Aymen_Sekhri: <p>What does reduction actually mean?</p>

ptrblck: <p>Reduction in this context means how to reduce a tensor with multiple elements to a scalar or generally how to reduce a dimension with a <code>size &gt; 1</code> to a size of 1.<br>
Loss functions accept the <code>reduction</code> argument, which specifies, if the mean, sum etc. should be used to yield the scalar loss value given a batch of model outputs and targets.</p>

moreshud: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">target = torch.tensor([[0,1,0,1,0,0]], dtype=torch.float32)
output = torch.randn(1, 6, requires_grad=True)
weights = torch.tensor([0.16, 0.16, 0.25, 0.25, 0.083, 0.083])

criterion = nn.BCEWithLogitsLoss(reduction='none')
loss = criterion(output, target)
loss = (loss * weights).mean()
loss.backward()
</code></pre>
</blockquote>
</aside>
<p>Can this approach be helpful in a multi-label, multiclass class imbalance image segmentation also?</p>
<p>Am I to use the <code>weight</code> or <code>pos weight</code> parameter in the <code>nn.BCEWithLogitsLoss</code> because it will be combined with a weighted dice loss function?</p>
<p>My approach of getting the class weights is by summing the 1’s in each class label of the tragets of the training dataset and taking the inverse as suggested.</p>

ptrblck: <aside class="quote no-group" data-username="moreshud" data-post="14" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> moreshud:</div>
<blockquote>
<p>Can this approach be helpful in a multi-label, multiclass class imbalance image segmentation also?</p>
</blockquote>
</aside>
<p>Yes, I would think so.</p>
<aside class="quote no-group" data-username="moreshud" data-post="14" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> moreshud:</div>
<blockquote>
<p>Am I to use the <code>weight</code> or <code>pos weight</code> parameter in the <code>nn.BCEWithLogitsLoss</code> because it will be combined with a weighted dice loss function?</p>
</blockquote>
</aside>
<p>Calculating the weights in my code snippet would add the corresponding weights to each sample of the loss before the reduction is done.<br>
The <code>pos_weight</code> acts directly on the positive term of the loss function as shown in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html?highlight=bcewithlogitsloss#torch.nn.BCEWithLogitsLoss">docs</a>, so they wouldn’t be equivalent.</p>

moreshud: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<aside class="quote no-group" data-username="moreshud" data-post="14" data-topic="37573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/f17d59/48.png" class="avatar"> moreshud:</div>
<blockquote>
<p>My approach of getting the class weights is by summing the 1’s in each class label of the tragets of the training dataset and taking the inverse as suggested.</p>
</blockquote>
</aside>
<p>Is this approach in the case of multiclass multilabel?</p>

ptrblck: <p>If you are using these calculated weights, I would start with the proposed calculation from the docs:</p>
<blockquote>
<p>For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100=3 . The loss would act as if the dataset contains 3×100=300 positive examples.</p>
</blockquote>
<p>On the other hand, if you want to weight each sample in the batch, I would normalize the weights to a sum of 1 so that each batch loss has the same mean.</p>

moreshud: <p>The proposed weight calculation is used for the <code>pos_weights</code> but incase of other weight calculations as suggested in the forum, do I take such for the <code>pos_weight</code> or the <code>wights</code> in the <code>BCEWithLogistLoss</code>?</p>

Paran0: <p>Just wanted to say that overall your comments are simple and right to the point … Tnx!!</p>

Ali_Aghababaei: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Actually, I have not completely figured out the answers to moreshud ’ s question.</p>
<p>Imagine that I have a multi-class, multi-label classification problem; my imbalanced one-hot coded dataset includes 1000 images with 4 labels with the following frequencies: class 0: 600, class 1: 550, class 2: 200, class 3: 100.<br>
As I said, the targets are in a one-hot coded structure.<br>
For instance, the target [0, 1, 1, 0] means that classes 1 and 2 are present in the corresponding image.</p>
<p>Well, in order to calculate the BCEWithLogitsLoss concerning the data imbalance, one way is that you suggested in this post: <a href="https://discuss.pytorch.org/t/multi-label-multi-class-class-imbalance/37573/2" class="inline-onebox">Multi-Label, Multi-Class class imbalance - #2 by ptrblck</a>. Here, we calculate the class weights by inverting the frequencies of each class, i.e., the class weight tensor in my example would be: torch.tensor ([1/600, 1/550, 1/200, 1/100]). After that, the class weight tensor will be multiplied by the unreduced loss and the final loss would be the mean of this tensor.</p>
<p>However, as far as I know, the pos_weight parameter of the BCEWithLogitsLoSS could also be used in this case. Here is my question:<br>
to my knowledge, the two tensors (class weight tensor in the previous paragraph and the pos_weight tensor) are totally different. For each class, the number of positive and negative samples should be calculated and the num_negative/ num_positive would be the pos_weight for that particular class. In my example, the pos_weight of class 0 would be (1000-600)/600  = 0.67, the pos_weight of class 1 would be (1000-550)/550 = .82, the pos_weight of class 2 would be (1000-200)/200 = 4, and the pos_weight of class 3 would be (1000-100)/100 = 9. Thus, the pos_weight tensor would be torch.tensor ([.67, .82, 4, 9]). Is this way of calculating pos_weight tensor the right one? If the answer is yes, I think that the previous method (I mean calculating the class weights and multiplying it with the unreduced loss) would be more convenient in the case of a dataset with a large number of labels since we should only invert the frequencies, am I right?</p>
<p>Also, another question is about the weight parameter of the BCEWithLogitsLoss. As represented in the formula, the weight parameter is a tensor that is multiplied by the whole loss, not merely the positive targets (as opposed to pos_weight). My question is that how is the weight parameter tensor different from the class weights tensor? since the class weights tensor is similarly multiplied by the whole loss. However, it is said that the weight parameter tensor is of size nbatch, and I do not understand what its function is.</p>
<p>I deeply appreciate your consideration.</p>

Ali_Aghababaei: <p>I have the similar question. Have you found the answer?</p>