Weighted BCE loss with logits

aleemsidra: <p>I am dealing with imbalanced dataset. I want to use weighted BCE loss with logits. <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">nn.BCEWithLogitsLoss </a> takes <code>pos_weight</code> argument.<br>
From the docs:<br>
<strong>pos_weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.</p>
<p>In my case I have two classes. However, when I pass a vector with a length equal to 2, the following error comes:<br>
<code>RuntimeError: The size of tensor a (2) must match the size of tensor b (16) at non-singleton dimension 0</code></p>
<p>Where 16 is the batch size and <code>pos_weight vector = [1.1734028683181226, 6.7669172932330826]</code></p>

Andrei_Cristea: <p>Could you share a minimal reproducible snippet that raises this error?</p>
<p>In particular, make sure to include a print out of the shape of your model output. My best guess is that you’re accidentally transposing the axes such that length 16 is showing up in a non-batch position.</p>

aleemsidra: <p><code>make_weights_for_balanced_classes</code> is used to generate weights. <code>get_loss</code> is being used to calculate the loss.</p>
<pre><code class="lang-auto">def make_weights_for_balanced_classes(labels, nclasses=2):                        
    count = [0] * nclasses                                                      
    for item in labels:                                                         
        count[item] += 1 
                                                        
    weight_per_class = [0.] * nclasses  
    N = float(sum(count))                                                   
    for i in range(nclasses):                                                   
        weight_per_class[i] = N/float(count[i])
    return    weight_per_class 

def get_loss(self, pred, label):

        print("pred shape:", pred.shape)
        print("labels", label.shape)
        print("weights", self.weight)
        criterion = nn.BCEWithLogitsLoss(pos_weight= self.weight)
        if torch.cuda.is_available():
            criterion.cuda()
        return criterion(pred, label)
</code></pre>
<p>model output:  torch.Size([16, 2])<br>
weight vector: torch.Size([2])<br>
pred shape: torch.Size([16])<br>
labels torch.Size([16])<br>
weights tensor([1.1734, 6.7669], dtype=torch.float64)<br>
It results in error:<br>
<code>RuntimeError: The size of tensor a (2) must match the size of tensor b (16) at non-singleton dimension 0</code></p>

Andrei_Cristea: <p>The issue is that the way you’re passing the model output and labels is not how <code>BCEWithLogitsLoss</code> expects them. It expects the following:</p>
<ul>
<li>model output (before sigmoid), in your case this should have shape (16, 2)</li>
<li>one-hot encoded target, which should also have shape (16, 2)</li>
</ul>
<p>So in your case it thinks you’re running it on a single batch with 16 classes, rather than 16 batches with two classes, which is probably what you intended.</p>
<p>The below code runs fine:</p>
<pre><code class="lang-auto">pred = torch.randn(16, 2)
label = torch.empty(16, 2).random_(2)
weight = torch.Tensor([1.1734, 6.7669]).float()

criterion = nn.BCEWithLogitsLoss(pos_weight=weight)

print("pred shape:", pred.shape)
print("labels", label.shape)
print("weights", weight)

print(criterion(pred, label))

Output:
pred shape: torch.Size([16, 2])
labels torch.Size([16, 2])
weights tensor([1.1734, 6.7669])
tensor(2.4276)
</code></pre>
<p>So in your case you’ll need to one-hot encode your labels, and pass the model output directly, so something like this:</p>
<pre><code class="lang-auto">criterion(model_output, nn.functional.one_hot(label, num_classes=2))
</code></pre>
<p>Hope this helps!</p>

KFrank: <p>Hi Sidra!</p>
<aside class="quote no-group quote-modified" data-username="aleemsidra" data-post="1" data-topic="150134" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aleemsidra/48/38723_2.png" class="avatar"> aleemsidra:</div>
<blockquote>
<p>From the docs:<br>
<strong>pos_weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.<br>
…<br>
In my case I have two classes.</p>
</blockquote>
</aside>
<p>Is it possible that you are conflating two differing usages of the notion of<br>
“two classes?”</p>
<p>In one usage – the situation I think you are dealing with – “two classes”<br>
would typically mean the two classes in a binary-classification problem.<br>
That is the “positive class” (“yes class,” foreground, etc.) and the “negative<br>
class” (“no class,” background, etc.).</p>
<p>In such a case, the <code>input</code> you pass to <code>BCEWithLogitsLoss</code> (the output of<br>
your model) would typically have shape <code>[nBatch]</code>, as would the <code>target</code>.<br>
The <code>pos_weight</code> argument passed into <code>BCEWithLogitLoss</code>’s constructor<br>
would have shape <code>[1]</code>.</p>
<p>In the context of <code>BCEWithLogitsLoss</code>, the other usage of “number of<br>
classes” applies to the so-called *multi-label, multi-class" case where<br>
your have multiple classes (it could be two, but it could be more), <em>each<br>
of which</em> can be “active” or “inactive” for any given sample.  That is, each<br>
sample is labelled with none, some, or all of these classes (hence<br>
“multi-label”).  This is the usage of “number of classes” in your quoted<br>
section of the documentation.</p>
<p>In this situation, your <code>input</code> would have shape <code>[nBatch, nClass]</code>, as<br>
would your <code>target</code>.  <code>pos_weight</code> would now have shape <code>[nClass]</code>.</p>
<p>So, in the “plain-vanilla” version of binary classification, you have “two<br>
classes” (“positive” and “negative”), but for the purposes of the shape<br>
of <code>pos_weight</code>, you have “number of classes” = 1.  It’s just a confusing<br>
choice of terminology.</p>
<p>But what would it mean to have “two classes” in the multi-label, multi-class<br>
setting?  Consider this example:</p>
<p>Class A is “color” (vs. black and white) and class B is “animal” (vs. anything<br>
else, such as a rock or a tree).  None, some (i.e, one), or all (i.e., both) of<br>
these labels can be applied to an image you are classifying.</p>
<p>Thus a color picture of a rock would be “class A – yes” and “class B – no,”<br>
that is, it <em>is</em> a color picture, but it is <em>not</em> an animal.  But you could also<br>
have an image labelled with both classes, that is, “class A – yes” and<br>
“class B – yes”, or labelled with neither class.</p>
<p>If you have further questions, please let us know whether you are dealing<br>
with “plain-vanilla” binary classification or whether you have are working<br>
with a multi-label, multi-class problem.</p>
<p>Best.</p>
<p>K. Frank</p>

aleemsidra: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> , I am calling the get_loss function as:</p>
<pre><code class="lang-auto"> losses = self.get_loss(infer[:,0],labels1.float()) + 0.5 * self.get_loss(infer[:, 1],labels2.float())
</code></pre>
<p>where <code>infer</code> is the <code>model prediction</code>. So, the<code> pred.shape</code> is <code> torch.Size([16])</code>, which is being passed to <code>get_loss</code> function.  I had mistakenly reported the wrong shape originally, it was the original model prediction, but in loss, I am sending <code>torch.Size([16])</code>. Can you please help with this.</p>

aleemsidra: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> , I am dealing with plain vanilla binary classification.</p>
<p><em>In such a case, the <code>input</code> you pass to <code>BCEWithLogitsLoss</code> (the output of</em><br>
<em>your model) would typically have shape <code>[nBatch]</code> , as would the <code>target</code> .</em></p>
<p>The model output is <code>model output: torch.Size([16, 2])</code>. I am calling <code>get_loss()</code> as follows:</p>
<p><code>losses = self.get_loss(infer[:,0],labels1.float()) + 0.5 * self.get_loss(infer[:, 1],labels2.float())</code></p>
<p>So, the <code>pred.shape</code> becomes equal to <code>[nBatch]</code> which is 16 in my case.</p>
<p><em>The <code>pos_weight</code> argument passed into <code>BCEWithLogitLoss</code> 's constructor</em><br>
*would have shape <code>[1]</code><br>
How <code>pos_weight</code> shape should be<code> [1]</code>? Does that mean we only pass the minority class weight and not the majority class?</p>

Andrei_Cristea: <p>Per <a class="mention" href="/u/kfrank">@KFrank</a>’s insightful observation above, it sounds like what you really have is a single binary classification, whereas an output shaped [16,2] and a pos_weight shaped [2] is meant for a two-class binary classification. In writing this I’m realizing the terminology is confusing, maybe we need better terms <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> but I think KFrank did a good job above explaining the distinction.</p>
<p>To be clear, it sounds like what you have is a single class, and for each image your labels say either “no” the image does not contain that class, or “yes” the image does contain that class.</p>
<p>In any case, in your scenario, I think you want the following:</p>
<ul>
<li>Your model output ought to be shaped like [nBatch]. Its value should be quite negative if you want to predict “no” and quite positive if you want to predict “yes”. The reason is that this would map to 0 for “no” and 1 for “yes” once you apply sigmoid, which BCEWithLogitsLoss does. This is because <code>sigmoid(-inf) ~= 0</code> and <code>sigmoid(+inf) ~= 1</code>.</li>
<li>Your label should be shaped like [nBatch] and equal to either 0 or 1, 0 for “no” and 1 for “yes” (unless you use label smoothing, in which case you can use something like 0.05 instead of 0, and 0.95 instead of 1.0)</li>
<li>Your <code>pos_weight</code> should be shaped like [1] since you only have one class. The higher the <code>pos_weight</code>, the bigger the weight you’ll assign, inside your loss function, to how well you did classifying the true positives (i.e. where the labels is 1, meaning “yes”). Per the docs, the purpose of this is to trade off precision and recall (depending on your specific problem, you may care more about avoiding false positives <strong>or</strong> false negatives, and this argument helps you bake that preference in to your fitting).</li>
</ul>
<p>To illustrate, look at this small example:</p>
<pre><code class="lang-auto">model_output = torch.Tensor([+5., -5., -5., -5.])  # you predict "yes, no, no, no"
targets = torch.Tensor([1., 1., 0., 0.])    # actual labels are "yes, yes, no, no"
pos_weight = torch.Tensor([2.]).float()

criterion_simple = nn.BCEWithLogitsLoss()
criterion_posweight = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

print(criterion_simple(model_output, targets))
print(criterion_posweight(model_output, targets))

Output:
tensor(1.2567)
tensor(2.5101)
</code></pre>
<p>When applying <code>pos_weight</code> we get a bigger loss, because the place where the error occurs is where the true label was “yes”.</p>

aleemsidra: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> thank you for your detailed answer.</p>
<aside class="quote no-group" data-username="Andrei_Cristea" data-post="8" data-topic="150134" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>Your <code>pos_weight</code> should be shaped like [1] since you only have one class. The higher the <code>pos_weight</code>, the bigger the weight you’ll assign, inside your loss function, to how well you did classifying the true positives (i.e. where the labels is 1, meaning “yes”)</p>
</blockquote>
</aside>
<p>I am still confused about what actually <code>pos_weght </code> is for. Does it represent the <strong>‘weight of the minority class’</strong> or the <strong>‘weight of the positive class only’</strong> (irrespective of whether it’s a majority or minority class)?  From your explanation, I understood that  <code>pos_weght </code> is for the positive class only (irrespective of whether it’s the majority or minority class). Is my understanding correct? In my case, positive class (1) is a minority class and I want to assign more weight to that. Is <code>pos_weight</code> suitable for this scenario? Or should I use the <code>weight</code> attribute for this scenario.</p>
<p><strong>weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.</p>
<p>Below is the code and respective shapes of tensor, can you please look at it and advise.</p>
<pre><code class="lang-auto">  def get_loss(self, pred, label):

        weights =self.weight[1]
        weights = weights.cuda().half() # using amp, so using half precision
        print("weights", weights, weights.shape)
        print("pred", pred.shape)
        print("label", label.shape)
        criterion = nn.BCEWithLogitsLoss(pos_weight = weights )
        if torch.cuda.is_available():
            criterion.cuda()

        return criterion(pred, label)
</code></pre>
<p>weights tensor(6.7656, device=‘cuda:0’, dtype=torch.float16)<br>
pred torch.Size([16])<br>
label torch.Size([16])</p>

Andrei_Cristea: <p>Yes, from what you’ve described, <code>pos_weight</code> does seem suitable for your scenario. Below I’ll illustrate that it does indeed assign more weight to the positive class, inside the loss calculation.</p>
<p>Before I do that, I just want to mention that you can also achieve a similar effect by using a the vanilla loss function (without <code>pos_weight</code>) but just sampling more frequently from your desired label by using a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedRandomSampler</a>. Personally I prefer this approach, if your objective here is simply to deal with a label imbalance within the class (and you could also use it to deal with class imbalance, though in your case that’s not required since you have a single class). I think the two approaches might be equivalent under some particular parameters and assumptions (someone can correct me if that’s wrong) but intuitively if your positives are very much in the minority, and you try to address that via <code>pos_weight</code>, I feel like your training will be very jumpy. Your training batches either won’t have any positive labels, but when they do you’ll make a very large step in that direction. This feels less robust to me (and a bit less intuitive to think about) than just balancing out your sampling, by sampling from the positives more frequently.</p>
<p>Having said that, here’s an illustration that <code>pos_weight</code> does indeed represent a weight multiplier on the positive label.</p>
<pre><code class="lang-auto">label      = torch.Tensor([  0.,  +1.,   0.,  +1.])
prediction = torch.Tensor([-10., -10., +10., +10.])
#     the labels are      [    neg,   pos,   neg,     pos]
# model's predictions are [correct, wrong, wrong, correct]

crit_basic     = torch.nn.BCEWithLogitsLoss(reduction='none')
crit_posweight = torch.nn.BCEWithLogitsLoss(reduction='none', pos_weight=torch.Tensor([2.]))

print("losses by element:")
print("     basic", crit_basic(prediction, label))
print("pos_weight", crit_posweight(prediction, label))

Output:
losses by element:
     basic tensor([4.5418e-05, 1.0000e+01, 1.0000e+01, 4.5418e-05])
pos_weight tensor([4.5776e-05, 2.0000e+01, 1.0000e+01, 9.0835e-05])
</code></pre>
<p>In this setup, your model happens to be right half the time and wrong half the time. As a reminder, a model prediction of -10 means it expects a negative label, and a prediction of +10 means it expects a positive label (because sigmoid(-10) ~= 0 and sigmoid(+10) ~= +1).</p>
<p>You can see that your vanilla loss (excluding <code>pos_weight</code>) is 0 when the model is right and 1 when the model is wrong. It doesn’t care whether the model was wrong about the positive or the negative label, they are weighted the same.</p>
<p>The loss that uses <code>pos_weight</code> is still equal to 1 when the label was negative, (third element in the tensor) <strong>however it has doubled when the label is positive, the 2nd element in the tensor, going from 1 to 2</strong>. The doubling corresponds to the <code>pos_weight</code> passed, which is 2. Indeed, your loss function now assigns more weight to a mistake made on a positive label, but otherwise behaves the same as before.</p>

aleemsidra: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> , thank you for your detailed explanation. I was actually using <code>weighted random sampler</code> before, but because of the random sampling, the results come out to be quite different every time. I was looking for stability, so, I wanted to try out weighted BCE.</p>
<p>One last question: in my current case, the positive class was in minority. But what if the negative class is in minority, in that scenario too, <code>pos_weight</code> attribute can be for the negative class? Or is it used only when the positive class is in minority? The documentation is bit cinfusing.</p>

Andrei_Cristea: <p>Hello!</p>
<p>Regarding getting different results every time when using weighted random sampler, have you considered <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">controlling your random seed</a>? That would at least ensure that you can always reproduce a particular training experience.</p>
<p>Regarding your question, I think you have two options I can think of:</p>
<ol>
<li>Suppose you want to multiply the weight on your negative (minority) class by N. You can use a <code>pos_weight</code> of <code>1/N</code>. Since the weights multiplicatively toggle the <em>relative</em> importance between the classes, using a weight between 0 and 1 will down-weight the positive class, and therefore implicitly up-weight the negative class, by whatever you’re looking for.</li>
<li>Simply swap the labels on your dataset at training time (make the 1s into 0s and vice-versa), use regular <code>pos_weight</code> (which will boost the weights of your “1” labels, which now correspond to your negative class) and then remember to add a post-processing step to your net output where you swap the labels back (since your net will now learn to predict the swapped labels!)</li>
</ol>
<p>Hope this helps!<br>
Andrei</p>

aleemsidra: <p><a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a>  thank you so much for being patient and giving detailed answers.</p>