Predictions stuck at zero when positive label (1) is only 16% of data

username: Mona_Jalal
cooked: <p>So, I run the same code with a 50/50 split of 0 and 1 label, I get aboyt 70% accuracy on val set and my val preds are not stuck at 0.</p>
<p>However, when I run the code on a dataset with 84/16 % split of labels 0 and 1, all my val preds end up being 0. I used both cross-entropy loss as well as BCEWithLogitLoss with a weight vector (though I am not sure if I set the weight vector correctly). Also, is <code>  weight = torch.tensor([0.84, 0.16])</code> correct?</p>
<p>How can I fix this problem?</p>
<pre><code class="lang-auto">
        loss_type = 'BCEWithLogitsLoss'
        if loss_type == 'BCEWithLogitsLoss':
            self.criterion = nn.BCEWithLogitsLoss(reduction='none') # weighted loss for imbalanced dataset
            #self.criterion = nn.BCEWithLogitsLoss() 
        elif loss_type == 'CrossEntropyLoss':
            self.criterion = nn.CrossEntropyLoss() # this should work for binary classification

if loss_type == 'BCEWithLogitsLoss':
            labels = torch.as_tensor(labels, dtype=torch.float32) # we need float labels for BCEWithLogitsLoss
            weight = torch.tensor([0.84, 0.16]) # how to decide on this weights?
            #weight = torch.tensor([0.5, 0.5])
            weight_ = weight[labels.data.view(-1).long()].view_as(labels)
            m = nn.Sigmoid()
            with torch.cuda.amp.autocast():
                loss = self.criterion(m(out[:,1]-out[:,0]), labels.cuda())    
                loss_class_weighted = loss * weight_.cuda()
                loss_class_weighted = loss_class_weighted.mean()
                loss = loss_class_weighted
        elif loss_type == 'CrossEntropyLoss':
            labels = torch.as_tensor(labels)
            with torch.cuda.amp.autocast():   
                loss = self.criterion(out, labels.cuda())
        pred_labels = out.data.max(1)[1]
        #pred_labels = out.argmax(dim=1)
        labels = labels.int()  
        return pred_labels, labels, loss
</code></pre>

username: KFrank
cooked: <p>Hi Mona!</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="1" data-topic="145203" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>I used both cross-entropy loss as well as BCEWithLogitLoss with a weight vector</p>
</blockquote>
</aside>
<p>Let me outline how to use <code>BCEWithLogitsLoss</code> and <code>CrossEntropyLoss</code><br>
with class weights without commenting directly on your code.</p>
<p>When performing binary classification, I prefer using <code>BCEWithLogitsLoss</code>.<br>
Doing so more nearly “says what you mean” than <code>CrossEntropyLoss</code><br>
does, and is marginally more efficient.</p>
<p>It is perfectly reasonable, however, to treat binary classification as the<br>
two-class case of multi-class classification, and use <code>CrossEntropyLoss</code>.</p>
<p>Let’s assume that the input to your model is a batch of <code>nBatch</code> samples.<br>
In typical usage with <code>BCEWithLogitsLoss</code> the final layer of your network<br>
will be a <code>Linear</code> with <code>out_features = 1</code> and the output of your model<br>
will be a batch of <code>nBatch</code> <em>logit</em> predictions with shape <code>[nBatch, 1]</code>.</p>
<p>If you were to convert your logit to a probability (You don’t – this is done<br>
internally, in effect, in <code>BCEWithLogitsLoss</code>.), you would get the predicted<br>
probability of your sample being in “class-1” (the “positive” class).  Your<br>
<code>target</code> would be the known probability of your sample being in “class-1”<br>
and can be exactly <code>0.0</code> or <code>1.0</code>, in which case it is easy to think of this<br>
probability as being a 0 / 1 label where <code>0.0</code> means “class-0” and <code>1.0</code><br>
means “class-1.”</p>
<p>If your data is unbalanced, such as in your case where 16% of your training<br>
samples are in “class-1,” you can use <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code><br>
constructor argument to weight the “class-1” samples more heavily in the<br>
calculated loss.  You would typically use a weight of class-0-% / class-1-%;<br>
thus in your case you might use <code>pos_weight = torch.tensor ([5.25])</code>.</p>
<p>If, instead, you choose to treat this as the two-class case of a multi-class<br>
problem and use <code>CrossEntropyLoss</code>, your final layer should be a <code>Linear</code><br>
with <code>out_features = 2</code>, that is, separate output values for “class-0” and<br>
“class-1”, and the output of your model would be a batch of class<br>
predictions with shape <code>[nBatch, 2]</code>  (These are again logits that are, in<br>
effect, internally converted to probabilities in <code>CrossEntropyLoss</code>.)  Your<br>
<code>target</code> will be (a batch of) integer class labels that take on the values <code>0</code><br>
(for “class-0”) and <code>1</code> (for “class-1”), and will have shape <code>[nBatch]</code>.</p>
<p>To weight your (two) classes in the loss calculation, you would use<br>
<code>CrossEntropyLoss</code>’s <code>weight</code> constructor argument.  Now, instead<br>
of a single weight (such as <code>BCEWithLogitLoss</code>’s <code>pos_weight</code>) for<br>
“class-1,” you will have a weight for each of your (two) classes.  You<br>
would typically weight each class proportionally to the reciprocal of<br>
the frequency with which it appears in your training data.  So in your<br>
case, you could use <code>weight = torch.tensor ([1.0, 5.25])</code>.</p>
<p>Note, another approach to compensating for unbalanced training data<br>
is to sample the underrepresented class more heavily.  In your case<br>
you could build your training batches by sampling randomly from your<br>
training data, but sample any specific “class-1” sample <code>5.25</code> times as<br>
often as any specific “class-0” sample.  Now a given batch will contain,<br>
on average, an equal number of “class-1” samples and “class-0” samples.<br>
(You can use this technique with both the <code>BCEWithLogitsLoss</code> and the<br>
<code>CrossEntropyLoss</code> approach, and you would no longer use class weights<br>
in the loss calculation.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mona_Jalal
cooked: <p>Dear K. Frank, thanks a lot for your explanation. However, I am not sure how you arrived to <code>weight = torch.tensor ([1.0, 5.25])</code> if my class 0 is 84% of data and class 1 is 16% of the data. What is the formula you used?</p>
<p>Thanks a bunch,<br>
Mona</p>

username: KFrank
cooked: <p>Hi Mona!</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="3" data-topic="145203" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>I am not sure how you arrived to <code>weight = torch.tensor ([1.0, 5.25])</code></p>
</blockquote>
</aside>
<p>Typically, we reweight classes with the reciprocal of how often they appear.<br>
However, <code>CrossEntropyLoss</code> doesn’t care about the <em>overall</em> weight, as<br>
it computes the <em>weighted</em> average of the individual sample losses.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; class_frequencies = torch.tensor ([0.84, 0.16])
&gt;&gt;&gt; class_weightsA = 1 / class_frequencies
&gt;&gt;&gt; class_weightsA
tensor([1.1905, 6.2500])
&gt;&gt;&gt; class_weightsB = class_frequencies[0] * class_weightsA
&gt;&gt;&gt; class_weightsB
tensor([1.0000, 5.2500])
</code></pre>
<p>Here, <code>class_weightsA</code> and <code>class_weightsB</code> differ only in their overall<br>
scale – the relative weights of “class-0” and “class-1” are the same, so<br>
they’ll give the same result with <code>CrossEntropyLoss</code>.</p>
<p>I chose to use <code>class_weightsB</code> for my example to make numerically<br>
obvious the relationship to <code>pos_weight</code> in the <code>BCEWithLogitsLoss</code><br>
version.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mona_Jalal
cooked: <p>Thanks for the response. Just to be clear, does this apply also to <code>BCEWithLogitLoss</code> since that is what I intend to use.</p>
<p>Should I use <code>[1.0, 5.25]</code> instead of <code>[0.84, 0.16]</code> also for <code>BCEWithLogitLoss</code> or is that one correct only for <code>CrossEntropyLoss</code>?</p>
<p>^ Asking this because you mentioned:</p>
<blockquote>
<p>Typically, we reweight classes with the reciprocal of how often they appear.<br>
However, <code>CrossEntropyLoss</code> doesn’t care about the <em>overall</em> weight, as<br>
it computes the <em>weighted</em> average of the individual sample losses.</p>
</blockquote>
<pre><code class="lang-auto">if loss_type == 'BCEWithLogitsLoss':
            labels = torch.as_tensor(labels, dtype=torch.float32) # we need float labels for BCEWithLogitsLoss
            weight = torch.tensor([0.84, 0.16]) # how to decide on this weights?
            #weight = torch.tensor([0.5, 0.5])
            weight_ = weight[labels.data.view(-1).long()].view_as(labels)
            m = nn.Sigmoid()
            with torch.cuda.amp.autocast():
                loss = self.criterion(m(out[:,1]-out[:,0]), labels.cuda())    
                loss_class_weighted = loss * weight_.cuda()
                loss_class_weighted = loss_class_weighted.mean()
                loss = loss_class_weighted
</code></pre>

username: KFrank
cooked: <p>Hi Mona!</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="5" data-topic="145203" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<p>Just to be clear, does this apply also to <code>BCEWithLogitLoss</code> since that is what I intend to use.</p>
<p>Should I use <code>[1.0, 5.25]</code> instead of <code>[0.84, 0.16]</code> also for <code>BCEWithLogitLoss</code> or is that one correct only for <code>CrossEntropyLoss</code>?</p>
</blockquote>
</aside>
<p>No, you should use neither version with <code>BCEWithLogitsLoss</code>.  Instead,<br>
use a single <code>pos_weight</code> constructor argument:</p>
<pre data-code-wrap="python"><code class="lang-python">self.criterion = nn.BCEWithLogitsLoss (pos_weight = torch.tensor ([5.25]))
</code></pre>
<p>(This assumes that you are performing ordinary binary classification,<br>
rather than <em>multi-label,</em> multi-class classification.)</p>
<p>Some brief further comments:</p>
<aside class="quote no-group" data-username="Mona_Jalal" data-post="1" data-topic="145203">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mona_jalal/48/25715_2.png" class="avatar"> Mona_Jalal:</div>
<blockquote>
<pre><code class="lang-auto">        if loss_type == 'BCEWithLogitsLoss':
            self.criterion = nn.BCEWithLogitsLoss(reduction='none') # weighted loss for imbalanced dataset 
</code></pre>
</blockquote>
</aside>
<p>By using <code>pos_weight</code>, you will be telling <code>BCEWithLogitsLoss</code> to perform<br>
the weighting for you.  So you can also let it perform the <code>'mean'</code> reduction<br>
for you.  That is, you don’t need to perform the weighted average by hand.</p>
<p>Just use</p>
<pre data-code-wrap="python"><code class="lang-python">        if loss_type == 'BCEWithLogitsLoss':
            self.criterion = nn.BCEWithLogitsLoss (pos_weight = torch.tensor ([5.25]))
            # or equivalently
            self.criterion = nn.BCEWithLogitsLoss (pos_weight = torch.tensor ([5.25]), reduction = 'mean')
</code></pre>
<p>and then leave out the part where you do the weighting by hand:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">                loss = self.criterion(m(out[:,1]-out[:,0]), labels.cuda())    
                loss_class_weighted = loss * weight_.cuda()
                loss_class_weighted = loss_class_weighted.mean()
                loss = loss_class_weighted
</code></pre>
</blockquote>
</aside>
<p>Lastly:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            m = nn.Sigmoid()
            with torch.cuda.amp.autocast():
                loss = self.criterion(m(out[:,1]-out[:,0]), labels.cuda())    
</code></pre>
</blockquote>
</aside>
<p>you do not want to apply <code>Sigmoid</code> to the output of your model.  This<br>
is already done for you, in effect, inside of <code>BCEWithLogitsLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>