Pos_weight in binary cross entropy calculation

WuSht: <p>When we deal with imbalanced training data (there are more negative samples and less positive samples), usually <code>pos_weight</code> parameter will be used.<br>
The expectation of <code>pos_weight</code> is that the model will get higher loss when the <code>positive sample</code> gets the wrong label than the <code>negative sample</code>.<br>
When I use the <code>binary_cross_entropy_with_logits</code> function, I found:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

pos_weight = torch.FloatTensor([5])

preds_pos_wrong =  torch.FloatTensor([0.5, 1.5])
label_pos = torch.FloatTensor([1, 0])
loss_pos_wrong = F.binary_cross_entropy_with_logits(preds_pos_wrong, label_pos, pos_weight=pos_weight)

'''
loss_pos_wrong = tensor(2.0359)
'''

preds_neg_wrong =  torch.FloatTensor([1.5, 0.5])
label_neg = torch.FloatTensor([0, 1])
loss_neg_wrong = F.binary_cross_entropy_with_logits(preds_neg_wrong, label_neg, pos_weight=pos_weight)

'''
loss_neg_wrong = tensor(2.0359)
'''

</code></pre>
<p>The losses derived from wrong positive samples and negative samples are the same, so how does <code>pos_weight</code> work in the imbalanced data loss calculation?</p>

ptrblck: <p>I don’t quite understand the example as both losses calculate the same (you just swapped the order or the logits and targets), so did you intend to change the logit of the negative class in the second example?</p>

WuSht: <p>Thank you for replying. I just want to set positive label as (1,0) and negative label as (0,1).<br>
If I have 1 positive sample and 10 negative samples, how can I use <code>pos_weight </code> to make the loss derived from positive sample 10 times greater than negative ones?<br>
In the example, <code>preds_pos_wrong</code> and <code>label_pos</code> are the logits for just one case.<br>
For <code>(0.5, 1.5)</code> in <code>preds_pos_wrong</code>, <code>1.5</code> is the score for the positive class and <code>0.5</code> is the score for the negative class, and its label are (1, 0), which means the possibility of <code>positive class</code> are 1 and the possibility of <code>negative class</code> are 0.<br>
Have I got that right for the usage of <code>preds_pos_wrong</code> or I’m wrong?</p>

WuSht: <p>I found the solution! The mistake I made is that I use two element to indicate the binary label in pytorch,<br>
the correct format for positive label is 1 and negative label is 0, but I wrote (1,0) and (0,1)</p>

tom: <p>Glad you solved it!</p>
<p>If you allow me a comment:</p>
<aside class="quote no-group" data-username="WuSht" data-post="1" data-topic="128279">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wusht/48/40844_2.png" class="avatar"> WuSht:</div>
<blockquote>
<p>When we deal with imbalanced training data (there are more negative samples and less positive samples), usually <code>pos_weight</code> parameter will be used.</p>
</blockquote>
</aside>
<p>I think balancing the data by oversampling the minority class / undersampling the majority class is even more common and - at least in my experience - can be more effective.</p>
<p>Best regards</p>
<p>Thomas</p>

WuSht: <p>Thank you for your suggestion. It sounds great!<br>
I’m modifying my model and I’ll try the sampling method in it.</p>
<p>Sincerely,<br>
Wu Shiauthie</p>