Unclear about Weighted BCE Loss

username: string111
cooked: <p>Hey there super people!<br>
I am having issues understanding the <code>BCELoss</code> <code>weight</code> parameter. I am having a binary classification issue, I have an RNN which for each time step over a sequence produces a binary classification. Precisely, it produces an output of size <code>(batch, sequence_len)</code> where each element is in range <code>0 - 1</code> (confidence score of how likely an event happened at one time-step).<br>
Because the ground truth is imbalanced (more 0s than 1s), I would like to make use of <code>BCELoss</code> <code>weight</code> parameter, but I clearly do not understand the the DOCs well enough…</p>
<p>What is the size of this parameter and which elements does it take?</p>

username: ptrblck
cooked: <p>If you would like to use the weight as a class weight, I assume you have a binary target, i.e. it contains only zeros and ones.<br>
In that case I would create a weight tensor and just multiply it with your unreduces loss. Here is a small example:</p>
<pre><code class="lang-python">weight = torch.tensor([0.1, 0.9])
weight_ = weight[y.data.view(-1).long()].view_as(y)
criterion = nn.BCELoss(reduce=False)
loss = criterion(output, y)
loss_class_weighted = loss * weight_
loss_class_weighted = loss_class_weighted.mean()
</code></pre>
<p>Let me know, if that works for you.</p>

username: string111
cooked: <p>Didn’t quite work that well, but I just sampled batches with 1:1 ratio of background:activity labels whch turned out to be a neat solution!</p>

username: ptrblck
cooked: <p>Good to hear.<br>
Maybe the new <code>pos_weight</code> argument for <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss">nn.BCEwithLogitsLoss</a> might work better, since there is a difference in my weighting approach and the implemented one. See <a href="https://github.com/pytorch/pytorch/issues/5660">this discussion</a>.</p>

username: string111
cooked: <p>Do I understand it correctly, that if I have a binary model-output of size <code>(N, 1)</code> the <code>pos_weight</code> argument should be of size <code>(1)</code>? If I have a class distribution of let’s say 900:100 (900 zeros in my ground truth to 100 ones) then <code>pos_weight = torch.Tensor([(900 / 100)])</code>? I guess all zeros are negative examples and the ones are positives, that’s also confusing me <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=5" title=":thinking:" class="emoji" alt=":thinking:"></p>

username: ptrblck
cooked: <p>That’s also how I understand the docs. Did you make any progress using it?</p>

username: string111
cooked: <p>Training <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"> only got two 1080Ti GPUs for ActivityNet…test looked promising though.</p>

username: string111
cooked: <p>Alright, tried it <img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=5" title=":see_no_evil:" class="emoji" alt=":see_no_evil:"><img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=5" title=":see_no_evil:" class="emoji" alt=":see_no_evil:"><img src="https://discuss.pytorch.org/images/emoji/apple/see_no_evil.png?v=5" title=":see_no_evil:" class="emoji" alt=":see_no_evil:"> . My confusion matrix starts with only having FNs and TNs and the model never predicts the class I want it to <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"> (still always outputs tensors of 0s)</p>
<p>EDIT: The softmax output seems to be always a tensor of (0.0325, …, 0.0325), already after the first batch <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

username: ptrblck
cooked: <p>Could your try to play around with your learning rate a bit, i.e. increasing and lowering it?<br>
It smells like your training got stuck.</p>

username: xploiter-projects
cooked: <p>I have pretty much same problem. I have class imbalance of 8:1,<br>
I tried using BCE Loss with different weights i.e.</p>
<ol>
<li>[0.75, 1.25] seems working a bit well but got stuck at the end.</li>
<li>[0.5, 2.5] model diverged sooner.<br>
(Optimizer ASGD with LR 0.01)</li>
</ol>
<p>I used ADAM first, but it didn’t behave nicely so I switched to ASGD.</p>
<p>Is it necessary to have the sum of weights == 1? (if yes, may you please explain why?)</p>
<p>this is my loss function:</p>
<pre><code class="lang-auto">def weighted_binary_cross_entropy(output, target, weights=None):
    output = torch.clamp(output,min=1e-8,max=1-1e-8)

    if weights is not None:
        assert len(weights) == 2
        loss = weights[1] * (target * torch.log(output)) + \
               weights[0] * ((1 - target) * torch.log(1 - output))
    else:
        loss = target * torch.log(output) + (1 - target) * torch.log(1 - output)

    return torch.neg(torch.mean(loss))
</code></pre>

username: Rachel_Copperman
cooked: <p>why did u use the torch.clamp for?</p>
<p>And do u know what numbers are good if my data is 1:100,000 ?</p>

username: xploiter-projects
cooked: <p><a class="mention" href="/u/rachel_copperman">@Rachel_Copperman</a><br>
torch.clamp is used for numerical stability here since output can be “hard one or zero” and torch.log(0) becomes -infinity.</p>

username: yong_xu
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,  i just wonder how to apply the weight  in BCELoss in real training scenario. That is, the dataset is splitted into training, validation and testing subset. What is more,  during model training, there are some epoches and an  epoch  is comprised of one or more batches. So   the weight can be  calculated in  dataset level, training subset level  or even batch level. How to choose? Note that, in batch level, the number of positive or negative in each class may be zero, thus, the following will be occurred: pos_weight = torch.Tensor([(negative examples / 0)]).<br>
Another question is that Adamax and BCELoss is used but my F1 score is just 0.04,   how to tune the performance?</p>

username: ptrblck
cooked: <p>You would usually calculate the <code>pos_weight</code> using the complete training dataset and pass it to the instantiation of the criterion, so that a division by 0 should not occur.</p>