[SOLVED] Class Weight for BCELoss

miguelvr: <p>Hey there,</p>
<p>I’m trying to increase the weight of an under sampled class in a binary classification problem.</p>
<p><code>torch.nn.BCELoss</code> has a <code>weight</code> attribute, however I don’t quite get it as this weight parameter is a constructor parameter and it is not updated depending on the batch of data being computed, therefore it doesn’t achieve what I need.</p>
<p>What is the correct way of simulating a class weight, similar to the way Keras does?</p>
<p>Cheers</p>

miguelvr: <p>Solved with a custom loss function:</p>
<pre><code>def weighted_binary_cross_entropy(output, target, weights=None):
        
    if weights is not None:
        assert len(weights) == 2
        
        loss = weights[1] * (target * torch.log(output)) + \
               weights[0] * ((1 - target) * torch.log(1 - output))
    else:
        loss = target * torch.log(output) + (1 - target) * torch.log(1 - output)

    return torch.neg(torch.mean(loss))</code></pre>

htt210: <p>According to the doc here<br>
<a href="http://pytorch.org/docs/nn.html#bceloss" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/docs/nn.html#bceloss</a><br>
the <code>weight</code> parameter is a tensor of weight for each example in the batch. Thus, it must have the size equal to the batch size. You can set the weight at the beginning of each batch, for example:<br>
<code>criterion = nn.BCELoss() for batch in data: input, label, weight = batch criterion.weight = weight loss = criterion.forward(predct, label) ...</code></p>

miguelvr: <p>That’s a neat solution as well… I ended up changing my loss function again to NLLLoss, which supports class weights, and it’s probably the easiest native solution. My custom loss was giving me NaNs towards the end of training and I have no idea why!</p>

htt210: <p>Did you apply LogSoftmax before computing the loss. NLLLoss takes log probability as input, not the probability.</p>

miguelvr: <p>Yes, I did change the softmax to a log-softmax. The custom loss however, is working with a regular softmax, but I guess it could be related to the lack of an epsilon term to prevent the system to output a “hard one or zero”.</p>

Luckick: <p>Thank you. This works for me.</p>

Alex_Choy: <p>Ref to the c code, there is a safe_log function which returns log(1e-12) if the input is 0.</p>

chuong_nguyen: <p>That may be the numerical unstable. Applying clamp may help:</p>
<pre><code class="lang-auto">output = torch.clamp(output,min=1e-8,max=1-1e-8)  
loss =  pos_weight * (target * torch.log(output)) + neg_weight* ((1 - target) * torch.log(1 - output))
</code></pre>

miguelvr: <p>yes, that’s the easiest way… But you can simply use the NLLoss, it supports class weights now</p>

zl535320706: <p>there should be a “-” in loss function</p>

zl535320706: <p>sorry I’m wrong, I ignored the torch.neg:sweat:</p>

mohamed_ouftou: <p>Hello thanks for ur costum function i use it but i had this eror<br>
The size of tensor a (32) must match the size of tensor b (2) at non-singleton dimension 1<br>
can u help with this problem ?</p>

Alex_Fann: <p>Just a quick question. When applying BCELoss with weights, do we need to normalize the weights with the batch size? Or raw weights would be fine?</p>

David_Ruhe: <p>Hi Miguel,</p>
<p>I’m wondering how you used NLLLoss for a binary classification problem?</p>

TheBloodthirster: <p>Thanks,I just don’t know how to use weight to join NNLLoss.</p>

xulingzhi951118: <p>I think ‘output = torch.clamp(output, 1e-9, 1-1e-9)’ to prevent the output from having 0.</p>

Surabhi_Gupta: <p>hey !</p>
<p>can you explain me how this works ?<br>
I am working on classification problem on celebA dataset, in which, for some features there’s a huge imabalance. How can I rectify this issue with your above mentioned code ?</p>

miguelvr: <p>Just replace BCELoss with CrossEntropyLoss, which has a weight per class, it is probably the easiest solution.</p>

Surabhi_Gupta: <p>criterion = nn.CrossEntropyLoss(weight= torch.tensor([1, 20.4]).to(device))</p>
<p>Is this okay ?<br>
Class1 samples=193140<br>
Class2 samples=9459</p>