[SOLVED] Class Weighed Binary Crossentropy not working, even with equal weights

username: tzeny
cooked: <p>Hello,</p>
<p>I have tried using the following custom loss class:</p>
<pre><code class="lang-auto"># class BCEWithLogitsLoss():
class CustomWBCE():
    def __init__(self, class_weights=None, **kwargs):
        self.class_weights = class_weights

    def __call__(self, output:Tensor, target:Tensor, **kwargs)-&gt;Rank0Tensor:
        output = torch.sigmoid(output)
        
        if output.min() &lt;= 0 or output.max() &gt;= 1:
#             print('Wrong value in output, will give loss nan')
            output = torch.clamp(torch.sigmoid(output),min=1e-8,max=1 - 1e-8)
        
        if self.class_weights is not None:
            assert len(self.class_weights) == 2

            loss = self.class_weights[1] * (target * torch.log(output)) + \
                   self.class_weights[0] * ((1 - target) * torch.log(1 - output))
        else:
            loss = target * torch.log(output) + (1 - target) * torch.log(1 - output)

        loss = torch.neg(torch.mean(loss))

        return loss
</code></pre>
<p>The reason I apply the sigmoid is because I am using this in a fast.ai Learner, which does not apply an activation function at the end of the network by default. Sometimes my model outputed a very small value (-136 for example) and torch.sigmoid’s result was 0., which led to a -inf in the torch.log; that’s why I added the torch.clamp.</p>
<p>I have tried it both with weights computed with the formula:</p>
<pre><code class="lang-auto">total = negative + positive
w0 = positive / total
w1 = negative / total
</code></pre>
<p>And with weights (0.5, 0.5) to test it out.</p>
<p>Both times I got very poor results:</p>
<p>AUC scores (this ranges from 0 to 1, 0.5 being random) after training with different weights:</p>
<ul>
<li>Calculated weights (0.20, 0.80): 0.51 AUC</li>
<li>Equal weights (0.5, 0.5): 0.45 AUC (worse than random somehow)</li>
</ul>
<p>Now, if I try to train the model in the same way, but using</p>
<p><code>loss_bce = fastai.layers.BCEWithLogitsFlat()</code> (flattens the tensors before applying torch.nn.modules.loss.BCEWithLogitsFlat) it gets to 0.83 AUC.</p>
<p>The model is a DenseNet121 based binary classifier; I trained it on X-Rays from the CheXPert14 dataset.<br>
I am on PyTorch version 1.0.1 and fast.ai version 1.0.52.</p>
<p>Do you have any idea what I’m doing wrong?</p>

username: KFrank
cooked: <p>Hi Andrei!</p>
<aside class="quote no-group quote-modified" data-username="tzeny" data-post="1" data-topic="47055" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tzeny/48/12689_2.png" class="avatar"> tzeny:</div>
<blockquote>
<p>Hello,</p>
<p>I have tried using the following custom loss class:</p>
<pre><code class="lang-auto"># class BCEWithLogitsLoss():
class CustomWBCE():
    def __init__(self, class_weights=None, **kwargs):
        self.class_weights = class_weights
...
</code></pre>
<p>…<br>
Sometimes my model outputed a very small value (-136 for example) and torch.sigmoid’s result was 0., which led to a -inf in the torch.log; that’s why I added the torch.clamp.</p>
<p>I have tried it both with weights computed with the formula:</p>
<pre><code class="lang-auto">total = negative + positive
w0 = positive / total
w1 = negative / total
</code></pre>
<p>And with weights (0.5, 0.5) to test it out.<br>
…</p>
</blockquote>
</aside>
<p>First, to answer the question I think you’re asking:</p>
<p>You should be using (as in the comment in your code)<br>
<code>BCEWithLogitsLoss</code>.</p>
<p><code>BCEWithLogitsLoss</code> supports sample weights, which you<br>
can use for class weights.</p>
<p>Let’s say you have class weight <code>w_1</code> for class 1, and <code>w_0</code><br>
for class 0.  Let <code>w_n</code> be the sample weight for sample <code>n</code>.<br>
Simply set <code>w_n</code> = <code>w_1</code> if <code>y_n</code> = 1, and <code>w_n</code> = <code>w_0</code> if<br>
<code>y_n</code> = 0.  (This assumes that the <code>y_n</code> are either 0 or 1, as<br>
they should be if they are binary class labels.)</p>
<p>Now some comments:</p>
<p>Note that using class weights <code>w_1</code> = <code>w_0</code> = 1/2 doesn’t give<br>
you the same result as an unweighted loss function.  It gives<br>
you 1/2 the unweighted loss function.  (The loss for each sample<br>
is multiplied by 1/2).  This doesn’t matter a lot, but, for example,<br>
with plain-vanilla stochastic-gradient-descent optimization, it<br>
has the effect of reducing your learning rate by a factor of 1/2.</p>
<p>Instead of clamping the sigmoid of your output, you should be<br>
using <code>torch.nn.LogSigmoid</code>.  This avoids the problem of<br>
large negative → sigmoid → 0 → log → -inf.</p>
<p>In general, when you are testing / debugging something like<br>
this, instead of running your full training code with a “default”<br>
value like weights = (0.5, 0.5), you should try calling your<br>
function on a single sample, with your default value and<br>
compare the single numerical result with the result of the<br>
standard unweighted function you are trying to mimic (in<br>
this case <code>BCEWithLogitsLoss</code>).  Only when you are happy<br>
that you have that working should you try running a single<br>
batch, and when that is working, try the training.</p>
<p>Lastly, I think this discussion – especially the comment about<br>
avoiding clamping – applies to your earlier thread:</p><aside class="quote quote-modified" data-post="1" data-topic="46819">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tzeny/48/12689_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/should-i-use-torch-clamp-after-torch-sigmoid/46819">[SOLVED] Should I use torch.clamp after torch.sigmoid?</a> 
  </div>
  <blockquote>
    Hello, I’ve tried to write a custom loss function, a Weighted Binary Cross Entropy. As suggested by <a class="mention" href="/u/miguelvr">@miguelvr</a> here: 
 
 
I’ve tried to use the following function (wrapped inside a class): 
def weighted_binary_cross_entropy(output, target, weights=None):
        
    if weights is not None:
        assert len(weights) == 2
        
        loss = weights[1] * (target * torch.log(output)) + \
               weights[0] * ((1 - target) * torch.log(1 - output))
    else:
        loss = target * torch…
  </blockquote>
</aside>

<p>and its linked thread:</p><aside class="quote" data-post="1" data-topic="3114">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miguelvr/48/366_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/solved-class-weight-for-bceloss/3114">[SOLVED] Class Weight for BCELoss</a> 
  </div>
  <blockquote>
    Hey there, 
I’m trying to increase the weight of an under sampled class in a binary classification problem. 
torch.nn.BCELoss has a weight attribute, however I don’t quite get it as this weight parameter is a constructor parameter and it is not updated depending on the batch of data being computed, therefore it doesn’t achieve what I need. 
What is the correct way of simulating a class weight, similar to the way Keras does? 
Cheers
  </blockquote>
</aside>

<p>Best regards.</p>
<p>K. Frank</p>

username: tzeny
cooked: <p>Thank you for the detailed explanation, and tips for debugging. I am trying the BCEWithLogitsFlat (fast.ai wrap for BCEWithLogits) right now. So far it seems to be as good as its unweighted version.</p>