Cross Entropy Loss for imbalanced set (binary classification)

Alice_NL: <p>Dear community,</p>
<p>I am trying to use the weights for the binary classification problem for CrossEntropyLoss and by now I am so lost in it….</p>
<p>In my network I set the output size as 1 and have sigmoid activation function at the end to ensure I get values between 0 and 1. I assume it is probability in my case. If output is set as 2 (for class 0 and 1) then for some reason the sum of the columns does not equal 1. This is why I set the output size as 1.</p>
<p>Loss function requires two columns in the output I assume for class 0 and class 1 but which order I am nor sure. I set weights as tensor([1., 5.]) assuming that I have five times more class 0 than class 1.</p>
<p>Thus in the training loop I have:</p>
<p>outputs = nn_model(X_batch)</p>
<h1>I think it is probability for class0. Output of shape 1.</h1>
<p>outputs = outputs.view(X_batch.shape[1]*X_batch.shape[0],output_size)</p>
<h1>I reshape it to one column.</h1>
<p>one = torch.ones(outputs.shape)</p>
<h1>I create a tensor of ones</h1>
<p>class1 = one-outputs</p>
<h1>I calculate probability of  class1 by substracting probability of class 0 from 1.</h1>
<p>outputs = torch.cat((outputs,two), 1)</p>
<h1>I concatenate having 1st column prob class 0 2nd column prob class 1.</h1>
<p>loss = criterion(outputs, labels)</p>
<h1>I assume 1st col is class 0 2nd class1, so weights are tensor([1., 5.]) showing that I have five times more zeroes.</h1>
<p>Is this right? if wrong then please help me to figure out what and why, PLEASE… Any comments are very much appreciated.</p>

KFrank: <p>Hi Alice!</p>
<aside class="quote no-group" data-username="Alice_NL" data-post="1" data-topic="106554" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alice_nl/48/29143_2.png" class="avatar"> Alice_NL:</div>
<blockquote>
<p>I am trying to use the weights for the binary classification problem for CrossEntropyLoss and by now I am so lost in it….</p>
</blockquote>
</aside>
<p>For a binary classification problem (two classes, “yes” and “no”)<br>
you will prefer to use <a href="https://pytorch.org/docs/1.7.1/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> rather than<br>
<code>CrossEntropyLoss</code>.</p>
<p>You will want your model to have a single output (so that the shape<br>
of the output is <code>[nBatch, 1]</code>) and have no final <code>Sigmoid</code> activation<br>
layer.</p>
<p>For your use case you will probably want to use the <code>pos_weight</code><br>
argument to <code>BCEWithLogitsLoss</code>’s constructor.</p>
<aside class="quote no-group">
<blockquote>
<p>showing that I have five times more zeroes.</p>
</blockquote>
</aside>
<p>In this case one would typically use:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_function = torch.nn.BCEWithLogitsLoss (pos_weight = torch.tensor (5.0))
</code></pre>
<p>so that you weight your positive sample five times more heavily than<br>
your five-times-more-frequent negative samples.</p>
<p>If you have questions about how or why to use <code>BCEWithLogitsLoss</code>,<br>
please follow up.</p>
<p>Best.</p>
<p>K. Frank</p>

Alice_NL: <p>Dear K. Frank,<br>
Thank you for your time looking into it, I switched from the BCE loss to CEL because of absence in the BCE the option to add the class weights.<br>
I however did not yet see the BCE with Logits Loss yet.<br>
Am I right that</p>
<ul>
<li>here the output and target will be of the same shape?</li>
<li>I remove sigmoid activation at the end as it is already part of the BCEWithLogitsLoss</li>
</ul>
<p>Thank you again and best regards,<br>
Alice</p>

KFrank: <p>Hi Alice!</p>
<aside class="quote no-group" data-username="Alice_NL" data-post="3" data-topic="106554" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alice_nl/48/29143_2.png" class="avatar"> Alice_NL:</div>
<blockquote>
<p>I switched from the BCE loss to CEL because of absence in the BCE the option to add the class weights.<br>
I however did not yet see the BCE with Logits Loss yet.</p>
</blockquote>
</aside>
<p>For reasons I don’t understand – I suppose that it’s just an inconsistency<br>
or oversight – <code>BCELoss</code> lacks <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code><br>
argument.  It’s not really an issue – <code>BCEWithLogitsLoss</code> should be<br>
used anyways because of its better numerical stability.</p>
<aside class="quote no-group">
<blockquote>
<p>Am I right that</p>
<ul>
<li>here the output and target will be of the same shape?</li>
<li>I remove sigmoid activation at the end as it is already part of the BCEWithLogitsLoss</li>
</ul>
</blockquote>
</aside>
<p>Yes, and yes.</p>
<p>Best.</p>
<p>K. Frank</p>

Alice_NL: <p>Dear K. Frank,</p>
<p>Thank you! It works and I now understand what I am doing there. As what I did before was a mess.</p>
<p>I am very happy.</p>
<p>Kind regards,</p>
<p>Alice</p>

Alice_NL: <p>Dear K. Frank,</p>
<p>I have one more follow up question. For my output I wish to get probabilities. That is why I had sigmoid function at the end of forward. Now with using BCEWithLogitsLoss function I deleted the sigmoid function and my output can be negative and can be bigger than 1 ;(</p>
<p>Do you know how to address this trouble? or did I completely miss something in out previous discussion?</p>
<p>Best, Alice</p>

KFrank: <p>Hi Alice!</p>
<aside class="quote no-group" data-username="Alice_NL" data-post="6" data-topic="106554" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alice_nl/48/29143_2.png" class="avatar"> Alice_NL:</div>
<blockquote>
<p>For my output I wish to get probabilities. That is why I had sigmoid function at the end of forward. Now with using BCEWithLogitsLoss function I deleted the sigmoid function and my output can be negative and can be bigger than 1</p>
</blockquote>
</aside>
<p>Yes, without the <code>Sigmoid</code> activation function, the output of your<br>
model will be raw-score so-called <em>logits.</em>  They run from <code>-inf</code> to<br>
<code>inf</code>, and are what you want as the <code>input</code> to <code>BCEWithLogitsLoss</code>.<br>
<code>Sigmoid</code> maps logits to probabilities that run from <code>0.0</code> to <code>1.0</code>.</p>
<p>You may well want probabilities for certain purposes.  In such a case,<br>
you still want your model to output logits (no <code>Sigmoid</code>) that you feed<br>
to <code>BCEWithLogitsLoss</code> (and then backpropagate).  When you want<br>
probabilities, just apply <code>Sigmoid</code> to your logits (separately from your<br>
model, loss function, and backpropagation) to convert them to<br>
probabilities for whatever subsequent processing you have.</p>
<p>A word of explanation:  The logits and probabilities contain the same<br>
information and can be transformed mathematically back and forth<br>
into one another.  For numerical reasons it’s better to pass logits<br>
from your model to your <code>BCEWithLogitsLoss</code> loss function.</p>
<p>Best.</p>
<p>K. Frank</p>

Alice_NL: <p>Dear K. Frank, Thank you for your prompt and detailed answer. It is very clear now.</p>
<p>Can I then transform logits to probabilities by a new network model (code below)</p>
<pre><code>class Sig(nn.Module):
    def __init__(self):
        super(Sig, self).__init__()
    
        self.activation = nn.Sigmoid()      
    def forward(self, x):
        h = self.activation(x) 
        return h

#create a network 
sigmoid = Sig()
</code></pre>
<p>and if I may cal this mini model within the training loop to calculate accuracy on the validation set?</p>
<p>Best regards,<br>
Alice</p>

KFrank: <p>Hi Alice!</p>
<p>Let me answer your question(s) two different ways.</p>
<aside class="quote no-group" data-username="Alice_NL" data-post="8" data-topic="106554" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alice_nl/48/29143_2.png" class="avatar"> Alice_NL:</div>
<blockquote>
<p>Can I then transform logits to probabilities by a new network model</p>
</blockquote>
</aside>
<p>You could, but doing so would be overkill.  You can just call the<br>
function (or class) version of <code>sigmoid()</code> directly:</p>
<pre data-code-wrap="python"><code class="lang-python">my_logits = my_model (my_batch)
with torch.no_grad():
    my_probabilities = torch.nn.functional.sigmoid (my_logits)
    # or instantiate Sigmoid function obejct on the fly and call it
    # my_probabilities = torch.nn.Sigmoid() (my_logits)
</code></pre>
<p>There is no need to wrap <code>sigmoid()</code> in a “mini-model” in order to<br>
apply it to <code>my_logits</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>calculate accuracy on the validation set?</p>
</blockquote>
</aside>
<p>You generally don’t need the actual probabilities to calculate the<br>
accuracy of your validation-set predictions.  You just need to turn<br>
the logits into binary yes/no predictions.</p>
<p>Also, you may want to calculate the validation-set loss, as well.</p>
<p>Something like:</p>
<pre data-code-wrap="python"><code class="lang-python"># assumes loss_criterion = BCEWithLogitsLoss (...)
with torch.no_grad():   # don't want or need gradients for validation calculations
    val_logits = my_model (val_batch)
    val_loss = loss_criterion (val_logits, val_targets)
    # do something with val_loss
    #
    # assumes that val_targets are exactly 0.0 and 1.0
    val_binary_preds = (val_logits &gt; 0.0).float()
    num_correct = (val_binary_preds == val_targets).sum()
    # use num_correct to calculate average validation accuracy, etc.
</code></pre>
<p>Note, that a logit of zero corresponds to a probability of one half.<br>
(<code>sigmoid (0.0) == 0.5</code>.)  So thresholding logits against <code>0.0</code> gives<br>
the same results as thresholding the corresponding probabilities<br>
against <code>0.5</code>, the idea being that <code>probability &gt; 0.5</code> means “yes”<br>
(and <code>probability &lt;= 0.5</code> means “no”).</p>
<p>Best.</p>
<p>K. Frank</p>

Alice_NL: <p>Thank you so much! This is all so great.</p>
<p>Thank you, Frank.</p>
<p>Kind regards,</p>
<p>Alice</p>