Loss function for binary classification

ykukkim: <p>Hey all,</p>
<p>I am trying to utilise BCELoss with weights, but I am struggling to understand. I currently am using  LSTM model to detect an event in time-series data. My output from the model and true_output are as follows<code>[batch_size, seq_length]</code>.</p>
<p>Currently, I think I have managed to do hard code it but it’s not the best way to achieve this.</p>
<pre><code class="lang-auto">         loss_get = self.criterion(predictions.float(), target.float())
            # Weighted Binary Corss_entropy Function
            loss_flat = loss_get.flatten()
            target_flat = target.flatten()
            loss_flat[target_flat == 1] *= self.pos_weight_factor
            loss = loss_flat.mean()
            loss.backward()
</code></pre>
<p>My datasets are imbalance, meaning that I do not have a constant length of the dataset as well as there are more 0’s than 1’s, approximately 100:1, hence I need to penalise the 0’s by multiplying it with an arbitrary number. I understand that there are a few topics on this, but I cannot quite get my head around it.<a href="https://discuss.pytorch.org/t/how-to-apply-a-weighted-bce-loss-to-an-imbalanced-dataset-what-will-the-weight-tensor-contain/56823" class="inline-onebox">How to apply a weighted BCE loss to an imbalanced dataset? What will the weight tensor contain?</a></p>
<p>Therefore If I wanted to apply weights with this using the built-in function or solution has suggested by this post. <a href="https://discuss.pytorch.org/t/solved-class-weight-for-bceloss/3114">https://discuss.pytorch.org/t/solved-class-weight-for-bceloss/3114?u=ykukkim</a></p>
<p>Can anyone guide me through this?</p>
<p>Thanks!</p>

KFrank: <p>Hello Yong Kuk!</p>
<aside class="quote no-group quote-modified" data-username="ykukkim" data-post="1" data-topic="72150" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ykukkim/48/21341_2.png" class="avatar"> ykukkim:</div>
<blockquote>
<p>I am trying to utilise BCELoss with weights, but I am struggling to understand.<br>
…<br>
My datasets are imbalance, meaning that I do not have a constant length of the dataset as well as there are more 0’s than 1’s, approximately 100:1,</p>
</blockquote>
</aside>
<p>The most straightforward way to do this (and also better for numerical<br>
reasons) is to adjust your network so that it outputs raw-score <em>logits</em><br>
for its predictions, rather than probabilities.  (For example, if the last<br>
layer of your network is a <code>Sigmoid</code> – that converts a logit to a<br>
probability – just get rid of the <code>Sigmoid</code> layer.)</p>
<p>Then use <code>BCEWithLogitsLoss</code> instead of <code>BCELoss</code>.  This is because<br>
<code>BCEWithLogitsLoss</code> offers a <code>pos_weight</code> argument that it uses to<br>
reweight positive samples in the loss function.  In your case you would<br>
set <code>pos_weight</code> to something like 100.  (<code>BCELoss</code> does <em>not</em> have a<br>
<code>pos_weight</code> argument – probably just an oversight, rather than for<br>
any particular reason.)</p>
<p>For some further details, please take a look at this recent thread:</p>
<aside class="quote quote-modified" data-post="1" data-topic="71667">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/3bc359/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/binary-segmentation-bcewithlogitsloss-pos-weight/71667">Binary segmentation BCEWithLogitsLoss pos_weight?</a> 
  </div>
  <blockquote>
    I’m segmenting foreground vs background using unet and there are many more 0s than 1s due to this. It looks like    i should be using BCEWithLogitsLoss as my loss function? 
It looks like this function takes as an argument the proportion of class imbalance - “pos_weight (Tensor, optional) – a weight of positive examples. Must be a vector with length equal to the number of classes.” 
Does that mean i need to know across all my images how many pixels are 1 v 0 <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title="sweat_smile" alt="sweat_smile" class="emoji"> If so is there any best…
  </blockquote>
</aside>

<p>Good luck!</p>
<p>K. Frank</p>

ykukkim: <p>Hey Frank,</p>
<p>Thank you for your reply! Your reply has cleared some aspects. Furthermore, reading through your thread it helped me even more.</p>
<p>However, I am very new to machine learning, and I am slightly confused with the following terms:</p>
<blockquote>
<p><em>multi-label, multi-class</em>  classification</p>
</blockquote>
<p>Would you care to explain this for me?</p>
<p>Furthermore, your method seems to me that what I already have done is pretty much the same? as the<code> sigmoid</code> function is internally performed with <code>BCEWithLogitsloss</code> Have I understood correctly?</p>
<p>Thank you!</p>

KFrank: <p>Hi Yong Kuk!</p>
<aside class="quote no-group quote-modified" data-username="ykukkim" data-post="3" data-topic="72150" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ykukkim/48/21341_2.png" class="avatar"> ykukkim:</div>
<blockquote>
<blockquote>
<p><em>multi-label, multi-class</em>  classification</p>
</blockquote>
<p>Would you care to explain this for me?</p>
</blockquote>
</aside>
<p>By way of example, in a conventional three-class (“cat,” “dog,” “bird”)<br>
classification problem, given an image, you would say that it is an<br>
image of exactly one of a cat or a dog or a bird.  (And you wouldn’t say<br>
it was “none of the above” unless you explicitly had a fourth, “none of<br>
the above” class.)</p>
<p>In a multi-label (and in this case, three-class) classification problem<br>
you would say that an images does or does not contain a cat, and<br>
also does or does not contain a dog, and also does or does not contain<br>
a bird.  It can contain any combination, and it might not contain any of<br>
the above, and it might contain all three.  You can see that such a<br>
multi-label problem is three <em>binary</em> problems (cat: yes or no, dog:<br>
yes or no, bird: yes or no) run at the same time with the same network.</p>
<aside class="quote no-group">
<blockquote>
<p>the<code> sigmoid</code> function is internally performed with <code>BCEWithLogitsloss</code> Have I understood correctly?</p>
</blockquote>
</aside>
<p>Yes, <code>BCEWithLogitsLoss</code> calculates <code>LogSigmoid</code> internally (in effect<br>
calculating <code>Sigmoid</code> internally).  This is numerically more stable than<br>
passing your logits through <code>Sigmoid</code> and then passing them to<br>
<code>BCELoss</code>.  (Unless you have specific reason why you <em>need</em> to use<br>
<code>BCELoss</code> – and understand it – you should always use<br>
<code>BCEWithLogitsLoss</code> instead.)</p>
<p>Best.</p>
<p>K. Frank</p>