Binary classification model outputs low probabilities

Nyakov: <p>Hi, I have binary classification model for detecting anomalies i data.<br>
The problem is, it outputs low probabilities (0.2 at maximum) when I expect from it to output 0.5 and more for related events.<br>
Events that I looking for is sparse so they appear rarely in train data. Can it be “unbalanced data” problem (I read somewhere about it).<br>
But I kind of want for network to learn on all negative examples as well.<br>
Am I doing something wrong?<br>
I am using BCELoss, Adam optimizer, and my model looks like this:</p>
<pre><code class="lang-auto">BinaryClassification(
  (gru): GRU(38, 1024, batch_first=True)
  (fc_layer): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): SELU()
    (2): Linear(in_features=512, out_features=1, bias=True)
    (3): Sigmoid()
  )
)
</code></pre>
<p>Maybe there is some different approach to solve such problems?</p>

KFrank: <p>Hi Nyakov!</p>
<aside class="quote no-group" data-username="Nyakov" data-post="1" data-topic="172197" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nyakov/48/51180_2.png" class="avatar"> Nyakov:</div>
<blockquote>
<p>Can it be “unbalanced data” problem</p>
</blockquote>
</aside>
<p>Yes, it very well could be.  If you train on a lot of negative samples and<br>
very few positive samples, your network can get trained to (almost)<br>
always predict “negative” without really “learning” anything about how<br>
your “negative” and “positive” samples differ and still get a low loss<br>
function because blindly predicting “negative” will almost always be<br>
correct.</p>
<p>My preferred approach is to sample from the training set with positive<br>
samples weighted enough more heavily that a typical batch has about<br>
equal numbers of positive and negative samples.</p>
<p>A second approach – which I would use if the number of positive samples<br>
in your training set is so small that the first approach frequently results in<br>
duplicate positive samples in a batch – is to use <code>BCEWithLogitsLoss</code>’s<br>
<code>pos_weight</code> constructor argument to weight the positive samples more<br>
heavily in the loss function.</p>
<aside class="quote no-group">
<blockquote>
<p>I am using BCELoss</p>
<pre><code class="lang-auto">    (2): Linear(in_features=512, out_features=1, bias=True)
    (3): Sigmoid()
</code></pre>
</blockquote>
</aside>
<p>As an aside, you should be using <code>BCEWithLogitsLoss</code> (instead of<br>
<code>BCELoss</code>) <em>without</em> the final <code>Sigmoid</code> layer as it has better numerical<br>
stability.</p>
<p>Best.</p>
<p>K. Frank</p>

Nyakov: <p>Thank you, <code>pos_weight</code> worked for me, although learning error go up, but it seems model is working.</p>
<p>Sampling is not the option for me, because positive samples is sparse (62 negative to one positive) and also I use one big tensor from with I take data by sliding window, sampling will up RAM consumption to high for my system, or makes things to complex for the moment.</p>
<p>I used (negative_samples / positive_samples) as a value for <code>pos_weight</code>, and it works for now, maybe it is a bit extreme?</p>

KFrank: <p>Hi Nyakov!</p>
<aside class="quote no-group" data-username="Nyakov" data-post="3" data-topic="172197" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nyakov/48/51180_2.png" class="avatar"> Nyakov:</div>
<blockquote>
<p>I used (negative_samples / positive_samples) as a value for <code>pos_weight</code>,</p>
</blockquote>
</aside>
<p>This is the most sensible value of <code>pos_weight</code> to start with unless you have<br>
some specific reason to use something different.</p>
<p>The next thing you can do – if you care – is to look at your false-negative and<br>
false-positive rates after you have trained.  You might want them to be about<br>
the same (if they aren’t) or you might care about improving one at the cost of<br>
degrading the other.</p>
<p>If you want to decrease your false-negative rate – that is, correctly predict<br>
“positive” for more of your actual positive inputs – you would train with a<br>
larger value of <code>pos_weight</code> (but doing so would typically come at the cost<br>
of an increased false-positive rate).</p>
<p>Best.</p>
<p>K. Frank</p>

Nyakov: <p>Sorry for noob question.<br>
But when I have multiple classes and use <code>BCEWithLogitsLoss</code> as loss function, when I evaluate the model should I apply <code>Softmax</code> to the output or <code>Sigmoid</code> ?</p>

KFrank: <p>Hi Nyakov!</p>
<aside class="quote no-group" data-username="Nyakov" data-post="5" data-topic="172197" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nyakov/48/51180_2.png" class="avatar"> Nyakov:</div>
<blockquote>
<p>I have multiple classes</p>
</blockquote>
</aside>
<p>Do you have what I would call a single-label, <em>multi-class</em> problem, where<br>
each sample is labelled with exactly one of multiple labels?  Or do you have<br>
a <em>multi-label,</em> multi-class problem, where each sample is labelled with none,<br>
some, or all of the labels, all at once?</p>
<p>In both cases, your final layer should be a <code>Linear</code> with <code>out_features</code> equal<br>
to the number of classes (and no subsequent layer – neither <code>Softmax</code> nor<br>
<code>Sigmoid</code>).  But for a single-label problem you would use <code>CrossEntropyLoss</code><br>
as your loss criterion, while for a multi-label problem, you would continue to<br>
use <code>BCEWithLogitsLoss</code>.</p>
<p>(Internally, <code>CrossEntropyLoss</code> applies <code>log_softmax()</code> and<br>
<code>BCEWithLogitsLoss</code> applies <code>logsigmoid()</code>, so you are, in effect,<br>
using <code>softmax()</code> and <code>sigmoid()</code> for the single-label and multi-label<br>
case, respectively, but don’t use them explicitly, as they’re built into<br>
the loss criteria.)</p>
<p>Best.</p>
<p>K. Frank</p>

Nyakov: <p>I have two labels for each sample that can be either 1 or 0, technically they cannot be both 1 at the same time but they can be 0 at the same time.<br>
Its dog/cat probability or neither of both.<br>
I used two separate networks before (one for dogs and one for cats) but now decided to merge them into one.</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="172197">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(Internally, <code>CrossEntropyLoss</code> applies <code>log_softmax()</code> and<br>
<code>BCEWithLogitsLoss</code> applies <code>logsigmoid()</code>, so you are, in effect,<br>
using <code>softmax()</code> and <code>sigmoid()</code> for the single-label and multi-label<br>
case, respectively, but don’t use them explicitly, as they’re built into<br>
the loss criteria.)</p>
</blockquote>
</aside>
<p>Ok, got it, so if I use <code>BCEWithLogitsLoss</code> and after training, when I use model for classification, and want to get probabilities from It, I should apply <code>sigmoid()</code> to model outputs?</p>

KFrank: <p>Hi Nyakov!</p>
<aside class="quote no-group" data-username="Nyakov" data-post="7" data-topic="172197" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nyakov/48/51180_2.png" class="avatar"> Nyakov:</div>
<blockquote>
<p>Its dog/cat probability or neither of both.</p>
</blockquote>
</aside>
<p>If I understand you correctly, a given sample could be a dog or a cat<br>
or neither, but not both.</p>
<p>That is, you might have a picture of a dog or a cat or an empty scene<br>
or a bunny, but you would not have a picture containing both a dog and<br>
a cat.</p>
<p>I would treat this as a single-label, three-class problem.  Your model<br>
should have three outputs, that is, its final layer should be a <code>Linear</code><br>
with <code>out_features = 3</code>, and the three outputs represent the predicted<br>
(unnormalized) log-probabilities for each of the three classes.</p>
<p>You should then use <code>CrossEntropyLoss</code> as your loss criterion.</p>
<aside class="quote no-group">
<blockquote>
<p>after training, when I use model for classification, and want to get probabilities from It, I should apply <code>sigmoid()</code> to model outputs?</p>
</blockquote>
</aside>
<p>If you need probabilities (but you don’t always need them), you would<br>
apply <code>softmax()</code> to the output of your model to convert the unnormalized<br>
log-probabilities to a proper probability distribution over the three classes<br>
where the three probabilities sum to one.</p>
<p>Best.</p>
<p>K. Frank</p>