How to interpret the probability of classes in binary classification?

username: shaun
cooked: <p>Hi,<br>
I’m working on a binary classification problem with <code>BCEWithLogitsLoss</code>. My classes are just 0 and 1, such that my output is just single number. During testing, I would like to get the probabilities for each class. After running the test set through the model, I pass the outputed values through <code>torch.sigmoid</code> to get the probabilities. What I would like to know is, what that number signifies.</p>
<p>For example, if one of the output probability is 0.2. Does this mean that there is 0.2 output that this example belongs to class 0 or class 1?</p>
<p>Thanks.</p>

username: KFrank
cooked: <p>Hi Shaun!</p>
<aside class="quote no-group" data-username="shaun" data-post="1" data-topic="45709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shaun/48/349_2.png" class="avatar"> shaun:</div>
<blockquote>
<p>Hi,<br>
I’m working on a binary classification problem with <code>BCEWithLogitsLoss</code>. My classes are just 0 and 1, such that my output is just single number. During testing, I would like to get the probabilities for each class. After running the test set through the model, I pass the outputed values through <code>torch.sigmoid</code> to get the probabilities. What I would like to know is, what that number signifies.</p>
<p>For example, if one of the output probability is 0.2. Does this mean that there is 0.2 output that this example belongs to class 0 or class 1?</p>
<p>Thanks.</p>
</blockquote>
</aside>
<p>In short, the output of <code>sigmoid()</code> is the (predicted) probability<br>
of your sample  being in class “1”.  So, for your example,<br>
P(class 1) = 0.2 (and P(class 2) = 0.8).</p>
<p>This can be found (a bit opaquely) in the <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a><br>
documentation, but you have to kind of parse through the<br>
equations to get this information.</p>
<p>(As an aside, and as you are probably aware, the <code>sigmoid</code><br>
function is, in effect, built into <code>BCEWithLogitsLoss</code>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: shaun
cooked: <p>Thank you for your reply.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="45709">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In short, the output of <code>sigmoid()</code> is the (predicted) probability<br>
of your sample being in class “1”. So, for your example,<br>
P(class 1) = 0.2 (and P(class 2) = 0.8).</p>
</blockquote>
</aside>
<p>So in effect P(class_label == 0) = 0.2 and P(class_label == 1) = 0.8?</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="45709">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(As an aside, and as you are probably aware, the <code>sigmoid</code><br>
function is, in effect, built into <code>BCEWithLogitsLoss</code> .)</p>
</blockquote>
</aside>
<p>Yes, I am aware of this. However, when predicting on the test set, I don’t use the loss function. I just pass the test set to the model and get the output which in this case are not probabilities and are just output values of the model. So I apply a <code>sigmoid</code> to the output to convert the values to the probabilities.</p>
<p>Essentially, after model training during final testing, I run this piece of code:</p>
<pre><code class="lang-auto">classifier = (...)
classifier.load_state_dict(saved_model)
y_pred = classifier(x_test)
probs = torch.sigmoid(y_pred)
predicted_vals = probs &gt; threshold
</code></pre>
<p>Given this code, my assumption is that <code>predicted_vals</code> will contain the predictions of the class labels by the model which I can then compare to <code>y_test</code> to get the final performance of the model. Am I correct in my assumption?</p>

username: jmaronas
cooked: <p>It is the probability that belongs to class 1. In other words, the network outputs p(t=1|x). This observation can be easily derived from applying Bayes Theorem in a binary classification problem. You can see that in the book Pattern Recognition from Duda and Hart of Pattern Recognition for Machine Learning from C.M Bishop</p>

username: shaun
cooked: <aside class="quote no-group" data-username="jmaronas" data-post="4" data-topic="45709">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jmaronas/48/2816_2.png" class="avatar"> jmaronas:</div>
<blockquote>
<p>It is the probability that belongs to class 1. In other words, the network outputs p(t=1|x).</p>
</blockquote>
</aside>
<p>I’m confused. Are you saying the same thing as the other person? When you say class 1 are you referring to the class with class label 0 or class label 1?</p>

username: jmaronas
cooked: <p>Yes the class labelled with a 1.</p>

username: KFrank
cooked: <p>Hi Shaun!</p>
<aside class="quote no-group" data-username="shaun" data-post="3" data-topic="45709">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shaun/48/349_2.png" class="avatar"> shaun:</div>
<blockquote>
<p>Essentially, after model training during final testing, I run this piece of code:</p>
<pre><code class="lang-auto">classifier = (...)
classifier.load_state_dict(saved_model)
y_pred = classifier(x_test)
probs = torch.sigmoid(y_pred)
predicted_vals = probs &gt; threshold
</code></pre>
<p>Given this code, my assumption is that <code>predicted_vals</code> will contain the predictions of the class labels by the model which I can then compare to <code>y_test</code> to get the final performance of the model. Am I correct in my assumption?</p>
</blockquote>
</aside>
<p>Yes, this is correct.  To give a little more detail, <code>y_pred</code> is a<br>
number that indicates how strongly (in some units) the model<br>
is predicting class = “1”.  This number is typically called the<br>
logit.  <code>probs = torch.sigmoid(y_pred)</code> is the predicted<br>
probability that class = “1”.  And <code>predicted_vals</code> is the<br>
predicted class label itself (0 or 1).</p>
<p>As a practical matter, you don’t need to calculate <code>sigmoid</code>.<br>
You can save a little bit of time (but probably trivial) by<br>
leaving it out.</p>
<p>If threshold were 0.5 (that is, predict class = “1” when<br>
P(class = “1”) &gt; 1/2), then you could use<br>
<code>predicted_vals = y_pred &gt; 0</code>.</p>
<p>More generally, you can compare <code>y_pred</code> with the<br>
inverse-sigmoid of the threshold you want.  This is typically<br>
called the logit function, and is given by <code>log (p / (1 - p)</code>.</p>
<p>(Given a probability value p, 0 &lt; p &lt; 1, inverse-sigmoid (p) =<br>
logit (p) = log (p / (1 - p)).)</p>
<p>So:</p>
<pre><code class="lang-auto">logit_threshold = torch.tensor (threshold / (1 - threshold)).log()
...
predicted_vals = y_pred &gt; logit_threshold
</code></pre>
<p>That is, instead of applying <code>sigmoid</code> to all of your <code>y_pred</code>s,<br>
you calculate <code>inverse-sigmoid</code> of <code>threshold</code> once.</p>
<p>Best.</p>
<p>K. Frank</p>

username: shaun
cooked: <p>Thank you for that, it is really helpful.</p>
<p>One point of confusion is when you say class “1”, which class are you actually referring to?</p>
<p>In my problem, I have the following class: non-imminent and imminent which have the labels as 0 and 1 respectively. In other words, non-imminent (class label 0) is the negative class and imminent (class label 1) is the positive class. Furthermore, this is an imbalanced dataset problem with the positive class (imminent, class label 1) having a prevalence of 0.2.</p>
<p>When you say class “1”, would that in this case refer to the imminent (class label 1) positive class? That is the assumption I’ve been working under.</p>
<p>Basically, I’m assuming that the model (after passing the output through a <code>sigmoid</code>) gives me the probability of an imminent (class label 1) positive class. I then define a threshold based on my metric which says that if the probability of the positive class (imminent, class label 1) is greater than the threshold, give the prediction a class label of 1. Else, give it a class label of 0. Is this correct?</p>
<p>Sorry for drilling this on, but this basically determines how I evaluate my model and how it performs and I want to make sure I do it correctly.</p>
<p>Thank you.</p>

username: Paulo_Mann
cooked: <p>Hey <a class="mention" href="/u/shaun">@shaun</a>, AFAIK, it’s exactly what you are saying, the result from the sigmoid for the test set is the probability of the positive class, imminent, class label 1, or even P(class = imminent), as the very good explanation from <a class="mention" href="/u/kfrank">@KFrank</a> points out. And since you have an imbalanced data set, you can work with the “weights” attribute from the BCEWithLogitsLoss, to give different weights for classes with fewer examples</p>

username: KFrank
cooked: <p>Hello Shaun!</p>
<aside class="quote no-group quote-modified" data-username="shaun" data-post="8" data-topic="45709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shaun/48/349_2.png" class="avatar"> shaun:</div>
<blockquote>
<p>…<br>
One point of confusion is when you say class “1”, which class are you actually referring to?<br>
…</p>
</blockquote>
</aside>
<p>In short, “class ‘1’” means whatever you trained you model for it<br>
to mean.</p>
<p>To explain this, let me go back to one of your earlier posts:</p>
<aside class="quote no-group" data-username="shaun" data-post="3" data-topic="45709">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shaun/48/349_2.png" class="avatar"> shaun:</div>
<blockquote>
<p>Essentially, after model training during final testing, I run this piece of code:</p>
<pre><code class="lang-auto">classifier = (...)
classifier.load_state_dict(saved_model)
y_pred = classifier(x_test)
probs = torch.sigmoid(y_pred)
predicted_vals = probs &gt; threshold
</code></pre>
<p>Given this code, my assumption is that <code>predicted_vals</code> will contain the predictions of the class labels by the model which I can then compare to <code>y_test</code> to get the final performance of the model.</p>
</blockquote>
</aside>
<p>You talk about <code>x_test</code> and <code>y_test</code> (and <code>y_pred</code>).  I assume<br>
that <code>y_test</code> is a vector of length the number of test samples,<br>
with each value in <code>y_test</code> being the number 0, meaning that<br>
this sample is in class “0”, or the number 1, meaning class “1”.<br>
(And <code>x_test</code> is the input data whose classes you are trying<br>
to predict.)</p>
<p>You don’t mention it, but I assume you also have an <code>x_train</code><br>
and <code>y_train</code> whose meanings are analogous, and that you<br>
used <code>x_train</code> and <code>y_train</code> to train your network.  The point<br>
is that the meaning of the output of your model (whether a<br>
given value for <code>y_pred = classifier(x_test)</code> means<br>
class “0” or class “1”) depends on how you trained your model.</p>
<p>If you train your model with values of <code>y_train</code> of 1 indicating<br>
class “1” (and feed it into <code>BCEWithLogitsLoss</code>) then larger<br>
(more positive) values of <code>y_pred</code> will mean that you are<br>
predicting class “1” to be more likely and smaller (more negative)<br>
values of <code>y_pred</code> predict class “0” to be more likely (and<br>
class “1” to be less likely), with the predicted probability of<br>
class “'1” given by the <code>sigmoid</code> of <code>y_pred</code>, as discussed in<br>
the earlier posts.</p>
<p>To summarize, the meaning of <code>y_pred</code> depends in a<br>
straightforward way on the meaning of the <code>y_train</code> that you<br>
used to train your classifier.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Paulo_Mann
cooked: <p>Do you have any reference to:</p>
<blockquote>
<p>but you have to kind of parse through equations to get this information</p>
</blockquote>
<p>??</p>

username: KFrank
cooked: <p>Hi Paulo!</p>
<aside class="quote no-group quote-modified" data-username="Paulo_Mann" data-post="11" data-topic="45709" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paulo_mann/48/8380_2.png" class="avatar"> Paulo_Mann:</div>
<blockquote>
<p>Do you have any reference to:</p>
<blockquote>
<p>but you have to kind of parse through equations to get this information</p>
</blockquote>
<p>??</p>
</blockquote>
</aside>
<p>Yes, but you have to read between the lines a little bit.</p>
<p>In short, for the binary (two-class) classification problem, if<br>
you use 0 and 1 as the class labels, such class labels can<br>
be understood as probabilities.</p>
<p>This is at least implicit in the <a href="https://pytorch.org/docs/stable/nn.html#bcewithlogitsloss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> documentation,<br>
where we have the equation (formatted more nicely in the link):</p>
<p>ℓ(x,y)=L={l1​,…,lN​}⊤,ln​=−wn​[yn​⋅logσ(xn​)+(1−yn​)⋅log(1−σ(xn​))]</p>
<p>We have two classes.  Understanding yn to be the given,<br>
known probability of one of the two classes, and therefore<br>
1−yn to be the given probability of the other, we recognize<br>
−​[yn​⋅logσ(xn​)+(1−yn​)⋅log(1−σ(xn​))] to be the cross-entropy<br>
for σ(xn​) to be the predicted probability for one class (and<br>
therefore (1−σ(xn​)) to be the predicted probability of the other).<br>
(xn is the logit for that class predicted by your model and<br>
σ(xn​) = sigmoid(xn​) is the predicted probability for that class.)</p>
<p>So this equation is telling us how we must interpret yn.</p>
<p>In the usual case for labelled training data, yn is the class<br>
label and is therefore equal to 0 or 1.  But this agrees<br>
with understanding yn as a probability:  yn = 0 means<br>
0% probability of being in class “1” which means 100%<br>
probability of being in class “0”.  And yn = 1 means 100%<br>
probability of being in class “1”.</p>
<p>To repeat this with slightly different wording:</p>
<p>yn = Prob (class “1”) = 0  →  class “0”, and<br>
yn = Prob (class “1”) = 1  →  class “1”.</p>
<p>(Later in the documentation, the yn are referred to as “the<br>
targets t[i],” a change of notation that doesn’t help matters.)</p>
<p>It would be much more understandable if the documentation<br>
made clear that this is how the class <strong>labels</strong> enter into<br>
the loss function and gave a concrete example.  But drilling<br>
down into the equation for the loss function does tell us<br>
what the class labels have to mean.</p>
<p>I hope that this is what you were looking for and explains<br>
what I was referring to in my earlier post.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Paulo_Mann
cooked: <p>Thanks, Frank. It helped a lot! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Abhishek_Kumar3
cooked: <p>hello sir. I’m using sigmoid activation fee prediction. I have 30 images in test set. Output will be in the form of<br>
Image.        Prediction.                         Label</p>
<ol>
<li>
<pre><code>          Tensor([0.94,0.123])        tensor([1.,0.])
</code></pre>
</li>
<li>
<pre><code>          Tensor([0.12,0.76])        tensor([0.,1.])
</code></pre>
</li>
<li>
<pre><code>          Tensor([0.96,0.153])        tensor([1.,0.])
</code></pre>
</li>
<li>
<pre><code>          Tensor([0.94,0.23])        tensor([1.,0.])
</code></pre>
</li>
</ol>
<p>And so on for 30 images.<br>
Can you please tell me about labels and how to predict single probability?</p>