Using BCELoss() with real-valued labels without any correspondance to a class

MngFrc: <p>Hi,</p>
<p>I start saying that I know the BCELoss is generally exploited when there’s a classification problem, but I’m also quite new with ML.<br>
I’m trying to implement a system that is explained in a paper, in which it is said they built a NN whose output layer is formed by a single neuron with Sigmoid activation function. Consequently, the output of the NN is a number comprised between zero and one. Based on the comparison of this latter and a threshold, they decide to accept or refuse a given parameter’s value (which was given as input to the NN). And that’s why they call this NN classifier, even if its output is a real value which doesn’t correspond to any class. Moreover, it is written that the loss function they use is the Binary Cross Entropy.</p>
<p>Summarizing, I should train a NN so that it outputs a certain value between 0 and 1 using the BCELoss. So the label for each input will be a real quantity - 0.02, 0.96, … .</p>
<p>So, what I would like to ask is: is it really possible to use the BCELoss function in such a case or maybe I’m missing something? I’m pretty sure to have well understood the paper, but maybe I should train the NN differently.</p>
<p>[ I say sorry if I do not report any code and/or if the type of question is anomalous, but I’m new also for these kind of forums. If there are problems with my topic, I will delete it asap ]</p>

KFrank: <p>Hi MngFrc!</p>
<aside class="quote no-group quote-modified" data-username="MngFrc" data-post="1" data-topic="77015" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> MngFrc:</div>
<blockquote>
<p>BCELoss is generally exploited when there’s a classification problem<br>
…<br>
Consequently, the output of the NN is a number comprised between zero and one. Based on the comparison of this latter and a threshold, they decide to accept or refuse a given parameter’s value (which was given as input to the NN). And that’s why they call this NN classifier</p>
</blockquote>
</aside>
<p>Yes, this is a conventional binary-classification problem.  The<br>
input value for the parameter is either accepted – class-“1”,<br>
“yes” – or refused – class-“0”, “no”.  That is, the input has been<br>
classified into one of two classes.</p>
<aside class="quote no-group">
<blockquote>
<p>even if its output is a real value which doesn’t correspond to any class.</p>
</blockquote>
</aside>
<p>This is standard.  The output of the classifier – a continuous<br>
number between 0 and 1 – is understood as the predicted<br>
<em>probability</em> of the input being in class-“1”.  This is good for a<br>
number of reasons, not least of which is that it is the result of<br>
a differentiable computation and can be fed into a differentiable<br>
loss function, so the model can be trained with back-propagation.</p>
<aside class="quote no-group">
<blockquote>
<p>Moreover, it is written that the loss function they use is the Binary Cross Entropy.<br>
…<br>
is it really possible to use the BCELoss function in such a case?</p>
</blockquote>
</aside>
<p>Yes.  The key is that the continuous output of the model is to be<br>
understood as the predicted probability of the input being in<br>
class-“1”.  And this is exactly what <code>BCELoss</code> expects.</p>
<p>(As an aside, for reasons of numerical stability, you’ll be better<br>
off removing the <code>Sigmoid</code> and using <code>BCEWithLogitsLoss</code>.<br>
This is mathematically – but not numerically – equivalent.)</p>
<p>Best.</p>
<p>K. Frank</p>

MngFrc: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>!</p>
<p>First of all, thank you for your reply.<br>
The thing that was not allowing me to interprete my problem as a classification one was that my threshold is not 0.5, but definitely smaller (let’s suppose 0.1)<br>
But thanks to your words I have understand the right point of view.</p>
<p>Can I ask you how, since my classes are unbalanced, can I train my NN? When I have a label equal to 0.9 for example - or any value greater than the threshold - should I round it to 1? Or should I differently weight somehow the 2 terms belonging to the BCELoss?</p>
<p>The fact is that if I have a label equal to 0.5, the BCE Loss will not decrease to zero by its definition of I leave the label unchanged. But also, I think that I should differentiate a label equal to 0.5 with respect to one equal to 0.9</p>

KFrank: <p>Hi MngFrc!</p>
<aside class="quote no-group" data-username="MngFrc" data-post="3" data-topic="77015" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> MngFrc:</div>
<blockquote>
<p>The thing that was not allowing me to interprete my problem as a classification one was that my threshold is not 0.5, but definitely smaller (let’s suppose 0.1)</p>
</blockquote>
</aside>
<p>Where does this threshold come from?  Is it somehow given as<br>
part of the problem, or is it something you choose?  In the typical<br>
case, you won’t be using a threshold while <em>training</em> your model.</p>
<aside class="quote no-group">
<blockquote>
<p>Can I ask you how, since my classes are unbalanced, can I train my NN?</p>
</blockquote>
</aside>
<p>The term <em>unbalanced</em> generally means that you have many<br>
more examples of one class than another.  Two approaches<br>
are to sample the less common classes more frequently when<br>
you train, or to weight the less common classes more heavily<br>
in your loss function.  (See, for example, the <code>weight</code> argument<br>
passed to the constructor of <code>BCELoss</code>.)</p>
<p>But I’m not sure that this is what you mean by “unbalanced.”</p>
<aside class="quote no-group">
<blockquote>
<p>When I have a label equal to 0.9 for example - or any value greater than the threshold - should I round it to 1? Or should I differently weight somehow the 2 terms belonging to the BCELoss?</p>
</blockquote>
</aside>
<p>You talk about having a label of “0.9”.  Therefore I understand<br>
that your samples are not labeled with integer class labels (for<br>
which <code>0</code> would mean that the sample is in class-“0” and <code>1</code><br>
would mean that the class is in class-“1”), but rather, that your<br>
labels are continuous values between 0 and 1 and represent<br>
the probability that your sample is in class-“1”.</p>
<p>This is fine.  You want to train your model to predict the values<br>
of the probabilities given by your labels.  Cross-entropy is a<br>
measure of how dissimilar two probability distributions are, so<br>
<code>BCELoss</code> is fully appropriate for this use case.</p>
<aside class="quote no-group">
<blockquote>
<p>The fact is that if I have a label equal to 0.5, the BCE Loss will not decrease to zero by its definition</p>
</blockquote>
</aside>
<p>This is true, but it’s not a problem.  The pytorch optimizers don’t<br>
care whether your loss is zero – they just drive the loss (using<br>
some flavor of gradient descent) to lower values (and if you<br>
run them long enough, perhaps to its minimum value).</p>
<p>If your labelled probability is <code>0.5</code>, then <code>BCELoss</code> takes on its<br>
minimum value when your predicted probability is also <code>0.5</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>But also, I think that I should differentiate a label equal to 0.5 with respect to one equal to 0.9</p>
</blockquote>
</aside>
<p>You should.  And <code>BCELoss</code> does this.  If your label is <code>0.5</code>,<br>
<code>BCELoss</code> will prefer a predicted probability of <code>0.5</code>.  And if your<br>
label is <code>0.9</code>, <code>BCELoss</code> will prefer a predicted probability of <code>0.9</code>.</p>
<p>In summary – if I understand what you are doing – you are<br>
working with a binary classification problem, but with the<br>
(perfectly reasonable) feature that your labels are continuous<br>
probabilities between 0 and 1, rather than being integer class<br>
labels equal to either 0 or 1.</p>
<p><code>BCELoss</code> is just fine for this use case, and there’s no need to<br>
use any kind of threshold in your training.</p>
<p>(But to repeat my earlier comment, you’ll be better off using the<br>
mathematically-equivalent <code>BCEWithLogitsLoss</code>.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

MngFrc: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> !</p>
<p>Again, thank you for your time. I really appreciate it.<br>
I have understood that I made a bit of confusion, so sorry for that. I will try to summarize what my problem consist on.</p>
<p>I am looking for a NN that, given an input that represents the parameters of a system, outputs its error probability. The final aim is to look for the right parameter tuning through an iterative way: I start with an input, and if its error probability is above a threshold, I reduce the input until I find a a satisfactory error probability [below the threshold].</p>
<p>So my dataset consists on a set of measurements which couple system parameters and error probabilities.</p>
<p>That is why I have said that my labels can not really be seed as classes.</p>
<p>I have know 2 possible approaches (I guess) :</p>
<ol>
<li>I leave the error probabilities as they are and I use them as labels</li>
<li>I round the measured error probabilities in the dataset in such a way that, if they are above the threshold they are substituted with 1, with 0 in the other case.</li>
</ol>
<p>The only constraint that I have is to use the BCELoss() function.</p>
<aside class="quote no-group" data-username="KFrank" data-post="4" data-topic="77015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You talk about having a label of “0.9”. Therefore I understand<br>
that your samples are not labeled with integer class labels (for<br>
which <code>0</code> would mean that the sample is in class-“0” and <code>1</code><br>
would mean that the class is in class-“1”), but rather, that your<br>
labels are continuous values between 0 and 1 and represent<br>
the probability that your sample is in class-“1”.</p>
<p>This is fine. You want to train your model to predict the values<br>
of the probabilities given by your labels. Cross-entropy is a<br>
measure of how dissimilar two probability distributions are, so<br>
<code>BCELoss</code> is fully appropriate for this use case.</p>
</blockquote>
</aside>
<p>From this I guess that the first option should be the preferable one (Please correct me if I’m wrong).<br>
[I tried this approach and I have obtained quite large losses, but I do not exlude implementation problems. Anyway that was the reason I opened this topic. I thought that BCELoss() was fine only in the case I have {0,1} classes (and so labels)]</p>

KFrank: <p>Hello MngFrc!</p>
<aside class="quote no-group" data-username="MngFrc" data-post="5" data-topic="77015" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> MngFrc:</div>
<blockquote>
<p>I am looking for a NN that, given an input that represents the parameters of a system, outputs its error probability. The final aim is to look for the right parameter tuning through an iterative way: I start with an input, and if its error probability is above a threshold, I reduce the input until I find a a satisfactory error probability [below the threshold].</p>
<p>So my dataset consists on a set of measurements which couple system parameters and error probabilities.</p>
</blockquote>
</aside>
<p>It’s not at all clear to me what you are trying to do, so it’s hard<br>
to judge whether your approach makes sense.</p>
<p>It might be a good idea for you to start a new thread.</p>
<p>If you do, could you make concrete what data you’re starting<br>
with, and what the numbers mean?  What is the <code>shape</code> of the<br>
input to your network?  Roughly – in words – what is the<br>
architecture of your network?  Just a few <code>Linear</code> layers?<br>
Lots of convolutions and poolings and dropouts, etc.?</p>
<p>You say that you have “input that represents the parameters of a<br>
system,” and that  you want “to look for the right parameter tuning<br>
through an iterative way.”  Does this mean that you want to tune<br>
your <em>input</em> parameters, rather than the parameters (e.g., weights<br>
and biases) in neural network?</p>
<p>You say “I reduce the input until I find a a satisfactory error<br>
probability.”  Does that mean that you simply make the values<br>
of your input parameters smaller?  Or do you adjust your input<br>
parameters in some more complicated way?  Are you hoping to<br>
use a pytorch optimizer to adjust your <em>input</em> parameters for you?</p>
<p>(Again, these questions aren’t specifically about <code>BCELoss</code>, so<br>
it would make sense to start a new thread.)</p>
<p>Best.</p>
<p>K. Frank</p>

MngFrc: <p>Ok, I will do!</p>
<p>Being new to ML let me be not so clear.</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="77015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You say that you have “input that represents the parameters of a<br>
system,” and that you want “to look for the right parameter tuning<br>
through an iterative way.” Does this mean that you want to tune<br>
your <em>input</em> parameters, rather than the parameters (e.g., weights<br>
and biases) in neural network?</p>
</blockquote>
</aside>
<p>Anyway yes, I want to tune my imput parameters.</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="77015">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>ou say “I reduce the input until I find a a satisfactory error<br>
probability.” Does that mean that you simply make the values<br>
of your input parameters smaller? Or do you adjust your input<br>
parameters in some more complicated way? Are you hoping to<br>
use a pytorch optimizer to adjust your <em>input</em> parameters for you?</p>
</blockquote>
</aside>
<p>And yes, I simply make the values of my input parameters smaller, without any advanced technique.</p>
<p>Thank you againg, best wishes!</p>