How to exploit BCELoss() criterion with no classes

MngFrc: <p>Hi everyone!</p>
<p>I’m trying to deploy a linear feedforward NN that, given an input that represents the parameters of a system, outputs its error probability. The final aim is to look for the right input parameter tuning through an iterative way: I start with an input, and if the corresponding error probability is above a threshold, I reduce one of the input parameters until I find a satisfactory error probability [below the threshold].</p>
<p>So, my dataset consists on a set of measurements which couple system parameters (which are simply real numbers) and error probabilities. The NN should learn the relationship between the system setting and its error probability. So, my first idea is to use the measured error probabilities as labels (obviously they are real numbers in the set <em>{0,1}</em>)</p>
<p>The structure of the NN is given: the input layer has the same dimensionality of the system parameters obviously, then there are a set of hidden layers with <code>ReLu() activation function</code> and finally there is one single-neuron output layer activated by a <code>sigmoid function</code>.</p>
<p>The criterion that has to be used is the <code>BCELoss()</code>.</p>
<p>[ I know that it is suggested to use the <code>nn.BCEWithLogitsLoss</code> without <code>sigmoid activation</code> function, but the structure should be given, so I would prefer to not change it].</p>
<p>My problem is that I am not able to well understand why and how the <code>BCELoss()</code> criterion is well suited for this kind of problem.</p>
<p>I knew that it is was suggested when there is a classification problem and the number of classes is 2: class 0 and class 1. In such a scenario, the NN learns how to classify its input, outputting the probability that the input belongs to class 1.</p>
<p>But in my case, how should I train the network?</p>
<p>I think to have two possible approaches:</p>
<ul>
<li>I forget about the concept of class and I train the NN supervising the error probability.</li>
<li>Since it is defined a threshold, which differentiate between acceptable and not acceptable error probabilities, I can use the same threshold to modify my labels in 0 and 1.</li>
</ul>
<p>The problem of the first approach is that I think that the Loss can not converge to small numbers Indeed, just try to put in the BCE formula <em>y=0.5</em> for example (where y is the label, the groundtruth).  I have tried to train the NN and it reaches 0.4 of losses, even if the pattern of its curve is really good.</p>
<p>The problem of the second approach is that I don’t want my NN to tell if the error probability corresponding to its input is above or below a threshold, but I want to know how much the error probability is.</p>
<p>These last two points are my considerations, but they could be definitely wrong. That’s why I’m asking a more reliable opinion.</p>

KFrank: <p>Hi MngFrc!</p>
<p>Various comments, in line, below:</p>
<aside class="quote no-group" data-username="MngFrc" data-post="1" data-topic="77284" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ea666f/48.png" class="avatar"> MngFrc:</div>
<blockquote>
<p>I’m trying to deploy a linear feedforward NN that, given an input that represents the parameters of a system, outputs its error probability.</p>
</blockquote>
</aside>
<p>Let me say what I <em>think</em> you’re trying to do, but please correct me<br>
where I’m wrong.</p>
<p>But first, let me comment on some terminology to avoid confusion.</p>
<p>We often call the weights and biases in the layers of a neural network<br>
(as well as other adjustable values) <em>parameters.</em>  And we often speak<br>
of training those parameters.</p>
<p>But you have a “system” that has “parameters” (that I assume are<br>
not part of any neural network).  Let me call these your <em>system<br>
parameters</em>,  or, sometimes, just parameters.  And to avoid confusion,<br>
let me call the parameters of the neural network the <em>model weights,</em><br>
or, sometimes, just the weights.</p>
<aside class="quote no-group">
<blockquote>
<p>The final aim is to look for the right input parameter tuning through an iterative way:</p>
</blockquote>
</aside>
<p>When we train model weights we use an iterative process – forward<br>
pass, backpropagation, optimizer step.  I assume that the iterative<br>
process you wish to use to tune your system parameters is separate<br>
from the pytorch optimization system used to train the weights.  Is<br>
this correct, or are you intending to use pytorch optimization to tune<br>
your system parameters?</p>
<aside class="quote no-group">
<blockquote>
<p>I start with an input, and if the corresponding error probability is above a threshold, I reduce one of the input parameters until I find a satisfactory error probability [below the threshold].</p>
</blockquote>
</aside>
<p>I am <em>guessing</em> that you assume that you have trained a neural<br>
network that, given a set of system parameters, gives you the<br>
system’s “error probability.”  Then you use the neural network<br>
as a black box, having it give you the the error probabilities as<br>
you reduce your parameters using your tuning scheme.</p>
<p>Is this correct?</p>
<aside class="quote no-group">
<blockquote>
<p>So, my dataset consists on a set of measurements which couple system parameters (which are simply real numbers) and error probabilities.</p>
</blockquote>
</aside>
<p>I assume that your dataset comes from actual measurements on an<br>
actual system – or perhaps from a (non-neural-network) model of<br>
your system – where you measure the error probabilities for a sets of<br>
system parameters.  In particular, you have this dataset independently<br>
of – and before you start to build – your neural network.</p>
<p>Is this correct?</p>
<aside class="quote no-group">
<blockquote>
<p>The NN should learn the relationship between the system setting and its error probability.</p>
</blockquote>
</aside>
<p>So, a set of floating-point numbers in (the system parameters), and a<br>
floating-point number out (the error probability).</p>
<p>And I assume that the error probability is sensibly continuous.  That is,<br>
that, say, 28.6% is “close to” 28.5% and 28.7% in a meaningful way,<br>
but is further away from 35.4%.</p>
<p>It sounds like you should try using <code>MSELoss</code> (mean-squared-error).</p>
<aside class="quote no-group">
<blockquote>
<p>So, my first idea is to use the measured error probabilities as labels (obviously they are real numbers in the set <em>{0,1}</em>)</p>
</blockquote>
</aside>
<p>Using the error probabilities as <em>labels</em> (in the typical neural-network<br>
sense) likely doesn’t make sense.  Usually, one label (say, “cat”) is not<br>
closer to nor further away from any other label (say “fish” or “bird”).</p>
<p>Trying to turn meaningfully continuous floating-point number into labels<br>
is artificial, and is likely to throw away some of the useful information in<br>
your dataset.</p>
<aside class="quote no-group">
<blockquote>
<p>The structure of the NN is given: the input layer has the same dimensionality of the system parameters obviously,</p>
</blockquote>
</aside>
<p>This makes sense.</p>
<aside class="quote no-group">
<blockquote>
<p>then there are a set of hidden layers with <code>ReLu() activation function</code></p>
</blockquote>
</aside>
<p>This sounds reasonable.</p>
<aside class="quote no-group">
<blockquote>
<p>and finally there is one single-neuron output layer</p>
</blockquote>
</aside>
<p>This makes sense.</p>
<aside class="quote no-group">
<blockquote>
<p>activated by a <code>sigmoid function</code>.</p>
</blockquote>
</aside>
<p>This sounds like it might be suboptimal to me.  It’s true that you know<br>
your output should be between zero and one, so passing it through a<br>
<code>sigmoid()</code> will force it to be between zero and one.  But given what<br>
I think your error probability means, my intuition tells me that it might<br>
be better (in this case) for the network to learn that values outside of<br>
<code>(0, 1)</code> aren’t good predictions, penalizing such values – but neither<br>
preventing nor correcting them – when they are output by the network.</p>
<aside class="quote no-group">
<blockquote>
<p>The criterion that has to be used is the <code>BCELoss()</code>.</p>
</blockquote>
</aside>
<p>Why?  “Has to be” is a problematic phrase when you are asking for<br>
suggestions about how to solve a problem.</p>
<aside class="quote no-group">
<blockquote>
<p>[ I know that it is suggested to use the <code>nn.BCEWithLogitsLoss</code> without <code>sigmoid activation</code> function, but the structure should be given, so I would prefer to not change it].</p>
</blockquote>
</aside>
<p>Again, why?  If it were me, I would <em>prefer</em> to change the structure, if<br>
doing so would better solve my problem.</p>
<p>Also, to be clear, based on my current understanding of what you<br>
are trying to do as outlined in my comments above, I am currently<br>
suggesting that you use <code>MSELoss</code> without (or perhaps with) a final<br>
<code>sigmoid()</code> activation.</p>
<aside class="quote no-group">
<blockquote>
<p>My problem is that I am not able to well understand why and how the <code>BCELoss()</code> criterion is well suited for this kind of problem.</p>
</blockquote>
</aside>
<p>Well, if your problem is indeed to predict a continuous output variable,<br>
then I think you’re right the <code>BCELoss</code> is not well-suited to your problem.</p>
<aside class="quote no-group">
<blockquote>
<p>I knew that it is was suggested when there is a classification problem and the number of classes is 2: class 0 and class 1. In such a scenario, the NN learns how to classify its input, outputting the probability that the input belongs to class 1.</p>
</blockquote>
</aside>
<p>To repeat what I said above, based on the assumptions I outlined,<br>
I no longer think that this is a classification problem.</p>
<aside class="quote no-group">
<blockquote>
<p>But in my case, how should I train the network?</p>
</blockquote>
</aside>
<p>I would train as outlined above, probably using <code>MSELoss</code> as the loss<br>
function.</p>
<aside class="quote no-group">
<blockquote>
<p>I think to have two possible approaches:</p>
<ul>
<li>I forget about the concept of class and I train the NN supervising the error probability.</li>
</ul>
</blockquote>
</aside>
<p>Yes, if by “supervising the error probability” you mean using a loss<br>
function to quantify the discrepancy between the measured error<br>
probability from your dataset with the network’s predicted error<br>
probability, and then using backpropagation of the loss function to<br>
train your network weights.</p>
<aside class="quote no-group">
<blockquote>
<ul>
<li>Since it is defined a threshold, which differentiate between acceptable and not acceptable error probabilities, I can use the same threshold to modify my labels in 0 and 1.</li>
</ul>
</blockquote>
</aside>
<p>This sounds like you would be throwing away useful information in<br>
your data set, and therefore would be making it harder to train your<br>
network.</p>
<aside class="quote no-group">
<blockquote>
<p>The problem of the first approach is that I think that the Loss can not converge to small numbers Indeed, just try to put in the BCE formula <em>y=0.5</em> for example (where y is the label, the groundtruth).  I have tried to train the NN and it reaches 0.4 of losses, even if the pattern of its curve is really good.</p>
</blockquote>
</aside>
<p>As I said above, I don’t think that <code>BCELoss</code> is the right loss.</p>
<p>But, as an aside, the optimizer <em>doesn’t</em> try to lower the loss to zero.<br>
It tries to lower the loss to its minimum value, and it doesn’t care<br>
whether that minimum value is 1,000,000 or 0, or -1,000,000.  In<br>
the case of <code>BCELoss</code>, if your target value is <code>0.5</code>, the minimum loss<br>
will <em>not</em> be zero, but it <em>will</em> occur for a predicted value of <code>0.5</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>The problem of the second approach is that I don’t want my NN to tell if the error probability corresponding to its input is above or below a threshold, but I want to know how much the error probability is.</p>
</blockquote>
</aside>
<p>I agree with this.  Assuming I understand your problem correctly, you<br>
want to train your network to predict the actual continuous floating-point<br>
error probability (not just whether is is above or below some threshold),<br>
based on the known, measured values in your dataset.</p>
<p>Good luck.</p>
<p>K. Frank</p>