How to design a NN that targets probabilities?

Chenoille: <p>Hello,<br>
I want to train a neural network on some image dataset for a regression task. The output would be a probability, so a value between 0 and 1; I have targets that are also probabilities. I want to know what the best training strategy would be.<br>
Is is meaningful to set the last layer to be sigmoid such that the previous layers can output values from -inf to inf? Or should I simply directly output a value between 0 and 1, that we can later calculate the loss using MSE or MAE. But it would make no sense that the network outputs a value that is more than one/less than 0.<br>
Also, Should I scale the sigmoid? Because points close to 0 will be very sensitive to noise. If yes, should it be a hyperparameter?</p>
<p>I thank you in advance for your help.</p>

KFrank: <p>Hi Chenoille!</p>
<aside class="quote no-group" data-username="Chenoille" data-post="1" data-topic="167698" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenoille/48/45737_2.png" class="avatar"> Chenoille:</div>
<blockquote>
<p>I want to train a neural network on some image dataset for a regression task.</p>
</blockquote>
</aside>
<p>We would typically not use the term “regression task” when training a<br>
network to predict probabilities.</p>
<p>If you believe that your use case is a classic regression task, please<br>
explain it in a little more detail.</p>
<aside class="quote no-group">
<blockquote>
<p>The output would be a probability, so a value between 0 and 1; I have targets that are also probabilities.</p>
</blockquote>
</aside>
<p>Based on what I think your use case is, you should use <code>BCEWithLogitsLoss</code><br>
as your loss criterion.</p>
<p>Note that <code>BCEWithLogitsLoss</code> takes <em>logits</em> as its <code>input</code>, rather than actual<br>
probabilities.  It does take probabilities as its <code>target</code>.</p>
<p>Your network should be whatever architecture is appropriate for your overall<br>
use case, and will depend on the character of the problem you are trying to<br>
solve and the structure of your input data.</p>
<p>However, you would typically want your final layer to be a full-connected<br>
<code>Linear</code> with a single output (<code>out_features = 1</code>).  This will be your logit,<br>
and will range from <code>-inf</code> to <code>inf</code>.  Do <em>not</em> pass this output through a<br>
<code>sigmoid()</code> – doing so would convert your logit to a probability, which<br>
is not what <code>BCEWithLogitsLoss</code> expects.</p>
<aside class="quote no-group">
<blockquote>
<p>Should I scale the sigmoid? Because points close to 0 will be very sensitive to noise.</p>
</blockquote>
</aside>
<p>In some sense, this is what using logits and <code>BCEWithLogitsLoss</code> is doing<br>
(in contrast to using probabilities and pytorch’s plain-vanilla <code>BCELoss</code>).</p>
<p>I’m not sure that I would call it noise exactly, but using <code>BCEWithLogitsLoss</code><br>
with logits is significantly more numerically stable than using <code>BCELoss</code> with<br>
probabilities.</p>
<p>Best.</p>
<p>K. Frank</p>

Chenoille: <p>Hello,<br>
Thank you for this answer. That was exactly what I was looking for.</p>