Sigmoid Activation Function for regression task

Gianluca_Maselli: <p>Hi,</p>
<p>I have built a neural network aiming to predict 5 continuous values from video samples in the range between 0 and 1. For the last activation, I used the Sigmoid Activation function and as a criterion the MSE loss. Is it good for both the choices?</p>
<p>Thanks in advance for the help.</p>

KFrank: <p>Hi Gianluca!</p>
<aside class="quote no-group" data-username="Gianluca_Maselli" data-post="1" data-topic="159079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gianluca_maselli/48/46711_2.png" class="avatar"> Gianluca_Maselli:</div>
<blockquote>
<p>I used the Sigmoid Activation function and as a criterion the MSE loss. Is it good for both the choices?</p>
</blockquote>
</aside>
<p><code>MSELoss</code> is usually the right choice for regression.  I would recommend<br>
that you always start with <code>MSELoss</code> and only use something different if<br>
you have good reason and can show that it works better.</p>
<p>As for the <code>Sigmoid</code>, I would <em>not</em> use it, even though your target values<br>
are in the range <code>[0.0, 1.0]</code>.  It is true that <code>Sigmoid</code> maps the real line<br>
(that is <code>(-inf, inf)</code>) to <code>(0.0, 1.0)</code>, so it might seem natural, however,<br>
this is probably an illusion.</p>
<p>If your target (ground truth) values can be close to (or equal to) <code>0.0</code> and<br>
<code>1.0</code>, then the output of your network (before passing it through <code>Sigmoid</code>)<br>
would have to be a very large negative number (for a target close to <code>0.0</code>)<br>
or a very large positive number (for a target close to <code>1.0</code>), which would<br>
be hard for your network to learn.</p>
<p>You can experiment with <code>Sigmoid</code> if you want, but you should only<br>
actually use it if you can show that it works better than leaving it out.</p>
<p>(I could think of use cases where your would want the <code>Sigmoid</code>, but<br>
they would be contrived, or at least very atypical.)</p>
<p>Best.</p>
<p>K. Frank</p>

Gianluca_Maselli: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="159079">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>I could think of use cases where your would want the <code>Sigmoid</code>, but<br>
they would be contrived, or at least very atypical.</p>
</blockquote>
</aside>
<p>Thank you very much for the answer and for the clarification. I have just another question. When you say “…it works better than leaving it out”, do you mean to consider just the logits coming out from the last FC layer? Or it is better to substitute the sigmoid with another activation function as ReLU?</p>

KFrank: <p>Hi Gianluca!</p>
<aside class="quote no-group" data-username="Gianluca_Maselli" data-post="3" data-topic="159079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gianluca_maselli/48/46711_2.png" class="avatar"> Gianluca_Maselli:</div>
<blockquote>
<p>do you mean to consider just the logits coming out from the last FC layer?</p>
</blockquote>
</aside>
<p>Yes, I was speaking only about whether or not you should have <code>Sigmoid</code><br>
after your final <code>Linear</code> layer.  I recommend using the output of your final<br>
<code>Linear</code> layer as your predictions and feeding them directly to <code>MSELoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>Or it is better to substitute the sigmoid with another activation function as ReLU?</p>
</blockquote>
</aside>
<p>I was not talking about the non-linear activations between various layers.<br>
Having said that, some of the lore suggests that <code>ReLU</code> is to be preferred<br>
over <code>Sigmoid</code> (but <code>Sigmoid</code> is a perfectly reasonable non-linear activation).</p>
<p>Best.</p>
<p>K. Frank</p>

Gianluca_Maselli: <p>Thank you Frank your suggestions were of huge help to me.<br>
I’ll remove the sigmoid in the last layer on the final layer then.</p>
<p>Kind regards.<br>
G.Maselli.</p>