Is it OK to use sigmoid in the output layer and using PyTorch cross entropy loss?

username: Mary_H
cooked: <p>Let’s consider I have a multi layer neural network that is doing multi label image segmentation. So each pixel belongs to one of N classes.</p>
<p>Now, let’s say the last layer has Sigmoid activation units. The used loss then is PyTorch cross entropy which includes already softmax inside it. My question that I am getting better performance when I added the sigmoid in the last layer compare to removing it from the last layer.<br>
I am not sure if it is OK to use sigmoid and if it makes sence in this situation,  any idea why I may get better results with using Sigmoid.</p>

username: KFrank
cooked: <p>Hi Mary!</p>
<aside class="quote no-group" data-username="Mary_H" data-post="1" data-topic="194367" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mary_h/48/66407_2.png" class="avatar"> Mary_H:</div>
<blockquote>
<p>I am getting better performance when I added the sigmoid in the last layer compare to removing it from the last layer.</p>
</blockquote>
</aside>
<p>Let me answer this carefully:</p>
<p>I can’t rule out the possibility that you have an unusual, perverse use<br>
case for which adding <code>Sigmoid</code> as the final layer improves performance.<br>
If your model trains well and you are also getting good results on your<br>
validation and / or test datasets, then use whatever works best for your<br>
use case.</p>
<p>Having said that, including <code>Sigmoid</code> as your final layer (when using<br>
pytorch’s <code>CrossEntropyLoss</code> as your loss criterion) is almost certainly<br>
a mistake.  Doing so won’t fully “break” your model, but it limits how<br>
well your model can perform.</p>
<p>As you note, <code>CrossEntropyLoss</code> includes <code>softmax()</code> (actually<br>
<code>log_softmax()</code>) internally.  But when you add the <code>Sigmoid</code> your model<br>
now only outputs values between zero and one and, when converted<br>
to probabilities by <code>CrossEntropyLoss</code>’s internal <code>softmax()</code>, the<br>
probabilities seen internally by <code>CrossEntropyLoss</code> will no longer be<br>
able to span their full range from zero to one.  So, in particular, your<br>
model can no longer predict your correct class with 100% probability.</p>
<p>Consider this two-class illustration:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.1.1'
&gt;&gt;&gt; t = torch.arange (1., 11).unsqueeze (-1).repeat (1, 2)
&gt;&gt;&gt; t[:, 0] *= -1
&gt;&gt;&gt; t
tensor([[ -1.,   1.],
        [ -2.,   2.],
        [ -3.,   3.],
        [ -4.,   4.],
        [ -5.,   5.],
        [ -6.,   6.],
        [ -7.,   7.],
        [ -8.,   8.],
        [ -9.,   9.],
        [-10.,  10.]])
&gt;&gt;&gt; t.softmax (1)
tensor([[1.1920e-01, 8.8080e-01],
        [1.7986e-02, 9.8201e-01],
        [2.4726e-03, 9.9753e-01],
        [3.3535e-04, 9.9966e-01],
        [4.5398e-05, 9.9995e-01],
        [6.1442e-06, 9.9999e-01],
        [8.3153e-07, 1.0000e+00],
        [1.1254e-07, 1.0000e+00],
        [1.5230e-08, 1.0000e+00],
        [2.0612e-09, 1.0000e+00]])
&gt;&gt;&gt; t.sigmoid().softmax (1)
tensor([[0.3865, 0.6135],
        [0.3183, 0.6817],
        [0.2880, 0.7120],
        [0.2761, 0.7239],
        [0.2716, 0.7284],
        [0.2699, 0.7301],
        [0.2693, 0.7307],
        [0.2691, 0.7309],
        [0.2690, 0.7310],
        [0.2690, 0.7310]])
</code></pre>
<p>When you pass the output of your model through <code>Sigmoid</code>, you<br>
convert the output of your model (which should be understood as<br>
unnormalized log-probabilities) to things that look like probabilities.<br>
As you can see, it is easy to predict some class has having 100%<br>
probability if you don’t have the <code>Sigmoid</code>.  But when you insert the<br>
<code>Sigmoid</code>, the best you can do (in the two-class case) is to predict<br>
your favored class as having about 73% probability.  With the <code>Sigmoid</code><br>
your model can never learn to predict the correct class with near-100%<br>
certainty.</p>
<p>So it is very unlikely that your added <code>Sigmoid</code> improves your model’s<br>
best performance.</p>
<p>Check your model and training code for bugs.  Check your results on<br>
a validation dataset as well as on the dataset you train with.  Also try<br>
training significantly longer because if you don’t train long enough, you<br>
won’t be able to see the <code>Sigmoid</code>-version’s performance top out.</p>
<p>Your goal, of course, is not to overfit, but to verify the point I’m making,<br>
try overfitting by training a whole lot on a smallish subset of your training<br>
data.  You should see that with the <code>Sigmoid</code> you <em>can’t</em> overfit your model<br>
to the point that it predicts your ground-truth target class with 100%<br>
probability.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Mary_H
cooked: <p>I will remove it, since what you said makes sense, I am working on unsupervised task so I don’t have train, validation tests.</p>