Add sigmoid layer

Chamroukhi: <p>Hello,<br>
im using an model pretrained i need to add classifier layer but i don’t understand how:<br>
1- it’s Linear layer apply softmax automatically ?<br>
2- can i use Linear layer and after this layer add a Softmax layer ?:</p>
<pre><code class="lang-auto">model = models.video.mc3_18(pretrained=True, progress=True)
        set_parameter_requires_grad(model, feature_extract)
        #change output layer FC
        model.fc = nn.Sequential(nn.Linear(512, 256),
                                 nn.ReLU(),
                                 nn.Linear(256,num_classes),
                                nn.Softmax())
</code></pre>
<p>3 - it’s ok to use crossentropy loss ?<br>
thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Manuel_Alejandro_Dia: <blockquote>
<ol>
<li>Its Linear layer applies softmax automatically ?</li>
</ol>
</blockquote>
<p>No, a Linear layer just applies the weights and biases for the inputs.</p>
<blockquote>
<ol start="2">
<li>Can i use Linear layer and after this layer add a Softmax layer ?</li>
</ol>
</blockquote>
<p>It is best that you don’t, because it may give you a problem when calculating the loss. You could set a condition in your model so it does the softmax operation if it is not training:</p>
<pre data-code-wrap="python"><code class="lang-python">if model.training:
    return output
else:
    return torch.nn.functional.softmax(output)
</code></pre>
<blockquote>
<ol start="3">
<li>Is it ok to use crossentropy loss ?</li>
</ol>
</blockquote>
<p>In order to train, you will need to give the raw output from <code>nn.Linear(256, num_classes)</code> to the Cross Entropy loss object, since the loss implementation applies <code>LogSoftmax</code> to calculate the loss: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc"><code>nn.CrossEntropyLoss</code></a></p>

Chamroukhi: <p>Thank you <a class="mention" href="/u/manuel_alejandro_dia">@Manuel_Alejandro_Dia</a>,<br>
it is possible to use this head ? :</p>
<pre><code class="lang-auto">model.fc = nn.Sequential(nn.Linear(512, 256),
                                 nn.Softmax())</code></pre>

Manuel_Alejandro_Dia: <p>Glad I could help! <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=10" title=":grin:" class="emoji" alt=":grin:"></p>
<p>You cannot use that head since it has a Softmax layer.</p>
<p>I would recommend something like:</p>
<pre><code class="lang-auto">model.fc = nn.Sequential(nn.Linear(512, 256),
                         nn.ReLU(),
                         nn.Linear(256,num_classes),
                         nn.ReLU()
                         )
</code></pre>
<p>Rmember that you don’t have to add the softmax operation in your <code>nn.Sequential</code> since it could interfere with the softmax done on <code>CrossEntropyLoss</code></p>

Chamroukhi: <p>Relu() generate probability?, can i use max of probability like predicted class ?</p>

Manuel_Alejandro_Dia: <p>What ReLU does is filter the negative values. What you can do if you just want to get the predicted class is to apply output.argmax(1) (or max, as you said)</p>