[HELP] output layer with softmax in pytorch

username: wfcrgt
cooked: <p>this is my network</p>
<pre><code class="lang-python">class ResNetWithBottleneck(nn.Module):
    def __init__(self, dim_bottleneck_layer, num_class):
        super(ResNetWithBottleneck, self).__init__()
        # base pretrained resnet
        self.base_resnet = ResNetRvLinear()
        self.base_resnet = make_pretrained_resnet50(self.base_resnet)
        # add fc_bottleneck layer
        self.dim_bottleneck_layer = dim_bottleneck_layer
        self.fc_bottleneck = nn.Linear(2048, dim_bottleneck_layer)

        self.fc_class = nn.Linear(dim_bottleneck_layer, num_class)
        self.softmax = nn.Softmax(dim=1)

        # init weight
        self.fc_bottleneck.apply(init_weights)
        self.fc_class.apply(init_weights)

    def forward(self, x):
        x = self.base_resnet(x)
        x = self.fc_bottleneck(x)
        y = self.fc_class(x)
        y = self.softmax(y)
        return x, y
</code></pre>
<p>when I remove function <code>forward</code> last <code>y = self.softmax(y)</code>, the code runining well.</p>
<p>when I add the <code>y = self.softmax(y)</code>, the loss not descent.</p>
<p>and I don’t know what’s the problem with me.</p>
<p>btw. I want know why torchvision’s net both not use softmax in the last output layer.</p>
<p>thanks!</p>
<hr>
<p>addtion:</p>
<p>I see some information and get Loss function will calculate softmax, when you add a addtion softmax will lower loss descent(but loss will still descent).</p>
<p>I review some pytorch code.</p>
<p>in <a href="https://github.com/yunjey" rel="nofollow noopener">yunjey</a>/ <strong><a href="https://github.com/yunjey/pytorch-tutorial" rel="nofollow noopener">pytorch-tutorial</a></strong>’s <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/generative_adversarial_network/main.py" rel="nofollow noopener">GAN code</a></p>
<pre><code class="lang-python">D = nn.Sequential(
    nn.Linear(image_size, hidden_size),
    nn.LeakyReLU(0.2),
    nn.Linear(hidden_size, hidden_size),
    nn.LeakyReLU(0.2),
    nn.Linear(hidden_size, 1),
    nn.Sigmoid())

criterion = nn.BCELoss()
</code></pre>
<p>why when predict a binary class should add a sigmoid funciton?</p>
<p>thanks!</p>
<p>in line 48 when predict</p>

username: Xiaoyu_Song
cooked: <p>Hi, for the first question, which loss function do you use?<br>
For the second question:<br>
For binary classification, the logistic function (a sigmoid) and softmax will perform equally well, but the logistic function is mathematically simpler and hence the natural choice. When you have more than two classes, you can’t use a scalar function like the logistic function as you need more than one output to know the probabilities for all the classes, hence you use softmax.</p>

username: wfcrgt
cooked: <p>thanks <a class="mention" href="/u/xiaoyu_song">@Xiaoyu_Song</a> !</p>
<p>in my first question, I do a multi-class prediction, and I use a softmax function as my output layer, and the loss descent big slower(like not descent) than not use softmax, my loss function use <code>CrossEntropyLoss</code>.</p>
<p>and then I figure out that:<br>
in <code>nn.BCELoss</code>, latest layer need a sigmoid function.<br>
in <code>nn.BCEWithLogitsLoss</code>, not need a sigmoid funtion in latest layer.<br>
in <code>nn.CrossEntropyLoss</code>, not need a <code>nn.Softmax(dim=1)</code> in latest layer, because the loss funtion already include softmax function.</p>

username: Joker
cooked: <p>Today I’m  doing  the  CNN multi-class prediction,  and  I  wan  to   output   the  probability  about   every  class,   but     in   pytorch ,  the   nn.CrossEntropyLoss   contains   a   log_softmax(),and  the  nn.NLLLoss<br>
function  also  need  log_softmax()  in  the last  layer  ,so  maybe  there is  no  loss  funtion  for  softmax.<br>
But   I  can train  the   model  as  usual   with   using<br>
nn.CrossEntropyLoss<br>
and  the last  layer  is  just  a      nn.Linear()  layer,<br>
At  last ,when  I  want  to  get  the    softmax    probability,    I  can  use  like  this  :<br>
out_put=model(intput)<br>
probability= torch.nn.functional.softmax(out_put,dim=1)<br>
Now the  probability  is   same  as  what  you  get  in  tensorflow  or keras   softmax</p>

username: Qu_Yukun
cooked: <p>Hi, I have a related question. I try to train a two-stream network, so I need a fusion of two softmax layers before giving to loss function. But the nn.CrossEntropyLoss already include softmax function. What should I do to prevent the nn.CrossEntropyLoss from automatically using the softmax function?</p>

username: chivalryq
cooked: <p>hi,I also have a related question,I want to apply my model on mobilephone,so I have to convert it to another frame(like NCNN or MNN) . I want softmax to be the last layer. I’ll try train the model without softmax and add softmax layer when saving it. After all,there is no weights in softmax layer.</p>