Shouldn't F.log_softmax (x, dim = 0) be used?

username: shirui-japina
cooked: <p>Here I have <strong>three questions</strong>.</p>
<p>By chance I saw the code <a href="https://techtech-sorae.com/pytorch%E3%82%92%E7%94%A8%E3%81%84%E3%81%A6%E3%83%87%E3%82%A3%E3%83%BC%E3%83%97%E3%83%A9%E3%83%BC%E3%83%8B%E3%83%B3%E3%82%B0%E3%81%AB%E3%82%88%E3%82%8B%E3%83%AF%E3%82%A4%E3%83%B3%E5%88%86%E9%A1%9E/" rel="noopener nofollow ugc">here</a>.<br>
What puzzles me is the <code>class Net(nn.Module)</code> and the <strong>loss function</strong>:</p>
<pre><code class="lang-auto">    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(13, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 128)
        self.fc5 = nn.Linear(128, 128)
        self.fc6 = nn.Linear(128, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x)) # ReLU: max(x, 0)
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        x = self.fc6(x)
        return F.log_softmax(x, dim=0)
</code></pre>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
output = model(train_x)
loss = criterion(output, train_y)
loss.backward()
optimizer.step()
</code></pre>
<p>My <strong>first question</strong> is, <code>F.log_softmax (x, dim = 0)</code> shouldn’t be used here, is my understanding right?</p>
<p>As discussed in <a href="https://discuss.pytorch.org/t/using-nn-crossentropyloss-how-can-i-get-softmax-output/52925" class="inline-onebox">Using nn.CrossEntropyLoss(), how can I get softmax output?</a>,</p>
<blockquote>
<p>nn.CrossEntropyLoss() automatically apply logSoftmax using FC layer output.</p>
</blockquote>
<p>The model class should be:</p>
<pre><code class="lang-auto">    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(13, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 128)
        self.fc5 = nn.Linear(128, 128)
        self.fc6 = nn.Linear(128, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x)) # ReLU: max(x, 0)
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        x = self.fc6(x)
        return x
</code></pre>
<p>My <strong>second question</strong> is, F.log_softmax() is used for what?</p>
<p>My <strong>third question</strong> is, whether <code>F.log_softmax()</code> is used or not, the model performance is about 90%. Why is this happening?<br>
Finally, I <strong>should</strong> use <code>F.log_softmax()</code> or I <strong>should NOT</strong> use <code>F.log_softmax()</code>?</p>

username: KFrank
cooked: <p>Hi Shirui!</p>
<aside class="quote no-group" data-username="shirui-japina" data-post="1" data-topic="73235" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shirui-japina/48/15285_2.png" class="avatar"> shirui-japina:</div>
<blockquote>
<p>By chance I saw the code <a href="https://techtech-sorae.com/pytorch%E3%82%92%E7%94%A8%E3%81%84%E3%81%A6%E3%83%87%E3%82%A3%E3%83%BC%E3%83%97%E3%83%A9%E3%83%BC%E3%83%8B%E3%83%B3%E3%82%B0%E3%81%AB%E3%82%88%E3%82%8B%E3%83%AF%E3%82%A4%E3%83%B3%E5%88%86%E9%A1%9E/" rel="noopener nofollow ugc">here</a>.</p>
</blockquote>
</aside>
<p>Well, yes, as you have recognized, this code is wrong.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">...
        return F.log_softmax(x, dim=0)
...
criterion = nn.CrossEntropyLoss()
</code></pre>
<p>My <strong>first question</strong> is, <code>F.log_softmax (x, dim = 0)</code> shouldn’t be used here, is my understanding right?</p>
</blockquote>
</aside>
<p>That is correct.  When using <code>CrossEntropyLoss</code> you should not use<br>
<code>log_softmax()</code> for the output of your model.  (You would, if you were<br>
using <code>NLLLoss</code>.)  You would typically pass the output of the last linear<br>
layer of your model into <code>CrossEntropyLoss</code> (as you indicated in the<br>
code you posted).</p>
<p>One more error in the code you linked to:</p>
<p>Even if it were appropriate to use <code>log_softmax()</code> (for example, with<br>
<code>NLLLoss</code>), <code>log_softmax (x, dim = 0)</code> is wrong.  <code>x</code> here has shape<br>
<code>(nBatch, nClass)</code>, so <code>log_softmax (x, dim = 0)</code> would perform<br>
the softmax operation across the batch dimension.  You would need<br>
instead <code>log_softmax (x, dim = 1)</code> in order to perform softmax<br>
across the class dimension.</p>
<p>(Unfortunately, there is a lot of misinformation on the internet.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: shirui-japina
cooked: <p>Got it, thanks for your help.<img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>