Why my custom CrossEntropyLoss got different loss with the nn.CrossEntropy() from the middle of the training?

NA_GONG: <p>Hi,<br>
I wrote a custom def CrossEntropy() to remove the softmax in the torch.nn.CrossEntropy():</p>
<pre><code class="lang-auto">def CrossEntropy(self, output, target):
        '''
        input: softmaxted output
        '''
        loss = nn.NLLLoss().to(self.device)
        return loss(torch.log(output), target)
</code></pre>
<p>As I tested with a complete model training, it can successfully reproduce the result of the torch.nn.CrossEntropy():<br>
(old class_loss: calculated by torch.nn.CrossEntropy(); new class_loss: calculated by custom def CrossEntropy():</p>
<pre><code class="lang-auto">class_loss = self.class_criterion(model_outputs, labels) # self.class_criterion = torch.nn.CrossEntropy()
print('old class_loss: ', self.class_criterion(model_outputs, labels))
print('new class_loss: ', self.CrossEntropy(self.softmax(model_outputs), labels))
</code></pre>
<pre><code class="lang-auto">old class_loss:  tensor(0.8386, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
new class_loss:  tensor(0.8386, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
old class_loss:  tensor(0.6758, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
new class_loss:  tensor(0.6758, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
old class_loss:  tensor(0.5868, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
new class_loss:  tensor(0.5868, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
old class_loss:  tensor(0.5376, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
new class_loss:  tensor(0.5376, device='cuda:0', grad_fn=&lt;NllLossBackward&gt;)
       ....
</code></pre>
<p>Problem: When I use my own def CrossEntropy() to run the same training procedure, both class_losses are still the same, but the general loss results of my own CrossEntropy(right cloumn below) became different from the training with torch.nn.CrossEntropy() (left column below) from the 5th training steps:</p>
<pre><code class="lang-auto">class_loss = self.CrossEntropy(self.softmax(model_outputs), labels)
print('old class_loss: ', self.class_criterion(model_outputs, labels))
print('new class_loss: ', self.CrossEntropy(self.softmax(model_outputs), labels))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9bc685b09db3bc53625d9c31bdb607115f3fc08.png" data-download-href="https://discuss.pytorch.org/uploads/default/b9bc685b09db3bc53625d9c31bdb607115f3fc08" title="2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9bc685b09db3bc53625d9c31bdb607115f3fc08.png" alt="2" data-base62-sha1="qv5ZB9Z9pGlqE71P66DFYnEeqJa" width="690" height="293" data-dominant-color="E0DEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">1396×593 18.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can anyone please let me know why this happens and how could I solve it so that my own def CrossEntropy() could completely reproduce the training results? Thanks a lot!</p>

ptrblck: <p><code>torch.log(torch.softmax(tensor))</code> is less numerically stable than <code>F.log_softmax</code>, which is used internally in <code>nn.CrossEntropyLoss</code>. The numerical stability is increased by applying the <a href="https://en.wikipedia.org/wiki/LogSumExp">log-sum-exp trick</a>. You should get the same values, if you use <code>F.log_softmax</code> instead of the separate function calls.</p>

NA_GONG: <p>Thanks, it works good.</p>