Is log_softmax + NLLLoss == CrossEntropyLoss?

username: Vanjoy
cooked: <p>If I’m not missing something, they should be the same. However, I tried the follow snippet, but they are not equal.</p>
<pre><code class="lang-python">#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from torch.autograd import Variable


class Net(nn.Module):

    def __init__(self, n_features, n_hiddens, n_classes):
        super(Net, self).__init__()
        self.gru = torch.nn.GRU(n_features, n_hiddens)
        self.linear = torch.nn.Linear(n_hiddens, n_classes)

    def forward(self, x, flag=True):
        o, h = self.gru(x)
        o = self.linear(o)

        if flag:
            o = F.log_softmax(o)

        return o


n_steps = 10
n_classes = 100
mb_size = 32
n_features = 50
n_hiddens = 60

net = Net(n_features, n_hiddens, n_classes)

loss1 = torch.nn.NLLLoss(size_average=False)
loss2 = torch.nn.CrossEntropyLoss(size_average=False)

x = Variable(torch.rand(n_steps, mb_size, n_features))
y = Variable(
    torch.LongTensor(np.random.randint(0, n_classes, (n_steps, mb_size))))

logits1 = net(x, flag=True).view(-1, n_classes)
logits2 = net(x, flag=False).view(-1, n_classes)

loss_val1 = loss1(logits1, y.view(-1))
loss_val2 = loss2(logits2, y.view(-1))

print(loss_val1)
print(loss_val2)
</code></pre>

username: richard
cooked: <p>They are the same (see <a href="http://pytorch.org/docs/master/_modules/torch/nn/functional.html#cross_entropy" rel="nofollow noopener">the implementation</a>). I think the reason why it isn’t working out for you because log_softmax gives different results depending on shape. The shape of x when passed into log_softmax in forward is different from the shape of logit2.</p>

username: Vanjoy
cooked: <p>Thank you so much. I can’t believe that iPython gives me</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1435838804bd5a4e2f176bd5a8bdf4a18d6fed7d.png" data-download-href="https://discuss.pytorch.org/uploads/default/1435838804bd5a4e2f176bd5a8bdf4a18d6fed7d" title="02 AM"><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1435838804bd5a4e2f176bd5a8bdf4a18d6fed7d.png" alt="02 AM" data-base62-sha1="2SMcxcbL7oBxaRV2jYUXgPKiUhf" width="690" height="273" data-dominant-color="F0EBE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">02 AM</span><span class="informations">748×296 33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t even know about there’s a <code>dim</code> param in the <code>log_softmax</code> function.</p>

username: colesbury
cooked: <p>The dim parameter is new and will be in the next release. The docs are fixed too. Here’s what it says in <code>master</code>, if you build from source:</p>
<pre><code class="lang-auto">In [5]: ?torch.nn.functional.log_softmax
Signature: torch.nn.functional.log_softmax(input, dim=None, _stacklevel=3)
Docstring:
Applies a softmax followed by a logarithm.

While mathematically equivalent to log(softmax(x)), doing these two
operations separately is slower, and numerically unstable. This function
uses an alternative formulation to compute the output and gradient correctly.

See :class:`~torch.nn.LogSoftmax` for more details.

Arguments:
    input (Variable): input
    dim (int): A dimension along which log_softmax will be computed.
File:      /data/users/sgross/pytorch/torch/nn/functional.py
Type:      function
</code></pre>
<p>You can always use <a href="http://docs.pytorch.org">docs.pytorch.org</a></p>

username: Vanjoy
cooked: <p>Thank you for you information.</p>

username: Albert_Chen
cooked: <p>Yes, according to the official website<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">CrossEntroyLoss definition</a><br>
they are equivalent</p>

username: Oleksandra_Sopova
cooked: <p>Could you elaborate on “log_softmax gives different results depending on shape”? I’ve printed the shapes and they look the same.</p>

username: bigc
cooked: <p>This code works for me:</p>
<p>You need to specify <code>            o = F.log_softmax(o, dim=-1)</code></p>
<pre><code class="lang-python">
import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from torch.autograd import Variable


class Net(nn.Module):

    def __init__(self, n_features, n_hiddens, n_classes):
        super(Net, self).__init__()
        self.gru = torch.nn.GRU(n_features, n_hiddens)
        self.linear = torch.nn.Linear(n_hiddens, n_classes)

    def forward(self, x, flag=True):
        o, h = self.gru(x)
        o = self.linear(o)

        if flag:
            o = F.log_softmax(o, dim=-1)

        return o


n_steps = 10
n_classes = 100
mb_size = 32
n_features = 50
n_hiddens = 60

net = Net(n_features, n_hiddens, n_classes)

loss1 = torch.nn.NLLLoss(size_average=False)
loss2 = torch.nn.CrossEntropyLoss(size_average=False)

x = Variable(torch.rand(n_steps, mb_size, n_features))
y = Variable(
    torch.LongTensor(np.random.randint(0, n_classes, (n_steps, mb_size))))

logits1 = net(x, flag=True).view(-1, n_classes)
logits2 = net(x, flag=False).view(-1, n_classes)

loss_val1 = loss1(logits1, y.view(-1))
loss_val2 = loss2(logits2, y.view(-1))

print(loss_val1)
print(loss_val2)
</code></pre>