Pytorch nn.CrossEntropyLoss output wrong values when using one hot encoding for label

username: Jared
cooked: <p>I would like to pass in a tensor that contains 4 values like this</p>
<pre><code>[.45, .78, .23, .56]
</code></pre>
<p>And get an output from the network with only the last neuron light up like this</p>
<pre><code>[0,0,0,1]
</code></pre>
<p>I would then like to pass in a second tensor <code>[.67, .89, .34, .12]</code> and get an output from that looks like this <code>[1,0,0,0]</code> with only the first neuron light up.</p>
<pre><code>import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

train = [
    [torch.FloatTensor([[.45, .78, .23, .56]]), torch.FloatTensor([[0,0,0,1]]) ],
    [torch.FloatTensor([[.67, .89, .34, .12]]), torch.FloatTensor([[1,0,0,0]]) ]
]


class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 2)
        self.fc2 = nn.Linear(2, 4)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return F.softmax(x, dim=0)

net = Net()
print(net)

loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1, momentum=0.9, nesterov=True)
criterion = nn.CrossEntropyLoss()

for epoch in range(10):
    for data in train:
        X, y = data
        
        net.zero_grad()
        output = net(X)
        loss = criterion(output, y)
        loss.backward()
        optimizer.step()
        
    print(loss) 

# test
X, y = torch.tensor([.45, .78, .23, .56]), torch.tensor([0,0,0,1])
print(net(X), y)

X, y = torch.tensor([.67, .89, .34, .12]), torch.tensor([1,0,0,0])
print(net(X), y)
</code></pre>
<p>unfortunately the network is output rubbish that doesn’t correspond to any particular neuron light up</p>
<pre><code>tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor(1.3863, grad_fn=&lt;DivBackward1&gt;)
tensor([0.3175, 0.1841, 0.2063, 0.2922], grad_fn=&lt;SoftmaxBackward0&gt;) tensor([0, 0, 0, 1])
tensor([0.3129, 0.1935, 0.2102, 0.2834], grad_fn=&lt;SoftmaxBackward0&gt;) tensor([1, 0, 0, 0])
</code></pre>
<p>I’m getting outputs like this <code>[0.3175, 0.1841, 0.2063, 0.2922]</code> which is not what the network should be outputing.</p>
<p>I think this might be due to the way in which the CrossEntropyLoss function is comparing the training data (X) and the label y. I think this line is causing the problem <code>loss = criterion(output, y)</code>.</p>
<p>What loss function should I be using to get the desired results?</p>

username: Andrei_Cristea
cooked: <p>Hello Jared,</p>
<p>I haven’t checked if the code runs all the way through but an important correction: you shouldn’t be running Softmax inside your forward() function, as CrossEntropyLoss automatically computes Softmax on your output. The target is fine the way you have it. See more details / examples <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">here</a>.</p>
<p>Try again after you’ve replaced your return statement with simply return x, and see if that helps.</p>
<p>Good luck!<br>
Andrei</p>
<p>EDIT: Also try lowering your learning rate, as it could be too high.</p>