Softmax not summing to 1

Tank: <p>I am using a softmax layer at the end of my fully connected network. It does not sum to 1. This is an issue because it tends to increase proportional to the number of epochs. Not sure why this is happening here is the code.</p>
<pre><code>class NN(nn.Module):
    def __init__(self):
        super(NN, self).__init__()

        self.fc1 = nn.Linear(1024, 128)
        self.fc2 = nn.Linear(128, 128)
        self.softmax = nn.Softmax(0)

    def forward(self, input):
        output = self.fc1(input)
        output = self.fc2(output)
        ouput = self.softmax(output)
        return output



def train(x_tensor, y_tensor):
    output = rnn(x_tensor.view(-1))
    output = output.unsqueeze(0)
    y_tensor = y_tensor[0].unsqueeze(0)
    y_guess = np.argmax(np.array(y_tensor))  #.long()
    y_guess = torch.from_numpy(np.array(y_guess)).unsqueeze(0)
    loss = criterion(output, y_guess)  
    loss.backward()
    optimizer.step()
    return output, loss.item()


      rnn = NN()
      lr = 0.001
      optimizer = torch.optim.SGD(rnn.parameters(), lr = lr)
      criterion = nn.CrossEntropyLoss()</code></pre>

albanD: <p>Hi,</p>
<p>It should sum to 1 in most cases.<br>
Are the individual values very different / very small / very large?<br>
Could you provide an example input where you see this problem so that we can reproduce it?</p>

Tank: <p>even from the start it is not adding to 1.</p>
<p>sample output after a few thousand epochs</p>
<p>tensor([[   0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0154,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0938,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.7029,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
203.8531,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.8635,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000,    0.0000,    0.0000,    0.0000,    0.0000,<br>
0.0000,    0.0000]])</p>

albanD: <p>What are the input to the softmax function that return such values?</p>

Tank: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="58526">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>e the input to the softmax function that return such</p>
</blockquote>
</aside>
<p>not sure what was happening but it is fixed.</p>

ptrblck: <p>Unrelated to your question, but note that <code>nn.CrossEntropyLoss</code> expects logits as the model output not probabilities coming from softmax.<br>
Internally <code>F.log_softmax</code> and <code>nn.NLLLOSS</code> will be used so you can just remove the softmax as the output activation.</p>
<p>Also note that you can call <code>torch.arxmax</code> directly without transforming to bumpy and back to PyTorch.</p>

Tank: <p>Thank you for this advice. What is happening now  is that after a few 1000 epochs the net just returns all 0’s…</p>