Loss is more than expected (>1) Neural net

username: the_coder
cooked: <p>I have a image dataset with 40 classes, i am trying to classify them using a neural network.<br>
The shape of my input tensor i (1,1,64,64), so the batch size is 1 and the image is grayscale.</p>
<p>i ran the following code , but i am getting initial loss in around 3.4 , which is more than 1. What could be the reason that my model is not learning ?</p>
<blockquote>
<pre><code>class neural_net(nn.Module):

def __init__(self):
    super().__init__()
    self.input = nn.Linear(64 * 64, 256)
    self.hidden1 = nn.Linear(256, 256)
    self.output = nn.Linear(256, 40)

def forward(self, x):
    x = F.relu(self.input(x))
    x = F.relu(self.hidden1(x))
    x = self.output(x)
    return F.log_softmax(x,dim=1)
</code></pre>
</blockquote>
<blockquote>
<pre><code>network1 = neural_net()
optimizer = optim.Adam(network1.parameters(), lr = 0.001)
</code></pre>
</blockquote>
<blockquote>
<pre><code>for epoch in range(10):
    for data in train_dl:  
         X, y = data  
        network1.zero_grad()  
        output = network1(X.view(-1,4096))  
       loss = F.cross_entropy(output, y)  
       loss.backward()  
       optimizer.step() 
print(loss) 
</code></pre>
</blockquote>
<p>Also there is another issue i found, i am giving cross_entropy loss to calculate the loss but i am getting NLLLossbackword in the output. What could be the reason for that.</p>

username: chetan_patil
cooked: <p>Hi,</p>
<ol>
<li>During the first forward-pass, loss is expected to be around <code>log(num_classes)</code> ie<br>
<code>log(40) = 3.688</code>.</li>
<li>Adding to your point of <code>loss&gt;1</code>, it is not necessary that loss should be always &lt;1.</li>
<li>
<code>nn.CrossEntropyLoss()</code> is actually a combination of <code>nn.LogSoftmax()</code> and <code>nn.NLLLoss()</code>. Refer  <a href="https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html" rel="nofollow noopener">this</a> doc.<br>
That’s why it when you print <code>loss</code>, it shows <code>&lt;grad_fn = NLLLossBackward&gt;</code>
</li>
</ol>

username: Haider_Ali_Shuvo
cooked: <p>The Cross Entropy loss applies softmax inside itself, So either use Cross Entropy loss on top of Unnormalized Scores or use Softmax+NLLL Loss</p>

username: chetan_patil
cooked: <aside class="quote no-group quote-modified" data-username="the_coder" data-post="1" data-topic="88779" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/the_coder/48/19504_2.png" class="avatar"> the_coder:</div>
<blockquote>
<blockquote>
<pre><code>def forward(self, x):
    x = F.relu(self.input(x))
    x = F.relu(self.hidden1(x))
    x = self.output(x)
    return F.log_softmax(x,dim=1)
</code></pre>
</blockquote>
</blockquote>
</aside>
<p>Your code uses a <code>LogSoftmax()</code> in the last layer.<br>
In that case, the loss function you must use will be <code>F.nll_loss(output, y)</code> instead of <code>F.cross_entropy(output, y)</code></p>

username: alpha_omega_alpha
cooked: <p>Hi <a class="mention" href="/u/the_coder">@the_coder</a>,</p>
<p>In the training loop, you might also want to zero out the optimizer gradients instead of the network .<br>
So, try:</p>
<pre><code class="lang-auto">for epoch in range(10):
    for data in train_dl:  
        X, y = data  
        optimizer.zero_grad()  
        output = network1(X.view(-1,4096))  
        loss = F.cross_entropy(output, y)  
        loss.backward()  
        optimizer.step() 
print(loss)
</code></pre>

username: the_coder
cooked: <p>Still my model is not learning, it is giving same classification for all samples in the test dataset. so accuracy is around 2.5 %</p>

username: chetan_patil
cooked: <p>Assuming the last layer is <code>F.log_softmax()</code>, did you change the loss function to <code>F.nll_loss()</code> ?</p>

username: the_coder
cooked: <p>Yes, i changed the loss from cross_entropy to nll_loss , still the model refuses to learn.</p>

username: chetan_patil
cooked: <p>Okay, then you can try increasing the layers in the neural network.<br>
Also, just take one mini-batch and keep on using the same mini-batch and see if the loss really goes to zero or accuracy approaches 100%.<br>
That’s one way to debug your network.</p>