PyTorch Constant Loss

username: Heisenberg_082001
cooked: <p>I am trying to do multiclass-classification using simple ANN .<br>
Dont know why my loss is not decreasing it remains constant</p>
<p><strong>CODE-</strong></p>
<pre><code class="lang-auto">class NET(nn.Module):
def __init__(self):
    super().__init__()
    self.model=nn.Sequential(
        nn.Linear(6,512),
        nn.ReLU(),
        nn.Linear(512,1024),
        nn.ReLU(),
        nn.Linear(1024,17),
        nn.Softmax()
    )
def forward(self ,x):
    return self.model(x)

net=NET().to(device)

opt=optim.Adam(net.parameters() , lr=0.01)
Loss_fn=nn.L1Loss()


%%time
epochs=10
loss_arr=[]

for epoch in range(epochs):
     opt.zero_grad()

     outputs=net(train_data)
     loss=Loss_fn(outputs ,train_out )
     loss.backward()

    opt.step()
    loss_arr.append(loss.item())
    print("Epochs : %d/%d ,loss :%f" % (epoch ,epochs ,loss))
</code></pre>
<p>I want Mean absolute error that’s why i am using nn.L1Loss()</p>
<p><strong>OUTPUT -</strong></p>
<pre><code class="lang-auto">Epochs : 0/10 ,loss :10.401250
Epochs : 1/10 ,loss :10.401250
Epochs : 2/10 ,loss :10.401250
Epochs : 3/10 ,loss :10.401250
Epochs : 4/10 ,loss :10.401250
Epochs : 5/10 ,loss :10.401250
Epochs : 6/10 ,loss :10.401250
Epochs : 7/10 ,loss :10.401250
Epochs : 8/10 ,loss :10.401250
Epochs : 9/10 ,loss :10.401250
Wall time: 1.62 s
</code></pre>

username: ptrblck
cooked: <p>The loss changes for random input data using your code snippet:</p>
<pre><code class="lang-python">train_data = torch.randn(64, 6)
train_out = torch.empty(64, 17).uniform_(0, 1)
</code></pre>
<p>so I would recommend to play around with some hyperparameters, such as the learning rate.</p>
<p>That being said, I’m not familiar with your use case, but a <code>softmax</code> output in <code>L1Loss</code> doesn’t seem to be the usual use case.<br>
If you are dealing with a multi class classification use case, I would recommend to try out <code>nn.CrossEntropyLoss</code> and pass the raw logits to it.</p>

username: Heisenberg_082001
cooked: <p>Thanks !! removing softmax layer worked</p>

username: Heisenberg_082001
cooked: <p>I want to Trouble you again .<br>
Sorry for this ,Plz help me<br>
I am facing same issue with another model.I am building simple digit classifier using ANN</p>
<p>code=</p>
<pre><code class="lang-auto">class NET(nn.Module):
    def __init__(self):
        super().__init__()
        self.model=nn.Sequential(
            nn.Linear(784, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
        
        )
    def forward(self,x):
            return self.model(x)


loss_fn=nn.CrossEntropyLoss()
opt=optim.Adam(net.parameters() , lr=0.01)

loss_epoch=[]
epochs=5
for i in range(epochs):
        
        opt.zero_grad()
        output=net(x_train)
        loss=loss_fn(output, y_train)
        loss.backward()
        opt.step()
        loss_epoch.append(loss.item())
        
        print("Epochs: {}/{} , Loss:{}".format(i,epochs,loss))
</code></pre>
<p>OUTPUT=</p>
<pre><code class="lang-auto"> Epochs: 0/5, Loss:2.301159143447876

 Epochs: 1/5, Loss:2.301161289215088

 Epochs: 2/5, Loss:2.3011562824249268

 Epochs: 3/5, Loss:2.3011701107025146

 Epochs: 4/5, Loss:2.3011698722839355
</code></pre>
<p>data- Mnist Digit data</p>

username: ptrblck
cooked: <p>Again, I would recommend to play around with hyperparameters, such as the learning rate.<br>
Since the loss changes and no dropout (or other “random” operations are used), it would mean that the parameters get some updates but the overall training isn’t able to reduce the loss.</p>
<p>Start by using a lower learning rate (e.g. <code>1e-3</code>) and try to overfit the current data sample.</p>

username: Heisenberg_082001
cooked: <p>Thank you very much .It was due to high LR.<br>
Loss starts decreasing when LR is <code>1e-7</code></p>