Loss is increasing and accuracy is decreasing

username: Lucky_Magna
cooked: <p>I’m trying to train a Pneumonia classifier using Resnet34. While training the model, the loss is increasing and accuracy is decreasing drastically (both in training and validation sets). What might be the potential reason behind this?</p>
<pre><code class="lang-auto">def train(model, dataloaders, loss, optimizer, epochs=5):

  train = dataloaders['train']

  valid = dataloaders['valid']

  device = 'cuda' if torch.cuda.is_available() else 'cpu'

  metric = Accuracy().to(device)

  for epoch in tqdm(range(epochs), desc="EPOCHS : "):

    model.train()

    cst = 0

    for x, y in tqdm(train, leave=True, desc="Trainning : "):

      optimizer.zero_grad()

      x = x.to(device)

      y = y.to(device)

      preds = model(x).to(device)

      acc = metric(preds.argmax(dim=1), y)

      cost = loss(preds, y)

      cst += cost.item()

      cost.backward()

      optimizer.step()

    acc = metric.compute()

    cst /= len(train)

    print(f'Train loss : {cst} \t Train acc : {acc}')

    model.eval()

    cst = 0

    for x, y in tqdm(valid, leave=True, desc="Validation : "):

      x = x.to(device)

      y = y.to(device)

      preds = model(x).to(device)

      acc = metric(preds.argmax(dim=1), y)

      cost = loss(preds, y)

      cst += cost.item()

    acc = metric.compute()

    cst /= len(valid)

    print(f'Valid loss : {cst} \t Valid acc : {acc}')

  return model
</code></pre>
<pre><code class="lang-auto">model = models.resnet34(pretrained=True)

for param in model.parameters():

  param.requires_grad = False

model.fc = nn.Sequential(

    nn.Dropout(p=.7),

    nn.Linear(in_features=model.fc.in_features, out_features=2),

    nn.LogSoftmax(dim=1)

)

model = model.to(device)

LR = 3e-3

WD = 1e-4

loss = nn.NLLLoss()

optimizer = optim.Adam(model.parameters(), lr=LR, weight_decay=WD)

md = train(model, dataloaders, loss, optimizer, epochs=5)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/1/311ed06b5ee6cf2840e108410a6dd6512f8fce8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/311ed06b5ee6cf2840e108410a6dd6512f8fce8b" title="Screenshot (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/311ed06b5ee6cf2840e108410a6dd6512f8fce8b_2_690x388.png" alt="Screenshot (1)" data-base62-sha1="70xpJpoLB7y8VdttYPFhTaNkkjN" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/311ed06b5ee6cf2840e108410a6dd6512f8fce8b_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/311ed06b5ee6cf2840e108410a6dd6512f8fce8b_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/1/311ed06b5ee6cf2840e108410a6dd6512f8fce8b_2_1380x776.png 2x" data-dominant-color="F5F8F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (1)</span><span class="informations">1920×1080 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: krishna511
cooked: <p>Well, the obvious answer is, nothing wrong here, if the model is not suited for your data distribution then, it simply won’t work for desirable results. And another thing is I think you should reframe your question If loss increase then certainly acc will decrease.<br>
That’s just my opinion, I may not be to the point here.</p>

username: Lucky_Magna
cooked: <p>I tried different architectures as well, but the result is the same. And I don’t think I should reframe the question, as you can see from the screenshot.</p>

username: krishna511
cooked: <p><a class="mention" href="/u/lucky_magna">@Lucky_Magna</a> By reframing I meant this is obvious if loss decrease acc will increase.</p>

username: Lucky_Magna
cooked: <p>Can you suggest any other solution to solve the problem.</p>

username: eqy
cooked: <p>Can you check the initial loss of your model with random data? It should be around -ln(1/num_classes). If this value is close then it suggests that your model is initialized properly. The next thing to check would be that your data format as input to the model makes sense (e.g., from the perspective of data layout, etc.)</p>
<p>From here, if your loss is not even going down initially, you can try simple tricks like decreasing the learning rate until it starts training. If the loss is going down initially but stops improving later, you can try things like more aggressive data augmentation or other regularization techniques.</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/eqy">@eqy</a>  Will try it and let you know.</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/eqy">@eqy</a> Loss of the model with random data is very close to -ln(1/num_classes), as you mentioned. As for the data, it is in the right format.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d942bd003e7f589f9d642c688b038aee39c05fed.png" data-download-href="https://discuss.pytorch.org/uploads/default/d942bd003e7f589f9d642c688b038aee39c05fed" title="Screenshot (2)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d942bd003e7f589f9d642c688b038aee39c05fed.png" alt="Screenshot (2)" data-base62-sha1="uZYB1EkrCcPrUtgc3f4YwfQmFU9" width="690" height="131" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (2)</span><span class="informations">1067×204 9.87 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: eqy
cooked: <p>Great, what does the loss curve look like with smaller learning rates?</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/eqy">@eqy</a> I changed the model from resnet34 to renset18. The loss is stable, but the model is learning very slowly. The accuracy is starting from around 25% and raising eventually but in a very slow manner. It is taking around 10 to 15 epochs to reach 60% accuracy. I tried increasing the learning_rate, but the results don’t differ that much.</p>

username: eqy
cooked: <p>Ok, that sounds normal. At this point I would see if there are any data augmentations that you can apply that make sense for you dataset, as well as other model architectures, etc.</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/eqy">@eqy</a> Ok let me explain about the project I’m working on. I’m trying to classify Pneumonia patients using X-ray copies. Below mentioned are the transforms I’m currently using.</p>
<pre><code class="lang-auto">transform = {

   'train' : T.Compose([

                        T.Resize(size=(224,224)),

                        T.RandomAffine(30),

                        T.RandomInvert(p=1),

                        T.RandomHorizontalFlip(),

                        T.ToTensor(),

                        T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

   ]),

   'valid' : T.Compose([

                        T.Resize(size=(224,224)),

                        T.RandomInvert(p=1),

                        T.ToTensor(),

                        T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

   ])

}
</code></pre>
<p>Before you may ask why am I using Invert transform on the validation set, I think this transform is able to capture the pneumonia parts in the x-ray copies. So, I used it on validation and test set as well (If it is a bad idea the correct me). After applying  the transforms the images look something like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c" title="68ac12a3-19c8-4c71-9e00-46c9fcc9248d"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c_2_488x499.jpeg" alt="68ac12a3-19c8-4c71-9e00-46c9fcc9248d" data-base62-sha1="v6mD9khFjz3DjfswXBtuaPTFBqA" width="488" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c_2_488x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9fb95ff92fed6feb5a56a90a037409b15ff5f4c.jpeg 2x" data-dominant-color="8C8C8C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">68ac12a3-19c8-4c71-9e00-46c9fcc9248d</span><span class="informations">670×686 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/1365b42cb8717723166f85e82f7c1b304941c838.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1365b42cb8717723166f85e82f7c1b304941c838" title="87aede22-f1af-4713-be47-59db9804d1bf"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/1365b42cb8717723166f85e82f7c1b304941c838_2_488x499.jpeg" alt="87aede22-f1af-4713-be47-59db9804d1bf" data-base62-sha1="2LAYdPJyw5CqPo0H6JOBoCyn0cE" width="488" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/1365b42cb8717723166f85e82f7c1b304941c838_2_488x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/1/3/1365b42cb8717723166f85e82f7c1b304941c838.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/1365b42cb8717723166f85e82f7c1b304941c838.jpeg 2x" data-dominant-color="8D8D8D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">87aede22-f1af-4713-be47-59db9804d1bf</span><span class="informations">670×686 98.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/573d89ac6b98ec782de322e2945d2073ebd34e68.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/573d89ac6b98ec782de322e2945d2073ebd34e68" title="311762ab-ee6c-438c-b216-9c1bbccdd958"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573d89ac6b98ec782de322e2945d2073ebd34e68_2_488x499.jpeg" alt="311762ab-ee6c-438c-b216-9c1bbccdd958" data-base62-sha1="crLnOd6MqY8lgiVtyO2cMwBuYpa" width="488" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/573d89ac6b98ec782de322e2945d2073ebd34e68_2_488x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/7/573d89ac6b98ec782de322e2945d2073ebd34e68.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/573d89ac6b98ec782de322e2945d2073ebd34e68.jpeg 2x" data-dominant-color="989898"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">311762ab-ee6c-438c-b216-9c1bbccdd958</span><span class="informations">670×686 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/eqy">@eqy</a> Solved it! I forgot to shuffle the dataset. It is overfitting to one class in the whole dataset. Thanks for the help though.</p>

username: Nahil_Sobh
cooked: <p>Nice.<br>
<a class="mention" href="/u/lucky_magna">@Lucky_Magna</a>  Could you please share the performance of your final model?<br>
Like the training and validation losses plots and possibly accuracy plots as well.</p>
<p>Thx.</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/nahil_sobh">@Nahil_Sobh</a> I posted the code on my github account you can see the performance there.</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/narayana8799/Pneumonia-Detection-using-Pytorch/blob/master/Pneumonia%20Detection.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/narayana8799/Pneumonia-Detection-using-Pytorch/blob/master/Pneumonia%20Detection.ipynb" target="_blank" rel="noopener nofollow ugc">narayana8799/Pneumonia-Detection-using-Pytorch/blob/master/Pneumonia Detection.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "764e6867-8a51-4090-b806-07a176c73fe6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "_CudaDeviceProperties(name='NVIDIA GeForce GTX 1050 Ti', major=6, minor=1, total_memory=4096MB, multi_processor_count=6)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
</code></pre>

  This file has been truncated. <a href="https://github.com/narayana8799/Pneumonia-Detection-using-Pytorch/blob/master/Pneumonia%20Detection.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Nahil_Sobh
cooked: <p>Thanks for sharing .</p>

username: Lucky_Magna
cooked: <p><a class="mention" href="/u/nahil_sobh">@Nahil_Sobh</a> Share your model performance once you have optimized it.<br>
Thank you.</p>