Help me decreasing loss

username: harshraj22
cooked: <p><em>Hello Community</em>, <img src="https://discuss.pytorch.org/images/emoji/apple/wave.png?v=9" title=":wave:" class="emoji" alt=":wave:"> <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"><br>
I have recently started deep learning, and was trying a project of image classification using transfer learning.<br>
I used <code>InceptionResnetV2</code> model in pytorch (from <a href="https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/inceptionresnetv2.py" rel="noopener nofollow ugc">this repo</a> ) pretrained on <a href="http://www.image-net.org/" rel="noopener nofollow ugc">imagenet</a> dataset.<br>
Sadly, even after 40 epochs, the accuracy is somewhere around 15% (Yes, I used ReduceOnPlateaue LR Schedular). I am not sure if it is because of some bug that I might have accidently created in code while writing training script, or any other silly mistakes. Attached is the training part of my code with loss &amp; accuracy details per epoch <strong>(full script at the bottom)</strong>:</p>
<pre><code class="lang-python">from tqdm.autonotebook import tqdm
from copy import deepcopy

n_epoch = 40
model.train()
epoch_losses = {
    'val': [],
    'train': []
}

epoch_accs = {
    'val': [],
    'train': []
}

best_acc = 0.0
best_model = deepcopy(model.state_dict())


for epoch in range(n_epoch):
  epoch_loss = 0.0
  epoch_acc = 0.0
  print('=' * 25, '[Epoch:', epoch+1, '/', n_epoch, ']', '=' * 25)

  for phase in ['train', 'val']:
    if phase == 'train':
      model.train()
    else:
      model.eval()

    running_loss = 0.0
    running_correct = 0.0

    for data in tqdm(snakeDataloader[phase], position=0, leave=False):
      img, label = data['img'], data['label']
      img = img.to(device)
      label = torch.tensor(label).to(device)

      optimizer.zero_grad()

      with torch.set_grad_enabled(phase=='train'):
        outs = model(img)
        preds = torch.argmax(outs, 1)
        loss = criterion(outs, label)

        if phase == 'train':
          loss.backward()
          optimizer.step()
        
        running_loss += loss.item() * img.size()[0]
        running_correct += torch.sum(preds == label.data)
    
    epoch_loss = running_loss / len(snakeDataloader[phase])
    epoch_acc = running_correct.double() / len(snakeDataloader[phase])

    print(f'[{phase}] =&gt; Acc: {epoch_acc :.2f}  Loss: {epoch_loss :.2f}')

    epoch_losses[phase].append(epoch_loss)
    epoch_accs[phase].append(epoch_acc)

    if phase == 'val' and epoch_acc &gt; best_acc:
      best_acc = epoch_acc
      best_model = deepcopy(model.state_dict())
    elif phase == 'train':
      scheduler.step(epoch_loss)


</code></pre>
<pre><code class="lang-auto">========================= [Epoch: 1 / 40 ] =========================
[train] =&gt; Acc: 5.92  Loss: 255.16
[val] =&gt; Acc: 8.46  Loss: 208.18
========================= [Epoch: 2 / 40 ] =========================
[train] =&gt; Acc: 8.82  Loss: 201.05
[val] =&gt; Acc: 8.35  Loss: 209.16
========================= [Epoch: 3 / 40 ] =========================
[train] =&gt; Acc: 9.70  Loss: 196.58
[val] =&gt; Acc: 10.73  Loss: 198.33
========================= [Epoch: 4 / 40 ] =========================
[train] =&gt; Acc: 10.25  Loss: 194.29
[val] =&gt; Acc: 10.69  Loss: 197.92
========================= [Epoch: 5 / 40 ] =========================
[train] =&gt; Acc: 10.38  Loss: 192.18
[val] =&gt; Acc: 10.92  Loss: 195.03
========================= [Epoch: 6 / 40 ] =========================
[train] =&gt; Acc: 11.51  Loss: 188.25
[val] =&gt; Acc: 10.92  Loss: 193.92
========================= [Epoch: 7 / 40 ] =========================
[train] =&gt; Acc: 10.44  Loss: 190.62
[val] =&gt; Acc: 10.92  Loss: 193.32
========================= [Epoch: 8 / 40 ] =========================
[train] =&gt; Acc: 11.74  Loss: 187.99
[val] =&gt; Acc: 10.85  Loss: 194.34
========================= [Epoch: 9 / 40 ] =========================
[train] =&gt; Acc: 11.41  Loss: 188.46
[val] =&gt; Acc: 11.35  Loss: 194.20
========================= [Epoch: 10 / 40 ] =========================
[train] =&gt; Acc: 11.93  Loss: 185.79
[val] =&gt; Acc: 11.35  Loss: 197.31
========================= [Epoch: 11 / 40 ] =========================
[train] =&gt; Acc: 12.20  Loss: 183.68
[val] =&gt; Acc: 10.92  Loss: 199.71
========================= [Epoch: 12 / 40 ] =========================
[train] =&gt; Acc: 11.69  Loss: 189.29
[val] =&gt; Acc: 12.04  Loss: 190.32
========================= [Epoch: 13 / 40 ] =========================
[train] =&gt; Acc: 12.49  Loss: 183.14
[val] =&gt; Acc: 11.58  Loss: 191.34
========================= [Epoch: 14 / 40 ] =========================
[train] =&gt; Acc: 11.92  Loss: 184.91
[val] =&gt; Acc: 11.85  Loss: 192.28
========================= [Epoch: 15 / 40 ] =========================
[train] =&gt; Acc: 12.59  Loss: 183.28
[val] =&gt; Acc: 12.23  Loss: 189.71
========================= [Epoch: 16 / 40 ] =========================
[train] =&gt; Acc: 12.21  Loss: 183.75
[val] =&gt; Acc: 11.58  Loss: 190.22
========================= [Epoch: 17 / 40 ] =========================
[train] =&gt; Acc: 12.57  Loss: 182.45
[val] =&gt; Acc: 11.65  Loss: 192.05
========================= [Epoch: 18 / 40 ] =========================
[train] =&gt; Acc: 13.02  Loss: 182.48
[val] =&gt; Acc: 11.35  Loss: 192.22
========================= [Epoch: 19 / 40 ] =========================
[train] =&gt; Acc: 12.30  Loss: 182.98
[val] =&gt; Acc: 12.38  Loss: 189.69
========================= [Epoch: 20 / 40 ] =========================
[train] =&gt; Acc: 12.70  Loss: 182.28
[val] =&gt; Acc: 12.62  Loss: 188.30
========================= [Epoch: 21 / 40 ] =========================
[train] =&gt; Acc: 12.31  Loss: 181.50
[val] =&gt; Acc: 12.92  Loss: 186.95
========================= [Epoch: 22 / 40 ] =========================
[train] =&gt; Acc: 12.98  Loss: 180.89
[val] =&gt; Acc: 12.04  Loss: 191.68
========================= [Epoch: 23 / 40 ] =========================
[train] =&gt; Acc: 12.57  Loss: 182.21
[val] =&gt; Acc: 12.54  Loss: 188.88
========================= [Epoch: 24 / 40 ] =========================
[train] =&gt; Acc: 13.28  Loss: 178.77
[val] =&gt; Acc: 11.81  Loss: 192.00
========================= [Epoch: 25 / 40 ] =========================
[train] =&gt; Acc: 12.80  Loss: 179.73
[val] =&gt; Acc: 12.69  Loss: 189.20
========================= [Epoch: 26 / 40 ] =========================
[train] =&gt; Acc: 13.39  Loss: 178.53
[val] =&gt; Acc: 13.50  Loss: 186.30
========================= [Epoch: 27 / 40 ] =========================
[train] =&gt; Acc: 13.13  Loss: 179.55
[val] =&gt; Acc: 11.88  Loss: 191.30
========================= [Epoch: 28 / 40 ] =========================
[train] =&gt; Acc: 12.74  Loss: 180.83
[val] =&gt; Acc: 11.50  Loss: 192.44
========================= [Epoch: 29 / 40 ] =========================
[train] =&gt; Acc: 13.61  Loss: 178.84
[val] =&gt; Acc: 12.38  Loss: 186.29
========================= [Epoch: 30 / 40 ] =========================
[train] =&gt; Acc: 13.67  Loss: 177.28
[val] =&gt; Acc: 12.69  Loss: 188.18
========================= [Epoch: 31 / 40 ] =========================
[train] =&gt; Acc: 13.31  Loss: 179.06
[val] =&gt; Acc: 11.69  Loss: 190.83
========================= [Epoch: 32 / 40 ] =========================
[train] =&gt; Acc: 13.67  Loss: 177.76
[val] =&gt; Acc: 12.96  Loss: 188.28
========================= [Epoch: 33 / 40 ] =========================
[train] =&gt; Acc: 13.98  Loss: 177.72
[val] =&gt; Acc: 12.81  Loss: 186.19
========================= [Epoch: 34 / 40 ] =========================
[train] =&gt; Acc: 13.23  Loss: 178.71
[val] =&gt; Acc: 12.62  Loss: 186.99
========================= [Epoch: 35 / 40 ] =========================
[train] =&gt; Acc: 13.26  Loss: 178.14
Epoch    35: reducing learning rate of group 0 to 1.0000e-03.
[val] =&gt; Acc: 12.19  Loss: 187.05
========================= [Epoch: 36 / 40 ] =========================
[train] =&gt; Acc: 13.92  Loss: 174.30
[val] =&gt; Acc: 13.23  Loss: 184.83
========================= [Epoch: 37 / 40 ] =========================
[train] =&gt; Acc: 14.70  Loss: 172.65
[val] =&gt; Acc: 13.08  Loss: 183.59
========================= [Epoch: 38 / 40 ] =========================
[train] =&gt; Acc: 14.64  Loss: 172.27
[val] =&gt; Acc: 13.38  Loss: 183.50
========================= [Epoch: 39 / 40 ] =========================
[train] =&gt; Acc: 14.79  Loss: 172.33
[val] =&gt; Acc: 13.12  Loss: 183.73
========================= [Epoch: 40 / 40 ] =========================
[train] =&gt; Acc: 15.10  Loss: 170.81
[val] =&gt; Acc: 13.27  Loss: 182.62
</code></pre>
<p>Full notebook can be found <a href="https://colab.research.google.com/github/harshraj22/DeepLearning/blob/main/submissions/HackerEarth:%20%5BSnake%20In%20The%20Hood%5D/hackerearths_snakes_in_the_hood.ipynb#scrollTo=gsb0JLTQtYQ9" rel="noopener nofollow ugc">here</a></p>
<p>P.S. I have already gone through posts with similar title (eg <a href="https://discuss.pytorch.org/t/loss-not-decreasing-despite-reducing-model-complexity/60455">this</a>) but they didn’t seem any helpful with this case.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> ?</p>

username: Preetham_R_Patlolla
cooked: <p>Can you show your model? How are you tweaking it to your particular application? Usually, people replace the last FC layer with their own layer enabling the model to learn the low-level features specific to the particular dataset. Try replacing your FC layer and then train it.</p>

username: harshraj22
cooked: <p>Yes, <a href="https://colab.research.google.com/github/harshraj22/DeepLearning/blob/main/submissions/HackerEarth:%20%5BSnake%20In%20The%20Hood%5D/hackerearths_snakes_in_the_hood.ipynb#scrollTo=gsb0JLTQtYQ9" rel="noopener nofollow ugc">here</a> is the colab notebook that I used. and <a href="https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/inceptionresnetv2.py" rel="noopener nofollow ugc">this</a> is source of my model (which automatically downloads the weights). For last layer, I did the following:</p>
<pre><code class="lang-auto">model.last_linear = nn.Sequential(
    nn.Linear(1536, 700),
    nn.ReLU(inplace=True),
    nn.Linear(700, 35)
)
</code></pre>