Model memorizing patterns not generalizing - resnet 50 transfer learning

username: szymonindy
cooked: <p>Hello everyone, I am training my model so that It could recognize pneumonia and normal condition based on the following dataset.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia" target="_blank" rel="noopener nofollow ugc">kaggle.com</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://storage.googleapis.com/kaggle-datasets-images/17810/23340/c8372ebbe20b0f671c2f3c501ba51412/dataset-card.jpeg?t=2018-03-24-19-05-18" class="thumbnail onebox-avatar">

<h3><a href="https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia" target="_blank" rel="noopener nofollow ugc">Chest X-Ray Images (Pneumonia)</a></h3>

  <p>5,863 images, 2 categories</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I want to apply transfer learning to this problem. I am using resnet50 network.</p>
<pre><code class="lang-auto">model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False 

model.fc = nn.Sequential(
               nn.Linear(2048, 128),
               nn.ReLU(inplace=True),
               nn.Linear(128, 2)).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.fc.parameters(), lr=learning_rate)

model.to(device)

**23,770,562 total parameters.**
**262,530 training parameters.**

</code></pre>
<p>Here is my training process:</p>
<pre><code class="lang-auto"># initialize the early_stopping object
model.eval()
early_stopping = pytorchtools.EarlyStopping(patience=patience, verbose=True)
for epoch in range(num_epochs):
    ##########################    
    #######TRAIN MODEL########
    ##########################
    epochs_loss=0
#     model.train()
    for i, (images, labels) in enumerate(train_dl):
        # Move tensors to the configured device
        model.train()
        images = images.to(device)
        labels = labels.to(device)
        # Forward pass
        outputs = model(images).to(device)
        loss = criterion(outputs, labels)
        
        # Backprpagation and optimization
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        
        #calculate train_loss
        train_losses.append(loss.item())
    
    ##########################    
    #####VALIDATE MODEL#######
    ##########################
    for images, labels in val_dl:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images).to(device)
        loss = criterion(outputs,labels)
        valid_losses.append(loss.item())
    
    # print training/validation statistics 
    # calculate average loss over an epoch
    train_loss = np.average(train_losses)
    valid_loss = np.average(valid_losses)
#     print(train_loss)
    avg_train_losses.append(train_loss)
    avg_valid_losses.append(valid_loss)
    
    print_msg = (f'train_loss: {train_loss:.5f} ' + f'valid_loss: {valid_loss:.5f}')
    
    print(print_msg)
    
    # clear lists to track next epoch
    train_losses = []
    valid_losses = []
    
    early_stopping(valid_loss, model)
    print(epoch)
        
    if early_stopping.early_stop:
        print("Early stopping")
        break
</code></pre>
<p>Here are the results. My model is memorizing this may be caused be overfitting but I have no idea how to fix it and what’s wrong with it.</p>
<blockquote>
<p>train_loss: 0.59755 valid_loss: 0.82625<br>
Validation loss decreased (inf → 0.826249).  Saving model …<br>
0<br>
train_loss: 0.52524 valid_loss: 0.83933<br>
EarlyStopping counter: 1 out of 5<br>
1<br>
train_loss: 0.48533 valid_loss: 0.89458<br>
EarlyStopping counter: 2 out of 5<br>
2<br>
train_loss: 0.43887 valid_loss: 0.97882<br>
EarlyStopping counter: 3 out of 5<br>
3<br>
train_loss: 0.40483 valid_loss: 1.03101<br>
EarlyStopping counter: 4 out of 5<br>
4<br>
train_loss: 0.37320 valid_loss: 1.04973<br>
EarlyStopping counter: 5 out of 5</p>
</blockquote>
<p>I would be really grateful if you could suggest me what’s wrong with my code or my logic behind this experiment <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1Xd5E7Amqw62aa21hpSLzpzyacIc_rwdB?authuser=2#scrollTo=7_yC8Ecx5E6v">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/2580b6a0a5b3a23f1efbba8a2eebb5a3/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1Xd5E7Amqw62aa21hpSLzpzyacIc_rwdB?authuser=2#scrollTo=7_yC8Ecx5E6v" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1Xd5E7Amqw62aa21hpSLzpzyacIc_rwdB?authuser=2#scrollTo=7_yC8Ecx5E6v" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/szymonrucinski/x-ray/blob/master/pneumonia.ipynb">
  <header class="source">

      <a href="https://github.com/szymonrucinski/x-ray/blob/master/pneumonia.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/szymonrucinski/x-ray/blob/master/pneumonia.ipynb" target="_blank" rel="noopener nofollow ugc">szymonrucinski/x-ray/blob/master/pneumonia.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchvision.models as models\n",
    "from pathlib import Path\n",
    "import os\n",
    "import glob\n",
    "import cv2\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from torch.utils.data import Dataset, random_split, DataLoader\n",
    "from PIL import Image\n",
    "import torchvision.models as models\n",
</code></pre>



  This file has been truncated. <a href="https://github.com/szymonrucinski/x-ray/blob/master/pneumonia.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: AleAyotte
cooked: <p>To avoid overfitting, you could add a dropout layer before your first fully connected layer, and see if it help. If it do not help, you should try to unfreeze a couple of layers in your model, from my experience, it could help a lot to give your model enough capacity to learn deeper pattern that can be generalized.</p>

username: pascal_notsawo
cooked: <p>Just a remark.<br>
If it is a binary classification as the notebook suggests, why not just use an output of dimension 1 (<code>nn.Linear(128, 1)</code>) and use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html" rel="noopener nofollow ugc">binary cross-entropy with logits loss</a>.</p>