CNN overfitting

Mallrat: <p>Hi everyone! I’m fairly new to deep learning and I’m trying to build an image classification model. Currently, there are four classes of various military aircraft (F15, F16, F18 and F35, the dataset was downloaded from here: <a href="https://www.kaggle.com/datasets/a2015003713/militaryaircraftdetectiondataset/data" class="inline-onebox" rel="noopener nofollow ugc">Military Aircraft Detection Dataset | Kaggle</a>) with roughly 1300-1500 images each (I only took the ones from the ‘crop’ folder). I split each class’s images into 80% training set, 10% validation set and 10% test set.<br>
The only transforms I applied were Resize (to 128 * 128) and ToTensor. I’m using the following architecture for the model:</p>
<p>class AircraftVision(nn.Module):</p>
<pre><code>def __init__(self, input_shape: int, hidden_units: int, output_shape: int):
    super().__init__()
    self.conv_block_1 = nn.Sequential(
        nn.Conv2d(in_channels=input_shape,
                  out_channels=hidden_units,
                  kernel_size=3,
                  padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Dropout(p=0.5)
    )
    self.conv_block_2 = nn.Sequential(
        nn.Conv2d(in_channels=hidden_units,
                  out_channels=output_shape,
                  kernel_size=3,
                  padding=1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Dropout(p=0.5)
    )
    self.conv_block_3 = nn.Sequential(
        nn.Conv2d(in_channels=output_shape,
                  out_channels=128,
                  kernel_size=3,
                  padding=1),z
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=2, stride=2),
        nn.Dropout(p=0.5)
    )
    self.classifier = nn.Sequential(
        nn.Flatten(),
        nn.Linear(in_features=output_shape*32*16,
                 out_features=256),
        nn.ReLU(),
        nn.Linear(in_features=256, out_features=len(class_names))
    )

def forward(self, x):
    x = self.conv_block_1(x)
    x = self.conv_block_2(x)
    x = self.conv_block_3(x)
    x = self.classifier(x)
    return x
</code></pre>
<p>The loss function is CrossEntropyLoss and the optimizer is Adam (learning rate = 0.001, weight decay = 1e-5). The number of epochs is 30.</p>
<p>After several attempts at training the model, I always have a pretty high training accuracy (~95%) and low training loss.However, the validation loss falls slightly (with the accuracy rising) but usually around epoch 15 the accuracy fluctuates at around 60%-65%, while the loss rises slightly but steadily.</p>
<p>As you can see, I already tried adding dropout layers and weight decay. I’ve played around with the learning rate, weight decay, kernel_size and padding in the convolutional layers and a few other minor changes to the architecture.</p>
<p>I have several suspicions why the model is overfitting so much (in no particular order):</p>
<ol>
<li>There’s something wrong with my architecture.</li>
<li>The data I’m using is not good enough, as some of the images are of pretty low quality.</li>
<li>The training/validaiton loop I’m using (slightly modified from one of Sebastian Raschka’s books) is wrong. Here it is:</li>
</ol>
<p>def train_val(model, num_epochs, train_dl, val_dl):</p>
<pre><code>loss_hist_train = [0] * num_epochs
accuracy_hist_train = [0] * num_epochs
loss_hist_valid = [0] * num_epochs
accuracy_hist_valid = [0] * num_epochs
for epoch in range(num_epochs):
    model.train()
    for x_batch, y_batch in train_dl:
        x_batch, y_batch = x_batch.to(device), y_batch.to(device)
        pred = model(x_batch)
        loss = loss_fn(pred, y_batch)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        loss_hist_train[epoch] += loss.item()*y_batch.size(0)
        is_correct = (torch.argmax(pred, dim=1) == y_batch).float()
        accuracy_hist_train[epoch] += is_correct.sum()
    loss_hist_train[epoch] /= len(train_dl.dataset)
    accuracy_hist_train[epoch] /= len(train_dl.dataset)

    model.eval()
    with torch.no_grad():
        for x_batch, y_batch in valid_dl:
            x_batch, y_batch = x_batch.to(device), y_batch.to(device)
            pred = model(x_batch)
            loss = loss_fn(pred, y_batch)
            loss_hist_valid[epoch] += loss.item()*y_batch.size(0)
            is_correct = (torch.argmax(pred, dim=1) == y_batch).float()
            accuracy_hist_valid[epoch] += is_correct.sum()
    loss_hist_valid[epoch] /= len(valid_dl.dataset)
    accuracy_hist_valid[epoch] /= len(valid_dl.dataset)

    print(f"Epoch: {epoch+1} | Train loss: {loss_hist_train[epoch]:.3f} | Train accuracy: {accuracy_hist_train[epoch]:.3f} | Validation loss: {loss_hist_valid[epoch]:.3f} | Validation accuracy: {accuracy_hist_valid[epoch]:.3f}")
return loss_hist_train, loss_hist_valid, accuracy_hist_train, accuracy_hist_valid
</code></pre>
<p>Sorry if this is a bit too much, but I’ve been working on this model for several days now and this overfitting is really starting to bug me. Does anyone have any advice on how to deal with this issue? Many thanks in advance!</p>

Mallrat: <p>Here are the results of my latest run. The only difference here is that I tried resizing the images to 224 *224. As you can see, the model start to overfit at epoch 10, after which the loss starts to rise and the accuracy starts to fluctuate:</p>
<p>Epoch: 1 | Train loss: 1.496 | Train accuracy: 0.260 | Validation loss: 1.383 | Validation accuracy: 0.261<br>
Epoch: 2 | Train loss: 1.383 | Train accuracy: 0.273 | Validation loss: 1.382 | Validation accuracy: 0.271<br>
Epoch: 3 | Train loss: 1.381 | Train accuracy: 0.282 | Validation loss: 1.383 | Validation accuracy: 0.271<br>
Epoch: 4 | Train loss: 1.382 | Train accuracy: 0.285 | Validation loss: 1.379 | Validation accuracy: 0.288<br>
Epoch: 5 | Train loss: 1.366 | Train accuracy: 0.296 | Validation loss: 1.345 | Validation accuracy: 0.346<br>
Epoch: 6 | Train loss: 1.292 | Train accuracy: 0.387 | Validation loss: 1.296 | Validation accuracy: 0.414<br>
Epoch: 7 | Train loss: 1.174 | Train accuracy: 0.470 | Validation loss: 1.249 | Validation accuracy: 0.447<br>
Epoch: 8 | Train loss: 1.028 | Train accuracy: 0.551 | Validation loss: 1.194 | Validation accuracy: 0.495<br>
Epoch: 9 | Train loss: 0.860 | Train accuracy: 0.640 | Validation loss: 1.165 | Validation accuracy: 0.534<br>
Epoch: 10 | Train loss: 0.672 | Train accuracy: 0.720 | Validation loss: 1.143 | Validation accuracy: 0.565<br>
Epoch: 11 | Train loss: 0.516 | Train accuracy: 0.797 | Validation loss: 1.276 | Validation accuracy: 0.530<br>
Epoch: 12 | Train loss: 0.413 | Train accuracy: 0.832 | Validation loss: 1.488 | Validation accuracy: 0.549<br>
Epoch: 13 | Train loss: 0.344 | Train accuracy: 0.866 | Validation loss: 1.365 | Validation accuracy: 0.563<br>
Epoch: 14 | Train loss: 0.292 | Train accuracy: 0.886 | Validation loss: 1.491 | Validation accuracy: 0.567<br>
Epoch: 15 | Train loss: 0.264 | Train accuracy: 0.895 | Validation loss: 1.463 | Validation accuracy: 0.586<br>
Epoch: 16 | Train loss: 0.226 | Train accuracy: 0.915 | Validation loss: 1.733 | Validation accuracy: 0.602<br>
Epoch: 17 | Train loss: 0.196 | Train accuracy: 0.927 | Validation loss: 1.537 | Validation accuracy: 0.576<br>
Epoch: 18 | Train loss: 0.195 | Train accuracy: 0.924 | Validation loss: 1.714 | Validation accuracy: 0.567<br>
Epoch: 19 | Train loss: 0.167 | Train accuracy: 0.933 | Validation loss: 1.625 | Validation accuracy: 0.578<br>
Epoch: 20 | Train loss: 0.164 | Train accuracy: 0.938 | Validation loss: 1.844 | Validation accuracy: 0.561<br>
Epoch: 21 | Train loss: 0.160 | Train accuracy: 0.938 | Validation loss: 1.963 | Validation accuracy: 0.557<br>
Epoch: 22 | Train loss: 0.145 | Train accuracy: 0.945 | Validation loss: 1.734 | Validation accuracy: 0.600<br>
Epoch: 23 | Train loss: 0.142 | Train accuracy: 0.943 | Validation loss: 1.819 | Validation accuracy: 0.596<br>
Epoch: 24 | Train loss: 0.144 | Train accuracy: 0.946 | Validation loss: 2.218 | Validation accuracy: 0.586<br>
Epoch: 25 | Train loss: 0.130 | Train accuracy: 0.951 | Validation loss: 1.947 | Validation accuracy: 0.574<br>
Epoch: 26 | Train loss: 0.124 | Train accuracy: 0.953 | Validation loss: 2.069 | Validation accuracy: 0.596<br>
Epoch: 27 | Train loss: 0.125 | Train accuracy: 0.956 | Validation loss: 1.830 | Validation accuracy: 0.578<br>
Epoch: 28 | Train loss: 0.105 | Train accuracy: 0.963 | Validation loss: 2.143 | Validation accuracy: 0.607<br>
Epoch: 29 | Train loss: 0.096 | Train accuracy: 0.964 | Validation loss: 1.934 | Validation accuracy: 0.596<br>
Epoch: 30 | Train loss: 0.102 | Train accuracy: 0.965 | Validation loss: 2.452 | Validation accuracy: 0.605</p>

Soumya_Kundu: <ol>
<li>
<p>Would suggest using a pretrained ResNet-18. Given its overfitting on such a small network, that maybe your best bet. Training from scratch isn’t really beneficial in smaller datasets. You could try mobilenet as well. Going on this, resizing to 224*224 may be the smart move.</p>
</li>
<li>
<p>You could try cross validation. Maybe your split is just difficult.</p>
</li>
</ol>

Mallrat: <p>Thank you very for your suggestions! I really appreciate it!</p>
<ol>
<li>
<p>The thing is, I really want to build something from scratch so that I can learn how to do things properly. You mentioned that my network/datasets are small, do you think that adding more data and/or layers to the model would help?</p>
</li>
<li>
<p>Ah, that’s a great idea! I’ll definitely try some cross-validation. The split I did was admittedly lame, since I simply copied the penultimate and ultimate 10% of the data and made them into a validation and test set.</p>
</li>
</ol>
<p>Does anything else come to mind? Is my training/validation loop ok?</p>
<p>Once again, thanks a lot!</p>

J_Johnson: <ol>
<li>BatchNorm2d after your Conv2d layers may help.</li>
<li>p = 0.5 is pretty high for dropout.</li>
<li>What is the ratio of classes in your train dataset?</li>
</ol>

Mallrat: <p>Thanks a lot! I’ll try the Batchnorm2d and lower the dropout rate, 0.3 should be ok, I assume?</p>
<p>I’m not at home right now, but the ratio is something like 1100-1200 images for two of the four classes and around 1200-1300 images for the other two. Not quite balanced, but still better than nothing.</p>

Soumya_Kundu: <p>You can try building resnet-18 from scratch then <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> My motive in saying this was, training from scratch isnt too hot with lesser data. Something pretrained would do much better!</p>
<p>I suggested a smaller/small network because your model seems to already overfit. Definitely more data would help.</p>
<p>Basic idea of the train val loops seem to be fine.</p>
<p>I too agree batchnorm is a good idea.</p>
<p>If you really want to stick to this, you could try running a hyperparameter grid search. But I still would generate a baseline with a pretrained model to understand what sort of performance you need to improve on (assuming it does improve).</p>

J_Johnson: <p>A couple more considerations/things you could try:</p>
<ol>
<li>SGD optimizer tends to work better  for generalization (which is what you’re looking for here). If compute isn’t an issue, wouldn’t hurt to try. See <a href="https://deepai.org/publication/improving-generalization-performance-by-switching-from-adam-to-sgd" rel="noopener nofollow ugc">here</a> and <a href="https://opt-ml.org/papers/2021/paper53.pdf" rel="noopener nofollow ugc">here</a>.</li>
<li>If the dataset is unbalanced, you could pass in the <code>weight</code> argument into the CE loss function.</li>
<li>Are you using any data augmentation in your dataloader? If not, I suggest putting in a lot of augmentation, in your case, given the small dataset. See <a href="https://pytorch.org/vision/stable/transforms.html" rel="noopener nofollow ugc">here</a> and <a href="https://developer.nvidia.com/tao-toolkit-usecases-whitepaper/3-quick-prototyping-small-dataset" rel="noopener nofollow ugc">here</a>.</li>
</ol>

Mallrat: <p>Right, I see what you mean now. I guess if all else fails, I could look up the architecture of Resnet-18 and try to build it from scratch <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I’ll also try to look for some more data in the meantime. Thanks for your help!</p>

Mallrat: <p>I’ll definitely try the SGD optimizer, I just didn’t know that it was better for generalization.<br>
As I said, the only data augmentation I used was Resize and ToTensor, but I’ll take a look at all the links you sent me and try to implement it. I guess this is a case of “How much regularization do you want? - Yes.” <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

mycul: <p>I’ve seen some references already to data augmentation (which you should definitely be using), but I suggest checking out <a href="https://pypi.org/project/albumentations/" rel="noopener nofollow ugc">Albumentations</a>. Its a library designed to help you perform a vast number of different augmentations.</p>
<p>Also what is the point of this project? To learn how to debug some of the issues of your DL model or are you going for peak performance to solve a particular use case that has to do with planes? If its the ladder, I suggest fine-tuning an existing model. Fine-tuning is probably the most useful thing in all of deep learning for anyone who wants a SoTA model with zero cost.</p>
<p>Welcome to the world of Deep Learning!</p>

Mallrat: <p>Thanks a lot, I’ll look into Albumentations and some other data augmentation techniques!</p>
<p>The purpose of this project is indeed to learn how to build and train a model from scratch. I’ve already learned so much during the process and I feel like I’ll learn a lot more when I implement all the suggestions in this thread. Plus, I really want to unlock the “Build a CNN From Scratch” achievement <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

mycul: <p>I think that’s a great idea. I’d highly suggest just building a regular NN from scratch first if you haven’t already. A CNN which adds in feature maps and a convolutional kernel and then having to update those kernels through backprop is a LOT to do from scratch.</p>