CNN low accuracy results

LukaMlink: <p>Hi, I am working on a classification CNN model with CIFAR10 dataset. My net looks like this:</p>
<p>self.conv1 = Conv2d(in_channels=num_channels, out_channels=16, kernel_size=(5, 5))<br>
self.bn1 = nn.BatchNorm2d(num_features=16)  # Corrected the argument here<br>
self.relu1 = ReLU()<br>
self.maxpool1 = MaxPool2d(kernel_size=(2, 2), stride=(2, 2))</p>
<p>self.conv2 = Conv2d(in_channels=16, out_channels=32, kernel_size=(5, 5))<br>
self.bn2 = nn.BatchNorm2d(num_features=32)  # Ensure this matches the output channels of conv2<br>
self.relu2 = ReLU()<br>
self.maxpool2 = MaxPool2d(kernel_size=(2, 2), stride=(2, 2))</p>
<p>self.fc1 = Linear(in_features=32<em>5</em>5, out_features=120) # fully connected layer<br>
self.relu3 = ReLU()<br>
self.fc2 = Linear(in_features=120, out_features=84) # fully connected layer 2<br>
self.relu4 = ReLU()<br>
self.fc3 = Linear(in_features=84, out_features=num_classes) # fully connected layer 3<br>
self.logSoftmax = LogSoftmax(dim=1) # softmax activation function</p>
<p>I have 10 epochs, the learning rate is 0.001 with 32 batch size and ADAM optimizer.</p>
<p>I get an accuracy of around 50% and I am not really sure, what should my next steps be to improve the models accuracy.</p>
<p>edit: I added batch normalization and accuracy went to 70%, how can I improve acc even more?</p>
<p>Thanks</p>

UMAR_MASUD: <p>Hi Luka, I believe some of the ways you can improve your accuracy results are:</p>
<ol>
<li>Increase the complexity of the network. Add more CNN layers with more filters to your custom model.</li>
<li>Try using residual connections or attention mechanisms.</li>
<li>Use Transfer learning approaches where you take a large pre-trained model and then fine-tune it using your data.</li>
<li>Apply data augmentation techniques to improve generalisation and performance.</li>
<li>Optimising hyperparameters like the number of epochs, learning rate, batch size, optimiser, etc can also help.</li>
</ol>
<p>Some SOTA methods for CIFAR10 are given here - <a href="https://myrtle.ai/learn/how-to-train-your-resnet/" rel="noopener nofollow ugc">https://myrtle.ai/learn/how-to-train-your-resnet/</a> and this bag of tricks - <a href="https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/" rel="noopener nofollow ugc">https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tricks/</a><br>
I hope this is useful!</p>

iman.abduljaleel: <p>could you give me an example code about attention mechanisms and how I could use it in CNN model</p>

Ayush_Aditya: <p>i will suggest you this architecture, try this</p>
<pre><code>class pre(nn.Module):

    def __init__(self, num_classes=10, dropout_p=0.5):

        super(pre, self).__init__()
        self.conv1 = nn.Conv2d(3, 128, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(128, 128, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(128, 128, kernel_size=3, padding=1)
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)

        self.conv4 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv5 = nn.Conv2d(256, 256, kernel_size=3, padding=1)
        self.conv6 = nn.Conv2d(256, 256, kernel_size=3, padding=1)
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)

        self.conv7 = nn.Conv2d(256, 512, kernel_size=3, padding=0)
        self.conv8 = nn.Conv2d(512, 256, kernel_size=1, padding=0)
        self.conv9 = nn.Conv2d(256, 128, kernel_size=1, padding=0)
        self.avg_pool = nn.AvgPool2d(kernel_size=6)
    
        self.fc = nn.Linear(128, num_classes)

    def forward(self, x):
        x = F.leaky_relu(self.conv1(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv2(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv3(x), negative_slope=0.1)

        x = self.pool1(x)

        x = F.leaky_relu(self.conv4(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv5(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv6(x), negative_slope=0.1)
    
        x = self.pool2(x)

        x = F.leaky_relu(self.conv7(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv8(x), negative_slope=0.1)
        x = F.leaky_relu(self.conv9(x), negative_slope=0.1)
        x = self.avg_pool(x)
        x = x.view(x.size(0), -1)
        x = torch.transpose(x, 0, 1)
        x = self.fc(x)
    
        return x
</code></pre>

iman.abduljaleel: <p>thank you for your code but could you explain to me what this code used and how I could choose the perfect CNN layers for my application? I am new in deep learning work</p>

iman.abduljaleel: <p>if you could hep me how i could add attention mechanism to CNN layer</p>

Ayush_Aditya: <p>Actually there is no formula find out which architecture will work the best, it just comes with practice. Just keep few things in mind while trying out few things like<br>
if the accuracy is low then try out different things like increasing/decreasing learning rate , increase model complexity by adding more layer and for reference try using research paper, kaggle to check which architecture works the best.<br>
if training accuracy is high that mean over fitting so reduce the complexity…</p>

iman.abduljaleel: <p>thank you so much sir</p>

UMAR_MASUD: <p>Hi, you can refer to this repo to get a list of plug-and-play attention mechanisms you can add to any CNN-based model. It is just a layer that you would insert between the CNN layers. Some prominent examples are SE-Net, ECA-Net, CBAM, etc.</p>
<p>Repo: <a href="https://github.com/pprp/awesome-attention-mechanism-in-cv" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pprp/awesome-attention-mechanism-in-cv: Awesome List of Attention Modules and Plug&amp;Play Modules in Computer Vision</a></p>

iman.abduljaleel: <p>thank you so much sir</p>