I can't increase accuracy

username: Oguz_Yesil
cooked: <p>i am trying image classification with <a href="https://www.kaggle.com/alxmamaev/flowers-recognition" rel="noopener nofollow ugc">this dataset</a> and i resize them (64, 64). but my test accuracy is unstable and so low. my model:</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5)
        self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5)

        self.norm1 = nn.InstanceNorm2d(64)

        self.maxpool1 = nn.MaxPool2d(3)

        self.relu1 = nn.LeakyReLU()

        self.conv4 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5)
        self.conv5 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5)
        self.conv6 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=5)        

        self.norm2 = nn.InstanceNorm2d(256)
        
        self.maxpool2 = nn.MaxPool2d(3)
        
        self.relu2 = nn.LeakyReLU()

        self.fc1 = nn.Linear(256, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 60)
        self.fc4 = nn.Linear(60, 50)
        self.fc5 = nn.Linear(50, 40)
        self.fc6 = nn.Linear(40, 30)
        self.fc7 = nn.Linear(30, 20)
        self.fc8 = nn.Linear(20, 10)
        self.fc9 = nn.Linear(10, 5)

        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        out = self.conv1(x)        
        out = self.conv2(out)        
        out = self.conv3(out)

        out = self.norm1(out)
        
        out = self.maxpool1(out)

        out = self.relu1(out)

        out = self.conv4(out)        
        out = self.conv5(out)        
        out = self.conv6(out)

        out = self.norm2(out)
        
        out = self.maxpool2(out)

        out = self.relu2(out)
        
        # flatten
        out = out.view(out.size(0), -1)
        
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fc3(out)
        out = self.fc4(out)
        out = self.fc5(out)
        out = self.fc6(out)
        out = self.fc7(out)
        out = self.fc8(out)
        out = self.sigmoid(self.fc9(out))

        return out
</code></pre>
<p>my train func:</p>
<pre><code class="lang-auto">batch_size = 5
n_iters = 5000
num_epochs = n_iters / (len(train) / batch_size)
num_epochs = int(num_epochs)

trainDatas = DataLoader(train, batch_size=batch_size, shuffle=True)
testDatas = DataLoader(test, batch_size=batch_size, shuffle=True)

# Create model
model = CNNModel()

 # Cross Entropy Loss
error = nn.CrossEntropyLoss()

learning_rate = 0.1
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

count = 0
loss_list = []
iteration_list = []
accuracy_list = []
best_correct = 0.0

model = model.float()

for epoch in range(num_epochs):
        print('epoch: {}/{}'.format(epoch+1, num_epochs))

        for i, (train, label) in enumerate(trainDatas):
            # train mode
            model.train()

            train = Variable(train)
            label = Variable(label)

            # Clear gradients
            optimizer.zero_grad()

            # forward propagation
            outputs = model(train.float())

            _, label = torch.max(label.data, 1)

            loss = error(outputs, label.long())

            loss.backward()
            optimizer.step()

            count += 1

            if count % 50 == 0:
                # Calculate Accuracy
                correct = 0
                total = 0

                # predict test dataset
                for j, (test, testLabel) in enumerate(testDatas):
                    # predict mode
                    model.eval()

                    test = Variable(test)
                    testLabel = Variable(testLabel)

                    # forward propagation
                    outputs = model(test.float())

                    _, pred = torch.max(outputs, 1)
                    _, testLabel = torch.max(testLabel.data, 1)
                    
                    correct += torch.sum(pred == testLabel)

                    total = len(testDatas)

                accuracy = 100 * (correct / (float(j)*batch_size))

                if correct &gt; best_correct:
                    torch.save(model, 'best.model')
                    best_correct = correct

                # store loss and iteration
                loss_list.append(loss.data)
                iteration_list.append(count)
                accuracy_list.append(accuracy)

            if count % 100 == 0:
                print('Iteration: {}  Loss: {}  Accuracy: {}%({}/{})'.format(
                    count, loss.data, int(accuracy), correct, total*batch_size))
</code></pre>
<p>and a train example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40e9fb56c603c89f2b9ae4964a5a2e93b1bea1c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/40e9fb56c603c89f2b9ae4964a5a2e93b1bea1c6" title="resim_2022-02-02_171945"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/0/40e9fb56c603c89f2b9ae4964a5a2e93b1bea1c6.png" alt="resim_2022-02-02_171945" data-base62-sha1="9gfRqIaJSyyXhzJPMt7LfYCDCu2" width="453" height="500" data-dominant-color="EBEBFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resim_2022-02-02_171945</span><span class="informations">709×782 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: my3bikaht
cooked: <p>I can immediately see two things here:</p>
<ol>
<li>Both conv and fc layers are just a linear functions. Without activations in between any combination of linear functions is still a linear function.</li>
<li>Your learning rate is too big, try 1e-3</li>
</ol>
<p>Also, sequence of fully connected layers in the bottom that long will hardly help in your case. You can replace it with single fc without losing any accuracy.</p>

username: Oguz_Yesil
cooked: <p>so should I put an activation between all conv layers? i’m sorry if it’s a stupid question, I’m trying to find out.</p>

username: my3bikaht
cooked: <p>I would start with all conv having activations, yes. It’s not an axiom (for example in recent paper on convnext model authors left just one activation per residual block), but it works.</p>

username: Oguz_Yesil
cooked: <p>I will try what you said. thank you for the replies</p>

username: Oguz_Yesil
cooked: <p>I did what you said but nothing much has changed.</p>
<p>model:</p>
<pre><code class="lang-auto">class CNNModel(nn.Module):
    def __init__(self):
        super(CNNModel, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5)
        self.conv3 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5)

        self.norm1 = nn.InstanceNorm2d(64)

        self.maxpool1 = nn.MaxPool2d(3)

        self.relu1 = nn.LeakyReLU()

        self.conv4 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=5)
        self.conv5 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=5)
        self.conv6 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=5)        

        self.norm2 = nn.InstanceNorm2d(256)
        
        self.maxpool2 = nn.MaxPool2d(3)
        
        self.relu2 = nn.LeakyReLU()

        self.fc1 = nn.Linear(256, 5)

        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        out = self.conv1(x)
        out = self.relu1(out)
        out = self.conv2(out)
        out = self.relu1(out)
        out = self.conv3(out)
        out = self.relu1(out)

        out = self.norm1(out)
        
        out = self.maxpool1(out)

        #out = self.relu1(out)

        out = self.conv4(out)
        out = self.relu1(out)
        out = self.conv5(out)
        out = self.relu1(out)
        out = self.conv6(out)
        out = self.relu1(out)

        out = self.norm2(out)
        
        out = self.maxpool2(out)

        #out = self.relu2(out)
        
        # flatten
        out = out.view(out.size(0), -1)
        
        out = self.sigmoid(self.fc1(out))

        return out
</code></pre>
<p>and i change learning rate with 1e-3.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7aa8260eead615872fd0bc769ad3a19aabdad81.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7aa8260eead615872fd0bc769ad3a19aabdad81" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7aa8260eead615872fd0bc769ad3a19aabdad81.png" alt="image" data-base62-sha1="qcMGSnccCn0ta4t0ELRVT5MyyL7" width="690" height="365" data-dominant-color="EFEFFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">729×386 8.55 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i think i think it’s not going well</p>

username: my3bikaht
cooked: <p>try to drop sigmoid in the end</p>
<p>and you can try this smaller model:<br>
conv(3,64, kernel_size=7, stride=1, padding=3, bias=False) &gt; batchnorm(64) &gt; relu &gt; avgpool(2,2) &gt; conv(64,128,kernel_size=5, stride=2, padding=2, bias=False) &gt; batchnorm(128) &gt; relu &gt; conv(128,256,kernel_size=5, stride=2, padding=2, stride=2, bias=False) &gt; batchnorm(256) &gt; relu &gt; adaptiveavgpool(1) &gt; fc(256,5)</p>

username: Oguz_Yesil
cooked: <p>yes this model better then my model. it especially reduces the losses considerably but accuracy still very low <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> .</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef1a849eba4bdf8a71ddcf209d3fab170837c075.png" data-download-href="https://discuss.pytorch.org/uploads/default/ef1a849eba4bdf8a71ddcf209d3fab170837c075" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/ef1a849eba4bdf8a71ddcf209d3fab170837c075.png" alt="image" data-base62-sha1="y7cVFgIOV3N2mzchZXMNKQCI5Vj" width="690" height="325" data-dominant-color="EDEDFF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">741×350 9.04 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why bias are false?<br>
I think that if I increase the learning speed a little bit, the accuracy rate will increase.</p>

username: my3bikaht
cooked: <p>With regularization done by batchnorm you don’t need bias.<br>
Increasing learning rate can speed up training, but with lr too big you’ll keep overshooting the solution.</p>
<p>I think you need to check on labels, there is a chance of mix-up. With 5 classes, probability of random guess is 20%, and your results are very close to this probability.</p>
<p>Also you don’t need this: <code>train = Variable(train)</code>, Variable API has been deprecated long time ago</p>
<pre><code class="lang-auto">_, label = torch.max(label.data, 1)
</code></pre>
<p>calling torch.max is not neccessary anymore, CrossEntropyLoss now can work with multi-class labels (as long as you update torch to at least 1.10)</p>
<pre><code class="lang-auto">outputs = model(test.float())
</code></pre>
<p>when in model.eval() mode, add with torch.no_grad(): before calling model, it will speed up calculation</p>

username: Oguz_Yesil
cooked: <p>oh, thanks for everything</p>