Setting model.eval() makes accuracy much worse

username: llll1l1l1l1l1l1l1l1
cooked: <p>Hi all</p>
<p>I’m new to this forum but have some experience with ml, cnn and pytorch, image vision</p>
<p>I’m trying to use transfer learning to fine-tune a <code>resnet18</code> on my image classification</p>
<p>everything seems fine except one strange point when I’m using <code>model.eval()</code></p>
<p>codes are roughly like:</p>
<pre><code class="lang-auto">
for epoch in range(30):

    resnet.train()
    train_pred=[]
    train_true=[]
    for data in trainloader:

        img, lbl = data['image'].to(device),data['label'].to(device)

        
        optimizer.zero_grad()
        
        logits = resnet(img)
        

        loss=criterion(logits,lbl)
        preds=logits.max(dim=1)[1]

        loss.backward()
        
        optimizer.step()

        
    
        train_pred.append(preds.clone().detach().cpu().numpy())
        train_true.append(lbl.clone().detach().cpu().numpy())
       

    train_true = np.concatenate(train_true)
    train_pred = np.concatenate(train_pred)


    print('epoch {:d}: train mean accuracy: {:.3f}'.format(epoch+1,accuracy_score(train_true,train_pred)),end=' ')
    


    vald_pred=[]
    vald_true=[]

    # !!!
    resnet.eval()  # Setting this makes result even worse

    for data in valdloader:

        img = data['image']
        lbl = data['label']
        

        img, lbl = img.to(device), lbl.to(device)
        logits = resnet(img)
        
        preds=logits.max(dim=1)[1]
        vald_pred.append(preds.clone().detach().cpu().numpy())
        vald_true.append(lbl.clone().detach().cpu().numpy())
        

    vald_true = np.concatenate(vald_true)
    vald_pred = np.concatenate(vald_pred)
   


    print('valid mean accuracy: {:.3f}'.format(accuracy_score(vald_true,vald_pred)))


</code></pre>
<p>I believe it’s suggested to set <code>resnet.eval()</code> in above snippet</p>
<p>However, if I set this, validation accuracy is 38% and it fluctuates a lot(epoch 25-30 seems in same accuracy to epoch 1-5)(giving me a feeling that something is wrong, and I don’t really <em>trained</em> my model…)</p>
<p>if I don’t set it, validation accuracy is from 75% and generally, gradually increasing to 85% at last several epochs<br>
(it also fluctuates, but much more like a normal training procedure)</p>
<p>(in both cases, training accuracy ever increases from 60% to 97%)</p>
<p>Any suggestions on this?</p>
<p>should I disable it in favor of validation accuracy?</p>
<p>big thanks in advance</p>

username: llll1l1l1l1l1l1l1l1
cooked: <p>Also, after inspecting <code>image-pred-true</code> output of my model in test epochs</p>
<p>it almost just outputs one label(first one) for all images<br>
( i have 4 classes, but first one is in roughly 1/3 proportion)</p>

username: llll1l1l1l1l1l1l1l1
cooked: <p>Anyway, after reading this very meaningful and useful <a href="https://discuss.pytorch.org/t/performance-highly-degraded-when-eval-is-activated-in-the-test-phase/3323">post</a></p>
<p>I tried to increase my <code>batch_size</code> from 12 to 128, and it works</p>
<p>this time, validation accuracy with <code>model.eval()</code> looks much more normal</p>
<p>Still leave this post for your reference, as I think it’s quite interesting</p>