Model.eval() accuracy is low

Anto_Skar: <p>Hello,<br>
I am using a pretrained resnet50 to classify some images. My problem is that when I had, in the same training function, both model.train and model.eval, the accuracies where fine (about 65% train and validation accuracies) but when I tried to separate them and use different functions  for each (one for the model.train and one for the model.eval), the validation accuracy dropped to 20% and it remains constant for each epoch. Does someone have an idea of what’s happening?<br>
I quite new to all this and I don’t know why it behaves like that.</p>

eqy: <p>There can be many different causes of this (e.g., inadvertently using different transformations for the validation data vs. the training data). Can you post a code snippet of the evaluation functions?</p>

Anto_Skar: <p>Yes sure.<br>
The transforamations I used are these ones:</p>
<pre><code class="lang-auto">data_transforms = {

    'train': transforms.Compose([

        transforms.RandomResizedCrop(224),

        transforms.RandomHorizontalFlip(),

        transforms.ToTensor()]),

    'val': transforms.Compose([

        transforms.Resize(256),

        transforms.CenterCrop(224),

        transforms.ToTensor()])

}
</code></pre>
<p><strong>The functions:</strong></p>
<pre><code class="lang-auto">def train_model(model, dataloaders, criterion, optimizer, scheduler, batch_size=5, num_epochs=10):#

    since = time.time()

    val_acc_history = []

    best_model_wts = copy.deepcopy(model.state_dict())

    best_acc = 0.0

    #pdb.set_trace()

    for epoch in range(num_epochs):

        print('Epoch {}/{}'.format(epoch, num_epochs - 1))

        print('-' * 10)

        # Each epoch has a training and validation phase

        for phase in ['train']:#, 'val']:

            if phase == 'train':

                model.train()  # Set model to training mode

            running_loss = 0.0

            running_corrects = 0

            average_precis_train = 0.001

            average_precis_train_per_class = 0.001

            loss_values = []

            gr_truth_array = np.array([]) #convet to int dtype

            preds_array = np.array([])

            gr_truth_array = gr_truth_array.astype(int)

            preds_array = preds_array.astype(int) 

            average_precision_array = np.array([]).astype(float)


            print('Iterating over data:')

            for batch_idx, (inputs, labels) in enumerate(dataloaders[phase]):

                inputs = inputs.to(device)

                labels = labels.to(device).float()

                gt_data = labels

                gt_data = gt_data.to(device)

                gt_data = gt_data.cpu().data.numpy()

                #average_precision_array = []

                # zero the parameter gradients

                optimizer.zero_grad()

                # forward

                # track history if only in train

                #pdb.set_trace()

                if phase == 'train':

                  with torch.set_grad_enabled(phase == 'train'):

                      outputs = model(inputs)

                      outputs = outputs.cpu()#.data.numpy()

                      preds = outputs.cpu().data.numpy()

                      preds = np.round(preds) #set a condition for binary

                      preds_int = preds.astype(int)

                      gt_data_np = np.round(gt_data)

                      gt_data_int = gt_data_np.astype(int)

                      gt_data = torch.from_numpy(gt_data_np)

                      loss = criterion(outputs, gt_data)

                      gr_truth_array = np.append(gr_truth_array, gt_data_int)

                      preds_array = np.append(preds_array ,preds_int)

                    # backward + optimize only if in training phase

                      if phase == 'train':

                          loss.backward()

                          optimizer.step()

                # statistics

                  gr_truth_array = np.reshape(gr_truth_array, (-1, 40))

                  preds_array = np.reshape(preds_array, (-1, 40))

                  running_loss += loss.item() * inputs.size(0)

                  running_corrects += f1_score(gt_data, preds, average="samples")

                

            if phase == 'train':

                scheduler.step()

                average_precis_train += average_precision_score(gr_truth_array, preds_array, average= "macro")

                average_precis_train_per_class += average_precision_score(gr_truth_array, preds_array, average=None)

                average_precision_array = np.append(average_precision_array, average_precis_train_per_class)

                #pdb.set_trace()

                av_precis_array = [j for i in zip(average_precision_array, attributes) for j in i]

                av_precis_array = np.array(av_precis_array)

                print("Average precision Training:", average_precis_train)      

                print("Average precision per Class Training:", av_precis_array)

            #pdb.set_trace()   

            epoch_loss = running_loss / len(dataloaders[phase].dataset)

            epoch_acc = running_corrects / len(dataloaders[phase].dataset) #running_corrects.float()

            epoch_acc = np.round(epoch_acc, decimals=4)

             

            print('{} Loss: {:.4f}'.format(phase, epoch_loss))

            print("Acc:", epoch_acc)

 

    time_elapsed = time.time() - since

    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))

    model.load_state_dict(best_model_wts)

    return model, val_acc_history
</code></pre>
<p>The evaluation is almost the same but the model is set to model.eval and I use <code>with torch.no_grad():</code> instead of set_grad_enabled</p>

eqy: <p>I see the condition for <code>model.train()</code> statement in the code but it looks like <code>model.eval()</code> doesn’t have a corresponding branch?</p>

Anto_Skar: <p>Ok can you explain a bit more? Is this what is causing this?</p>

eqy: <p>I’m not sure this is the issue yet, but I don’t see <code>model.eval()</code> anywhere in the code you posted, just <code>model.train()</code>.</p>

eqy: <p>Have you inspected the outputs of the model to see if they behave strangely during validation? For example, are they stuck at the same output (or the same class) for every example? Does the validation accuracy change at all between epochs?</p>

Anto_Skar: <p>I will and I will let you know.</p>

Anto_Skar: <p>The accuracy stays the same in every epoch</p>

eqy: <p>What happens when you remove the <code>model.load_state_dict(best_model_wts)</code>? It looks like the best model is never updated so this may just return the same model every iteration.</p>

Anto_Skar: <p>took it out but didn’t work. Nothing changes <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
The accuracy stays the same again</p>

eqy: <p>Ok, then can you verify the data is changing along with the model predictions during validation? Or are the predictions the same regardless of the input?</p>

Anto_Skar: <p>It seems that the outputs change with every iteration, so I guess there is no issue there</p>

eqy: <p>You might want to also add a sanity check that the model parameters are changing between validation epochs.</p>

Anto_Skar: <p>Can you tell me how to do that? Maybe give me an example or something?</p>

eqy: <p>This code gives an example of how to count the number of parameters in the model.<br>
<a href="https://discuss.pytorch.org/t/how-do-i-check-the-number-of-parameters-of-a-model/4325/9">How do I check the number of parameters of a model? - PyTorch Forums</a><br>
If you want to check that the parameters are changing, you can try printing the sum of the parameters rather than the count and see if this is changing between training epochs.</p>

Anto_Skar: <p>Thank you very much. I’ll try it tomorrow <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

Anto_Skar: <p>Hello again,<br>
So in this line of code<br>
<code>def count_parameters(model):     return sum(p.numel() for p in model.parameters() if p.requires_grad)</code><br>
since it returns the sum of the parameters, I should only take out the <code>numel()</code> in order to get the sum right?</p>

eqy: <p>Something like that. You might need to do a second sum if you end up with just a list of summed parameters for each layer (or you can just compare them directly if the ordering is the same).</p>

Anto_Skar: <p>Ok, because I got this error here</p>
<pre><code class="lang-auto">----&gt; return sum(p for p in model.parameters() if p.requires_grad)

RuntimeError: The size of tensor a (7) must match the size of tensor b (64) at non-singleton dimension 3
</code></pre>