Confusion matrix for valiation data

username: Rosa
cooked: <p>Hi, I want to plot a confusion matrix for validation set. The code for my training is given below. Which is taken from pytorch example. During training phase it also computes and show the validation loss and accuracy as well as training loss and accuracy. But I also want to see the confusion matrix for validation set. How can I do it ?</p>
<pre><code class="lang-auto">def train_model(model, criterion, optimizer, scheduler, num_epochs):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch +1, num_epochs))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'valid']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                    

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            #For graph generation
            if phase == "train":
                train_loss.append(running_loss/dataset_sizes[phase])
                train_acc.append(running_corrects.double() / dataset_sizes[phase])
                epoch_counter_train.append(epoch)
            if phase == "valid":
                val_loss.append(running_loss/ dataset_sizes[phase])
                val_acc.append(running_corrects.double() / dataset_sizes[phase])
                epoch_counter_val.append(epoch)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            #for printing        
            if phase == "train":    
                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]
            if phase == "valid":    
                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]
            
            
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the best model
            if phase == 'valid' and epoch_acc &gt; best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    #save the  best model
    torch.save(model.load_state_dict(best_model_wts), savePath+'bestModel.pth')
    return model
</code></pre>
<p>Now , I know how to plot confusion matrix for my test set too, like</p>
<pre><code class="lang-auto">with torch.no_grad():
    for i, (inputs, labels) in enumerate(dataloaders['test']):
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = model_ft(inputs)
        _, predicted = torch.max(outputs, 1)
        confusion_matrix.add(predicted, labels)
</code></pre>
<p>But I am confused how to add the  code snippet:<br>
<code>confusion_matrix.add(predicted, labels)</code><br>
add where to incorporate the line in my training function. If I add it in the training function after the line</p>
<pre><code class="lang-auto">if phase == "valid":  
   confusion_matrix.add(preds, labels)
</code></pre>
<p>It seems, the result is not correct if I plot it.<br>
So, Can you please suggest how to get the confusion matrix on validation set? Should I have to do it in my training function where I am both training and validating or just like the test set, outside training function, looping through validation set again ?</p>

username: AnBucquet
cooked: <p>Hi <a class="mention" href="/u/rosa">@Rosa</a>,<br>
I think you can do it in your training function. This snippet should work:</p>
<pre><code class="lang-auto">confusion_matrix = torch.zeros(nb_labels, nb_labels)
for inputs, labels in dataloaders[phase]:
       inputs = inputs.to(device)
       labels = labels.to(device)
       optimizer.zero_grad()

       with torch.set_grad_enabled(phase == 'train'):
              outputs = model(inputs)
              _, preds = torch.max(outputs, 1)
              loss = criterion(outputs, labels)
              ....

              if phase == 'valid':
                      for l, p in zip(labels.view(-1), preds.view(-1)):
                      confusion_matrix[l.long(), p.long()] += 1
print(confusion_matrix)
</code></pre>

username: krishna511
cooked: <p>I am using this function confusion matrix. and getting this<br>
Confusion Matirx<br>
[[16  0  1  0  8  0  0]<br>
[ 0 15  0  0  1  0  0]<br>
[ 1  1  6  0  1  0  0]<br>
[ 1  0  0  9  2  0  1]<br>
[ 3  0  0  0 11  0  0]<br>
[ 0  3  0  0  0  9  0]<br>
[ 0  0  0  0  0  1 14]]</p>
<p>and it says accuracy score is  0.76923. It has 7 classes, Total number of samples are 104 So the overall accuracy is</p>
<p>Is it called the Weighted accuracy or unweighted accuracy?<br>
And for other , we need to do the manual calculation?</p>

username: krishna511
cooked: <p>Got it ,It is weighted , ie overall accuracy. In case of unweighted accuracy , equal weightage is given to each class.</p>