Calculate train accuracy of the model in segmentation task

Neda: <p>I think I don’t have a good understanding of train accuracy. This is the snippet for train the model and calculates the loss and train accuracy for segmentation task.</p>
<pre><code class="lang-auto"> for epoch in range(2):  # loop over the dataset multiple times
        
        running_loss = 0
        total_train = 0
        correct_train = 0
        for i, data in enumerate(train_loader, 0):
            # get the inputs
            t_image, mask = data
            t_image, mask = Variable(t_image.to(device)), Variable(mask.to(device))
        

            # zeroes the gradient buffers of all parameters
            optimizer.zero_grad()
            # forward + backward + optimize
            outputs = model(t_image) # forward
            loss = criterion(outputs, mask) # calculate the loss
            loss.backward() # back propagation
            optimizer.step() # update gradients
            running_loss += loss.item()
            
            # accuracy
            _, predicted = torch.max(outputs.data, 1)
            total_train += mask.size(0)
            correct_train += predicted.eq(mask.data).sum().item()
            train_accuracy = 100 * correct_train / total_train
           #avg_accuracy = train_accuracy / len(train_loader)                                     
            
            print('Epoch {}, train Loss: {:.3f}'.format(epoch, loss.item()), "Training Accuracy: %d %%" % (train_accuracy))
</code></pre>
<p>I am getting a number like below for the training accuracy which I assume is the amount of pixel which corresponds with the ground truth (mask). is that correct?</p>
<pre><code class="lang-auto">Epoch 0, train Loss: 0.721 Training Accuracy: 500300 %
Epoch 0, train Loss: 0.707 Training Accuracy: 676000 %
</code></pre>
<p>How can I calculate the accuracy of the network that shows the performance of the model? (ex. <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py" rel="nofollow noopener">The accuracy of the network on the 10000 test images: 54 % in pytorch tutorial but this is for the classification task</a> )</p>

ptrblck: <p>You are currently summing all correctly predicted pixels and divide it by the batch size. To get a valid accuracy between 0 and 100% you should divide <code>correct_train</code> by the number of pixels in your batch.<br>
Try to calculate <code>total_train</code> as <code>total_train += mask.nelement()</code>.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> yes it works. Thanks a lot</p>

singhvishal0209: <p>I this piece of code of yours</p>
<pre><code class="lang-auto"># accuracy
            _, predicted = torch.max(outputs.data, 1)
            total_train += mask.size(0)
            correct_train += predicted.eq(mask.data).sum().item()`
</code></pre>
<p>what are you trying to do in first line with torch.max especially why 1 and not 0. In 3rd line with predicted.eq()</p>

Neda: <p><a class="mention" href="/u/singhvishal0209">@singhvishal0209</a> sorry for late reply.</p>
<p>first line: return the indices of max values along rows in <code>softmax</code> probability output (<code>torch.max</code> returns a tuple containing the maximum value and the index of the maximum value within the tensor. Since the index in our case represents the classified category itself, we will only take that ignoring the actual probability).</p>
<p>second line: number of pixel in the batch</p>
<p>third line: By summing the output of the <code>.eq()</code> function, we get a count of the number of times the neural network has produced a correct output, and we take an accumulating sum of these correct predictions so that we can determine the accuracy of the network.</p>

Shariq_Ali: <p>I am getting 100% accuracy. I don’t know how to correct it.  I am using the code of following link on my own dataset. The dataset set 400 training images and 120 validation images. Please help me.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/meetps/pytorch-semseg">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/meetps/pytorch-semseg" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/1af620d3e8033104368e767707ed683312b80caad803660f5b17106d61f1a34f/meetps/pytorch-semseg" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/meetps/pytorch-semseg" target="_blank" rel="noopener nofollow ugc">GitHub - meetps/pytorch-semseg: Semantic Segmentation Architectures...</a></h3>

  <p>Semantic Segmentation Architectures Implemented in PyTorch - GitHub - meetps/pytorch-semseg: Semantic Segmentation Architectures Implemented in PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>dataloader code</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dd4fe7da0071e35aa122d7cc4370113c496d470.png" data-download-href="https://discuss.pytorch.org/uploads/default/4dd4fe7da0071e35aa122d7cc4370113c496d470" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd4fe7da0071e35aa122d7cc4370113c496d470_2_422x500.png" alt="image" data-base62-sha1="b6x6nT4rhLPZKKfOmXJ0H7GnYfm" width="422" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd4fe7da0071e35aa122d7cc4370113c496d470_2_422x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4dd4fe7da0071e35aa122d7cc4370113c496d470_2_633x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4dd4fe7da0071e35aa122d7cc4370113c496d470.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">663×784 48.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24c10edee13dea490ca2530bd8a920f9a53270c2.png" data-download-href="https://discuss.pytorch.org/uploads/default/24c10edee13dea490ca2530bd8a920f9a53270c2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24c10edee13dea490ca2530bd8a920f9a53270c2_2_690x436.png" alt="image" data-base62-sha1="5f8OsWUKqlHUWYztg733cgst55w" width="690" height="436" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24c10edee13dea490ca2530bd8a920f9a53270c2_2_690x436.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24c10edee13dea490ca2530bd8a920f9a53270c2_2_1035x654.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/4/24c10edee13dea490ca2530bd8a920f9a53270c2.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1230×778 64.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08e8669c33d8d0cb6f0fc9b3ffc09e3c4a957437.png" data-download-href="https://discuss.pytorch.org/uploads/default/08e8669c33d8d0cb6f0fc9b3ffc09e3c4a957437" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08e8669c33d8d0cb6f0fc9b3ffc09e3c4a957437.png" alt="image" data-base62-sha1="1gNJE3iL7n8zVFxWITJh9EZTOaH" width="690" height="375" data-dominant-color="FBFAF7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">887×483 20 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af.png" data-download-href="https://discuss.pytorch.org/uploads/default/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af_2_634x500.png" alt="image" data-base62-sha1="toVZ5WF48Vn5sp4LzXdyeCziKiX" width="634" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af_2_634x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af_2_951x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce10a0e8b1064e3b2f6a01559a258dfb716dc7af.png 2x" data-dominant-color="F8F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1002×790 53.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de91632eb85c586858a55ea2cae5f04939ebf158.png" data-download-href="https://discuss.pytorch.org/uploads/default/de91632eb85c586858a55ea2cae5f04939ebf158" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de91632eb85c586858a55ea2cae5f04939ebf158.png" alt="image" data-base62-sha1="vKVulVwEG4WRGRgViQ45796FxDG" width="605" height="500" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">939×776 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Are you concerned about possible errors in the calculation of the accuracy or are you concerned you might have another error in the code, e.g. data leakage?</p>

Shariq_Ali: <p>Yes i’m concerned about possible errors in the calculation of the accuracy. On each model i’m getting 100% accuracy after some epochs i think there some over fitting or some issue with accuracy calculation but i don’t know how to correct.</p>

ptrblck: <p>Could you post your code to calculate the accuracy by wrapping it in three backticks ``` so that we could have a look?</p>

Shariq_Ali: <p>I’m using the same code as in the link.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/meetshah1995/pytorch-semseg/blob/master/ptsemseg/metrics.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/meetshah1995/pytorch-semseg/blob/master/ptsemseg/metrics.py" target="_blank" rel="nofollow noopener">meetshah1995/pytorch-semseg/blob/master/ptsemseg/metrics.py</a></h4>
<pre><code class="lang-py"># Adapted from score written by wkentaro
# https://github.com/wkentaro/pytorch-fcn/blob/master/torchfcn/utils.py

import numpy as np


class runningScore(object):
    def __init__(self, n_classes):
        self.n_classes = n_classes
        self.confusion_matrix = np.zeros((n_classes, n_classes))

    def _fast_hist(self, label_true, label_pred, n_class):
        mask = (label_true &gt;= 0) &amp; (label_true &lt; n_class)
        hist = np.bincount(
            n_class * label_true[mask].astype(int) + label_pred[mask], minlength=n_class ** 2
        ).reshape(n_class, n_class)
        return hist

    def update(self, label_trues, label_preds):
        for lt, lp in zip(label_trues, label_preds):
</code></pre>

  This file has been truncated. <a href="https://github.com/meetshah1995/pytorch-semseg/blob/master/ptsemseg/metrics.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>// Code To Calculate Accuarcy.</p>
<p>import numpy as np</p>
<p>class runningScore(object):<br>
def <strong>init</strong>(self, n_classes):<br>
self.n_classes = n_classes<br>
self.confusion_matrix = np.zeros((n_classes, n_classes))</p>
<pre><code>def _fast_hist(self, label_true, label_pred, n_class):
    mask = (label_true &gt;= 0) &amp; (label_true &lt; n_class)
    hist = np.bincount(
        n_class * label_true[mask].astype(int) + label_pred[mask], minlength=n_class ** 2
    ).reshape(n_class, n_class)
    return hist

def update(self, label_trues, label_preds):
    for lt, lp in zip(label_trues, label_preds):
        self.confusion_matrix += self._fast_hist(lt.flatten(), lp.flatten(), self.n_classes)

def get_scores(self):
    """Returns accuracy score evaluation result.
        - overall accuracy
        - mean accuracy
        - mean IU
        - fwavacc
    """
    hist = self.confusion_matrix
    acc = np.diag(hist).sum() / hist.sum()
    acc_cls = np.diag(hist) / hist.sum(axis=1)
    acc_cls = np.nanmean(acc_cls)
    iu = np.diag(hist) / (hist.sum(axis=1) + hist.sum(axis=0) - np.diag(hist))
    mean_iu = np.nanmean(iu)
    freq = hist.sum(axis=1) / hist.sum()
    fwavacc = (freq[freq &gt; 0] * iu[freq &gt; 0]).sum()
    cls_iu = dict(zip(range(self.n_classes), iu))

    return (
        {
            "Overall Acc: \t": acc,
            "Mean Acc : \t": acc_cls,
            "FreqW Acc : \t": fwavacc,
            "Mean IoU : \t": mean_iu,
        },
        cls_iu,
    )

def reset(self):
    self.confusion_matrix = np.zeros((self.n_classes, self.n_classes))
</code></pre>
<p>class averageMeter(object):<br>
“”“Computes and stores the average and current value”""</p>
<pre><code>def __init__(self):
    self.reset()

def reset(self):
    self.val = 0
    self.avg = 0
    self.sum = 0
    self.count = 0

def update(self, val, n=1):
    self.val = val
    self.sum += val * n
    self.count += n
    self.avg = self.sum / self.count</code></pre>

Shariq_Ali: <p>Hello can you tell me how should i calculate accuracy against MSELoss function ?</p>

ptrblck: <p>How would you like to define your accuracy?<br>
If you are dealing with a multi-class classification use case, you could compare the predictions to the targets via:</p>
<pre><code class="lang-python">output = torch.randn(10, 10)
target = torch.randint(0, 10, (10,))
preds = torch.argmax(output, 1)
acc = preds.eq(target).float().mean()
</code></pre>
<p>Usually you wouldn’t use <code>nn.MSELoss</code> for a classification use case, so we would need some information about your use case.</p>

Shariq_Ali: <p>I have a dataset of wheat Crop n which i have two clases Ear and Spikelet. when i use crossEntropyLoss function i will get the following error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/3578f6596e50a59e2bb6d375b78a4fea36a3b8da.png" data-download-href="https://discuss.pytorch.org/uploads/default/3578f6596e50a59e2bb6d375b78a4fea36a3b8da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3578f6596e50a59e2bb6d375b78a4fea36a3b8da_2_690x180.png" alt="image" data-base62-sha1="7D2sV73I9j6a1lc2sKyvojxAuKK" width="690" height="180" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3578f6596e50a59e2bb6d375b78a4fea36a3b8da_2_690x180.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3578f6596e50a59e2bb6d375b78a4fea36a3b8da_2_1035x270.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/5/3578f6596e50a59e2bb6d375b78a4fea36a3b8da_2_1380x360.png 2x" data-dominant-color="F8F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1600×419 21.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>when i change<br>
target_var = Variable(target.float())<br>
to<br>
target_var = Variable(target.long())</p>
<p>i will get the following error</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f58908ae4a07bc324163a64611533b231201fdf.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f58908ae4a07bc324163a64611533b231201fdf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f58908ae4a07bc324163a64611533b231201fdf.png" alt="image" data-base62-sha1="fT0Knth2zyPQdKYrzfxY0ruvnsP" width="690" height="183" data-dominant-color="F8F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1547×412 21.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Shariq_Ali: <p>when i tried to use your code. i will get the following error.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/4830d8c299b765f3070d0bbce9faca8d0a565278.png" data-download-href="https://discuss.pytorch.org/uploads/default/4830d8c299b765f3070d0bbce9faca8d0a565278" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/4830d8c299b765f3070d0bbce9faca8d0a565278.png" alt="image" data-base62-sha1="aiD1X1Yw5ewlysRzE8x8mngZo1y" width="690" height="137" data-dominant-color="F8F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1058×211 9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sanjayvarma11: <p>hello can you mention your input size if you are having the problem right now.it is a like an 6 months after completing this thread</p>

Naruto: <p>Hey, I am computing the accuracy in a same way. As</p>
<pre><code class="lang-auto"> for epoch in range(epochs):
        net.train()

        epoch_loss = 0
        total_train = 0
        correct_train = 0
     
        with tqdm(total=n_train, desc=f'Epoch {epoch + 1}/{epochs}', unit='img') as pbar:
            for batch in train_loader:
                #continue
                imgs = batch['image']
                true_masks = batch['mask']
                n_batch = np.ceil(n_train/batch_size)
                assert imgs.shape[1] == net.n_channels, \
                    f'Network has been defined with {net.n_channels} input channels, ' \
                    f'but loaded images have {imgs.shape[1]} channels. Please check that ' \
                    'the images are loaded correctly.'

                imgs = imgs.to(device=device, dtype=torch.float32)
                mask_type = torch.float32 if net.n_classes == 1 else torch.long
                true_masks = true_masks.to(device=device, dtype=mask_type)

                logits,probs,masks_pred = net(imgs) #logits, probas, preds
                loss = criterion(logits, true_masks)
                epoch_loss += loss.item()
                #writer.add_scalar('Loss/train', loss.item(), global_step)

                pbar.set_postfix(**{'loss (batch)': loss.item()})

                optimizer.zero_grad()
                loss.backward()
                nn.utils.clip_grad_value_(net.parameters(), 0.1)
                #! nvidia-smi
                optimizer.step()
                torch.cuda.empty_cache()
                #! nvidia-smi
                total_train += true_masks.nelement()
                
                print('total_train',total_train)
                #print('masks_pred.eq(true_masks).sum()',masks_pred.eq(true_masks).sum())
                correct_train += masks_pred.eq(true_masks).sum().item()
                print('correct_train',correct_train)
                train_accuracy = 100*correct_train/total_train
                print('train_accuracy',train_accuracy)
</code></pre>
<p>The weird thing is that, when my batch_size=1, the accuracy is a value between 0-100%. But, when the batch_size =&gt; 2, the accuracy isn’t in a range of 100% any more. I tried to compute the</p>
<pre><code class="lang-auto"> train_accuracy = 
100*correct_train/total_train
</code></pre>
<p>outside of the</p>
<pre><code class="lang-auto">for batch in train_loader:
</code></pre>
<p>but, still didn’t get a reasonable value. I didn’t get why it doesn’t work properly, when the batch_size = &gt; 2. Could you please explain it? Thank u.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>  <a class="mention" href="/u/neda">@Neda</a></p>

ptrblck: <p>Some operations such as this one:</p>
<pre><code class="lang-python">masks_pred.eq(true_masks).sum().item()
</code></pre>
<p>might use broadcasting, if you don’t use tensors in the right shape.<br>
I would recommend to print the shapes of all tensors, which are used to compute the accuracy and make sure they have matching shapes.</p>

Naruto: <p>Many thanks for putting me into right direction. Yes, when the batch_size=1, my true_masks.shape =[1,1,640,959], when the batch_size=2, my true_masks.shape=[2,1,640,959]. Using torch.squeeze(true_masks,1) solved my problem. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

shrutishrestha: <p>what is your batch size for training?</p>

amir_sohail: <p>Is this formula for MIT App inventor or Personal Image Classifier???</p>