Ground-truth label normalization

keyur_paralkar: <p>I am currently working on pascal voc 2012 dataset. I am a bit confused regarding the Normalization of images. Do I need to normalize the ground-truth labels, if I am normalizing my input images?<br>
For experimentation, I tried to normalize the ground-truth labels for the corresponding input images using <code>transforms.Normalize()</code> function. When I try to train my network I get an error:</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-12-416ca5ee56ac&gt; in &lt;module&gt;()
     10 model2 = model2.to(device)
     11 
---&gt; 12 m1,loss_acc_dict = train_model(model=model2,criterion=criterion,optimizer=optimizer,num_epochs=175)

&lt;ipython-input-10-1744505e89c5&gt; in train_model(model, criterion, optimizer, scheduler, num_epochs)
     38             print(train_y)
     39             loss = criterion(outputs,train_y)
---&gt; 40             epoch_loss += float(loss)
     41 
     42             #calculating the accuracy:

RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/generic/THCStorage.cpp:36
</code></pre>
<p>Here is my code for train helper function:</p>
<pre><code class="lang-auto">#Create a function for training a model.
def train_model(model, criterion, optimizer, scheduler=None, num_epochs=10):
#     since = time.time() #Start the timer
    
    model.train() #Put in train mode.
    
    total_acc = []
    total_loss = []
    total_iou = []
    
    for epochs in range(num_epochs):
        epoch_loss = 0.0
        epoch_acc = 0
        epoch_iou = 0
        
        print('\n')
        print("Epoch {}/{}".format(epochs,num_epochs-1))
        print('-'*10)
        iteration = 0
        total_iteration = train_images_dataloader.__len__()
        
        for batch, data in enumerate(train_images_dataloader):
            
            #Keeping count of currect iteration
            iteration += 1
            
            #loading the data on the GPU:
            train_x, train_y = data['image_x'].to(device), data['image_y'].to(device)
            
            #zero the parameter gradients:
            optimizer.zero_grad()
            
            #forward pass:
            outputs = model(train_x)
            
            #compute loss:
            train_y = train_y.squeeze().long()
            print(train_y)
            loss = criterion(outputs,train_y)
            epoch_loss += loss
            
            #calculating the accuracy:
            _, predicted_indices = torch.max(outputs,1)
            total = train_y.size(0)*train_y.size(1)*train_y.size(2)
            correct = (predicted_indices == train_y).sum().item()
            acc = (correct/total)
            epoch_acc += float(acc)
            
            m_iou = compute_iou(predicted_indices,train_y)
            epoch_iou +=m_iou
            print('Iteration = {}/{} Loss = {} Accuracy = {}, IOU = {}'.format(iteration,total_iteration,loss.item(),acc,m_iou),end="\r")
            
            #back-propogate:
            loss.backward()
            
            #update the weights:
            optimizer.step()
            
            #Memory management:
            del outputs,train_x,train_y
            
        writer.add_scalar('Train/Loss',epoch_loss/total_iteration,global_step=epochs)
        writer.add_scalar('Train/Accuracy',(epoch_acc/total_iteration)*100,global_step=epochs)
        writer.add_scalar('Train/Mean IOU',(epoch_iou/total_iteration),global_step=epochs)
        total_acc.append(epoch_acc/total_iteration)
        total_loss.append(epoch_loss/total_iteration)
        total_iou.append(epoch_iou/total_iteration)
        
    print('\n')
    print('Total accuracy = ',sum(total_acc)/num_epochs)
    print('Total Loss = ',sum(total_loss)/num_epochs)
    print('Total mean IOU = ',sum(total_iou)/num_epochs)
    
    return model,{'Accuracy':total_acc,'Loss':total_loss,'mIOU':total_iou} #divide by num_opochs and then sotre in dict.
</code></pre>
<p>Custom dataset loader:</p>
<pre><code class="lang-auto">#Creating custom dataloader fro PASCAL VOC 2012

'''
__init__ : class variables such as, images_path, transform etc.
__len__ : would return len of the entire dataset
__getitem__ : returns ith image from the dataset
'''

class PascalVocDataset(Dataset):
    
    def __init__(self,anno_path,ip_images_path,seg_images_path,transform=None):
        '''
        anno_path = path to train.txt file for image identifers
        ip_images_path = path to input image folders
        seg_images_path = path to segmentation maps class folder
        '''
        self.anno_path = anno_path
        self.ip_images_path = ip_images_path
        self.seg_images_path = seg_images_path
        self.transform = transform
        
    def getImagesNames(self,path):
        #Returns image identifiers from file_name
        #path = .txt location
        
        images_names = []
        with open(path) as fp:
            images_names = list(map((lambda x: x[:-1]),fp.readlines()))
            fp.close()
        return images_names
    
    def __len__(self):
        return len(self.getImagesNames(self.anno_path))
    
    def __getitem__(self, idx):
        file_names = self.getImagesNames(self.anno_path)
        file_path_x = self.ip_images_path+file_names[idx]+'.jpg'
        file_path_y = self.seg_images_path+file_names[idx]+'.png'
        
        #X images:
        im = Image.open(file_path_x)
        image_x = im.convert('RGB')
        
        #Y images:
        img = Image.open(file_path_y)
#         img = img.resize((224,224),Image.ANTIALIAS)
#         image_y = torch.tensor(np.array(img),dtype=torch.long)
        
        sample = {'image_x':image_x,'image_y':img}
        
        if self.transform:
            sample['image_x'] = self.transform(sample['image_x'])
            sample['image_y'] = self.transform(sample['image_y'])

    
        return sample
</code></pre>

InnovArul: <p>I assume that you are talking about PASCAL VOC 2012 segmentation dataset.<br>
In my view, it depends on how you are calculating the loss. When you are using CrossEntropyLoss or NLLLoss2d (per pixel), you should not normalize the ground truth.</p>
<p>What kind of loss function are you using?</p>

keyur_paralkar: <p>I am using crossentropyloss. So I should just normalize the inputs ?. If I just normalize the input wouldn’t it produce greater loss as the range of values are different now for input and ground truth ?</p>

InnovArul: <p>Yes. you should just normalize the inputs in my view.<br>
The given ground truths are not images with ‘pixel values’, but they are labels that say which class the particular pixel belongs to. I am not seeing the reason why you need to normalize the labels.</p>

keyur_paralkar: <p>This makes sense So normalizing ground-truth labels completely depends upon the problem we are working on, right ?</p>

keyur_paralkar: <p>I was trying to normalize the train set because I was getting blank predictions after training. That’s the reason I thought I should normalize ground truth labels as well</p>

ptrblck: <p>That might be a problem of your model, if it only outputs one laben (e.g. the background class), or a visualization issue.<br>
If you call <code>torch.argmax</code> on your output, you will get the most likely classes as indices.<br>
Some libraries like <code>matplotlib</code> try to plot these “images” as <code>uint8</code> in the range <code>[0, 255]</code>.<br>
As you class labels might be small, e.g. <code>[0, 5]</code> you might see only a dark image.<br>
Try to convert your labels to colors using a colormap in this case.</p>

keyur_paralkar: <p>This is my method of replacing indicies with higher pixel intensity values:</p>
<pre><code class="lang-auto">#For generating colormap of given image:
def replace_pixels(img):

    labelsTC = [x for x in range(200,246)]
    temp_mask = np.zeros((img.shape),dtype=np.int)

    '''

      traverse through the given array img array. and compare each point like below

      if point &gt;= 0 &amp;&amp; point&lt;=20:

        temp_mask[i] = labelsTC[point]

    '''

    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
                if(img[i,j]&gt;=1 and img[i,j]&lt;=20):
                    temp_mask[i,j] = labelsTC[img[i,j]]



    return temp_mask
</code></pre>
<p>And these are the results that I get after visualisation:<br>
Image 1, 2, 3 are the Input image, predictions (segmentation maps), ground-truth labels respectively.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/21f43490240f0f8ea5b62aa0194ceb64b2174924.png" data-download-href="https://discuss.pytorch.org/uploads/default/21f43490240f0f8ea5b62aa0194ceb64b2174924" title="plane"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/21f43490240f0f8ea5b62aa0194ceb64b2174924_2_689x494.png" alt="plane" data-base62-sha1="4QmXLGO85oBlzIJRRzcnGspHqNC" width="689" height="494" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/21f43490240f0f8ea5b62aa0194ceb64b2174924_2_689x494.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/21f43490240f0f8ea5b62aa0194ceb64b2174924_2_1033x741.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/21f43490240f0f8ea5b62aa0194ceb64b2174924.png 2x" data-dominant-color="E1D9E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plane</span><span class="informations">1344×963 88.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/228a842cb42b2d6cdffd8fafaf1934cb5767b202.png" data-download-href="https://discuss.pytorch.org/uploads/default/228a842cb42b2d6cdffd8fafaf1934cb5767b202" title="bird"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/228a842cb42b2d6cdffd8fafaf1934cb5767b202_2_690x470.png" alt="bird" data-base62-sha1="4Vz0eF7LgRuhgvBS3bbhKa4vGhQ" width="690" height="470" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/228a842cb42b2d6cdffd8fafaf1934cb5767b202_2_690x470.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/228a842cb42b2d6cdffd8fafaf1934cb5767b202_2_1035x705.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/228a842cb42b2d6cdffd8fafaf1934cb5767b202.png 2x" data-dominant-color="E4DBE5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">bird</span><span class="informations">1343×916 94.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e" title="computer"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e_2_689x490.jpeg" alt="computer" data-base62-sha1="94gPJ7TuYjuugNWAdKyeOteV4PY" width="689" height="490" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e_2_689x490.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e_2_1033x735.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3f8f2d1e24ad5550af5b19c6ee2fb8266732029e.jpeg 2x" data-dominant-color="E4DDE4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">computer</span><span class="informations">1344×955 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4034a6ac4b1eecb4a84b4489979faff964e34f45.png" data-download-href="https://discuss.pytorch.org/uploads/default/4034a6ac4b1eecb4a84b4489979faff964e34f45" title="dogonchair"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4034a6ac4b1eecb4a84b4489979faff964e34f45_2_690x471.png" alt="dogonchair" data-base62-sha1="99ZmtrwOroDtF6GmGxNzx4BiUCN" width="690" height="471" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4034a6ac4b1eecb4a84b4489979faff964e34f45_2_690x471.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4034a6ac4b1eecb4a84b4489979faff964e34f45_2_1035x706.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4034a6ac4b1eecb4a84b4489979faff964e34f45.png 2x" data-dominant-color="E2DADE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dogonchair</span><span class="informations">1343×918 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>So your predictions are not blank but just look wrong.<br>
Do you just want to predict the edges or the classes as well?<br>
<code>labelsTC</code> seems to be some kind of color map with a small range. Did you try to use a colormap from e.g. <code>matplotlib</code> to spread the colors a bit?</p>

keyur_paralkar: <p>I want to obtain segmentation maps atleast equivalent to the ground-truth, also I am using over here is FCN-32 model. No I have not tried to use colormap from matplotlib to spread the colors can you provide any example of how to do it ? Sorry for my limited knowledge</p>

keyur_paralkar: <p>Yes <a class="mention" href="/u/ptrblck">@ptrblck</a> labelsTC is a color map with small range.</p>

keyur_paralkar: <p>After changing color map to ‘Paired’ I got follow segmentation maps:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c061c816be3433d0abc4db67182a38ea0a1e5888.png" data-download-href="https://discuss.pytorch.org/uploads/default/c061c816be3433d0abc4db67182a38ea0a1e5888" title="newcolormap"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c061c816be3433d0abc4db67182a38ea0a1e5888_2_690x471.png" alt="newcolormap" data-base62-sha1="rrTaz8DCAR2AdDFW447NntjPGje" width="690" height="471" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c061c816be3433d0abc4db67182a38ea0a1e5888_2_690x471.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c061c816be3433d0abc4db67182a38ea0a1e5888_2_1035x706.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/c061c816be3433d0abc4db67182a38ea0a1e5888.png 2x" data-dominant-color="E4E0E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newcolormap</span><span class="informations">1343×918 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For some cases like detecting boat in an image and generating its segmentation map, my current model completely fails at it and gives blank predictions. But for some examples like displayed above gives better results. Does this mean for better accuracy than this I should use FCN-16 or FCN-8 architecture ?</p>

ptrblck: <p>It’s hard to answer which architecture will definitely work better.<br>
Currently your model seems to fail at the segmentation task.<br>
However, the model might still be a good fit, but the hyper-parameters, e.g. learning rate, are messing up the training.<br>
Have you checked your per-class accuracies? Most likely some classes will be completely ignored.</p>

keyur_paralkar: <p>How can I calculate pre-class accuracies? using confusion matrix?<br>
Also this is my implementation of FCN-32 model, is this the correct implementation:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7dc53fa7f489bb2969853182f8160ab8266714d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/7dc53fa7f489bb2969853182f8160ab8266714d5" title="model_fcn32"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7dc53fa7f489bb2969853182f8160ab8266714d5_2_690x491.png" alt="model_fcn32" data-base62-sha1="hWChNCRtPR1oEfLsAXVGS72lKRL" width="690" height="491" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7dc53fa7f489bb2969853182f8160ab8266714d5_2_690x491.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7dc53fa7f489bb2969853182f8160ab8266714d5_2_1035x736.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/7dc53fa7f489bb2969853182f8160ab8266714d5.png 2x" data-dominant-color="EFF2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model_fcn32</span><span class="informations">1330×948 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5053329ad900d8fb1a465f4433d7f136db9856e1.png" data-download-href="https://discuss.pytorch.org/uploads/default/5053329ad900d8fb1a465f4433d7f136db9856e1" title="actual_training"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5053329ad900d8fb1a465f4433d7f136db9856e1_2_690x366.png" alt="actual_training" data-base62-sha1="bsArLtb3H0HdDGO8f3p79zMfCut" width="690" height="366" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5053329ad900d8fb1a465f4433d7f136db9856e1_2_690x366.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5053329ad900d8fb1a465f4433d7f136db9856e1_2_1035x549.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/5053329ad900d8fb1a465f4433d7f136db9856e1.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">actual_training</span><span class="informations">1343×713 81.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

InnovArul: <p>Why did you decide not to use any non-linearity in your decoder part?<br>
Is FCN not using it?</p>

keyur_paralkar: <p>I haven’t used any non-linearity in fcn because it gave me better results i.e. accuracy but not that much as I expected. Also I thought RelU  might be causing my values to diminish.</p>

keyur_paralkar: <p>I have read from some of the discussions from the forums of pytorch, that the current implementation of vgg16 of pytorch doesn’t give better performance as compared to actual Oxford original vgg16. I will try to implement and copy the weights of original vgg16 or will try to implement this via. Resnet.</p>

tr_marwa: <p>Hello<br>
i have the same problem, if you can help me ?<br>
i’m working on the IRM’s data, i had normalize my inputs data at the beginning, and I didn’t normalize the groudtruth segmentation !!!<br>
the problem is that I have 4 class [0: bachground, 1: LV, 2: MYO, 3: RV]<br>
but in output after training my model I only have the backgroud<br>
i’m ploting that like this :<br>
for i in range(0,len(outputs)):<br>
_imshow(inputs[i][0].cpu().numpy(),labels[i].cpu().numpy(),torch.argmax(outputs[i],0).cpu().numpy())<br>
and i get that<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/21f3f202f28af3eb7e760b36295f83110cc550ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/21f3f202f28af3eb7e760b36295f83110cc550ff" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21f3f202f28af3eb7e760b36295f83110cc550ff_2_690x269.png" alt="Capture" data-base62-sha1="4QmpeFGIIMyOWUg08nXCbWjlanZ" width="690" height="269" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/21f3f202f28af3eb7e760b36295f83110cc550ff_2_690x269.png, https://discuss.pytorch.org/uploads/default/original/3X/2/1/21f3f202f28af3eb7e760b36295f83110cc550ff.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/1/21f3f202f28af3eb7e760b36295f83110cc550ff.png 2x" data-dominant-color="6D567B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">955×373 52.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>thanks in advance, hope you could help me</p>

ptrblck: <aside class="quote no-group" data-username="tr_marwa" data-post="18" data-topic="26981">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tr_marwa/48/25960_2.png" class="avatar"> tr_marwa:</div>
<blockquote>
<p>i had normalize my inputs data at the beginning, and I didn’t normalize the groudtruth segmentation !!!</p>
</blockquote>
</aside>
<p>That should be correct, as the target mask should contain the class indices. Normalizing it might create floating point numbers, which would be wrong for a multi-class segmentation use case using e.g. <code>nn.CrossEntropyLoss</code>.</p>
<p>You could check the complete <code>output</code> tensor for the unique prediction labels via:</p>
<pre data-code-wrap="python"><code class="lang-python">preds = outputs.argmax(1)
print(torch.unique(preds))
</code></pre>
<p>If this prints only a 0, then your model is only predicting the background and seems to be overfitting to this class.</p>

tr_marwa: <p>Thank you for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=9" title=":pray:" class="emoji" alt=":pray:"></p>
<p>yes, actually when i tried it just gives me 0<br>
do you think that my model UNET (network) is wrong ??<br>
knowing that I initialized the weights which corresponds to the 4 labels and I introduced them to the loss as well<br>
weights = torch.tensor([0.04,0.32,0.32,0.32]).to(device)<br>
criterion = nn.CrossEntropyLoss(weights)<br>
sorry to bother you, i got a little lost, thanks again !!!</p>