Multiclass segmentation U-net masks format

username: Diauro
cooked: <p>I’m quite new to pytorch so I know I might say things which are not really completely correct.<br>
I’m trying out of curiosity to implement a multiclass segmentation using U-net code found here <code>https://github.com/usuyama/pytorch-unet</code>. Since I have my own dataset of images i’ve modified the Class “SimDataset” to upload my own dataset plus images in the following way:</p>
<p>The images are 224x224 png RGB images and the mask are indexed images with four classes (index 1,2,3 features and 0 background).</p>
<blockquote>
<p>from torch.utils.data import Dataset, DataLoader<br>
from torchvision import transforms, datasets, models</p>
<p>class SimDataset(Dataset):<br>
def <strong>init</strong>(self, image_paths, mask_paths, count , transform=None):<br>
self.image_paths = image_paths<br>
self.mask_paths = mask_paths</p>
<pre><code>def transforms(self, image, mask):
    #img = img.resize((wsize, baseheight), PIL.Image.ANTIALIAS)
    #image = transforms.Resize(size=(64, 64))(image)
    #mask = transforms.Resize(size=(64, 64))(mask)
    image = image.resize((64, 64), PIL.Image.NEAREST)
    mask = mask.resize((64, 64), PIL.Image.NEAREST)
    
    image = TF.to_tensor(image)
    mask = TF.to_tensor(mask) 
    
    return [image, mask]


def __getitem__(self, index):
    image = Image.open(self.image_paths[index])
    mask = Image.open(self.mask_paths[index])

    x, y = self.transforms(image, mask)
    return [x, y]

def __len__(self):
    return len(self.image_paths)
</code></pre>
<p>trans = transforms.Compose([<br>
transforms.ToTensor(), transforms.RandomHorizontalFlip()<br>
])</p>
<p>train_set = SimDataset(train_paths, train_masks_paths,100, transform = trans)<br>
val_set = SimDataset(train_paths, train_masks_paths,11, transform = trans)</p>
<p>image_datasets = {<br>
‘train’: train_set, ‘val’: val_set<br>
}</p>
<p>batch_size = 1</p>
<p>dataloaders = {<br>
‘train’: DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=0),<br>
‘val’: DataLoader(val_set, batch_size=batch_size, shuffle=True, num_workers=0)<br>
}</p>
<p>dataset_sizes = {<br>
x: len(image_datasets<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>) for x in image_datasets.keys()<br>
}</p>
<p>dataset_sizes</p>
</blockquote>
<p>I’ve been reading also here that passing the mask as tensor with the command to.Tensor() is not a good habit since the command normalises the values of the class index (and indeed it does) leading to instabilieties. I would like to ask how (using if possible the code snippet i’m using) how to pass correclty the masks during the training.</p>
<p>Also the number of classes that I need to train since I want 3 classes to be recognized in the training is nb_class = 3 ?( n_total_index - 1  meaning 0,1,2,3 but without the background )</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>Instead of <code>to_tensor</code> you could create the mask tensor via <code>torch.from_numpy()</code>.<br>
For 3 classes and a background class, I would assume your model should use <code>nb_class = 4</code>, which would most likely correspond to the number of output units.</p>

username: songyuc
cooked: <p>Hi, Diauro,<br>
I use the function of transforms.ToTensor() in the implemention of customized dataset code, and have not met any problem yet.</p>

username: Diauro
cooked: <p>Thanks for tips! Now I actually have a doubt with the code i’m using regarding how the masks are loaded. Since I have four classes in the masks, using the command PIL.nearest I get basically a 1x64x64 image while if i’m not wrong is a tensor like 1xnb_classx64x64 or i’m wrong?</p>
<p>I was looking here <a href="https://github.com/usuyama/pytorch-unet/blob/master/pytorch_unet.ipynb" rel="nofollow noopener">pytorch_unet</a> at the line:</p>
<pre><code class="lang-auto">import torchvision.utils

def reverse_transform(inp):
    inp = inp.numpy().transpose((1, 2, 0))
    inp = np.clip(inp, 0, 1)
    inp = (inp * 255).astype(np.uint8)
    
    return inp

# Get a batch of training data
inputs, masks = next(iter(dataloaders['train']))

print(inputs.shape, masks.shape)
for x in [inputs.numpy(), masks.numpy()]:
    print(x.min(), x.max(), x.mean(), x.std())

plt.imshow(reverse_transform(inputs[3]))
</code></pre>
<p>where basically the outpur is 25x6x192x192. So I guess the way I’m loading the masks is wrong. Any idea?</p>

username: Diauro
cooked: <p>Hi <a href="https://discuss.pytorch.org/u/songyuc">songyuc</a>,<br>
Thanks for you feedback! Do you use multiclass segmentation by chance? If you could check the reply I gave to <strong>ptrblck</strong> and if you could give me a feedback on how you deal with the mask loading and manipulation would be great. Sorry I’m quite new to pytorch so I might ask some very basic questions.</p>
<p>Best</p>

username: ptrblck
cooked: <p>If you are using a multi-class segmentation use case and therefore <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>, your mask should not contain a channel dimension, but instead contain the class indices in the shape <code>[batch_size, height, width]</code>.</p>
<p><code>PIL.NEAREST</code> is a valid option, as it won’t distort your color codes or class indices.<br>
Since you are loading the image via <code>PIL</code>, I assume your mask is an RGB image, where each color represents a class?</p>
<p>If so, you should map the color values to the corresponding class index.</p>
<p>Let me know, if that would work for you or if you need more information. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: songyuc
cooked: <p>Yes, I am learning about semantic segmentation, too.<br>
I read your and ptrblck’s replies,<br>
and I think there might be an error as you said you had wanted a 1x64x64 image but got a 25x6x192x192 tensor.<br>
I think ptrblck’s suggesstion is good, do the mask loading as:</p>
<ol>
<li>Reading the image file;</li>
<li>Mapping the pixel value into the train class ID.</li>
</ol>

username: Diauro
cooked: <p>I saw an example here <a href="https://discuss.pytorch.org/t/how-make-customised-dataset-for-semantic-segmentation/30881" class="inline-onebox">How make customised dataset for semantic segmentation?</a> of how to do a dataset but I must admit that I still have doubts.</p>
<p>Basically I have PNG image but already labelled with index 0 , 1 , 2, 3  <img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9593f9c358e3e31e59c95e476a06293796edad1.png" alt="23" data-base62-sha1="oa7Ucx9wOQOJXeEq7Wqkv5HFv7r" width="224" height="224">, It’s a costum pallette that I did so techically I should avoid the mapping as in the example or I’m wrong?</p>
<pre><code class="lang-auto">self.mapping = {
            0: 0,
            255: 1              
        }

def mask_to_class(self, mask):
        for k in self.mapping:
            mask[mask==k] = self.mapping[k]
        return mask
</code></pre>
<p>Do you have a snippet of a code that can create the class indices in the shape  <code>[batch_size, height, width]</code> as you said?</p>
<p>Currently I was trying just for one class to apply the  <code>mask = torch.from_numpy(mask)</code> but i’m not sure if it’s ok.</p>
<pre><code>def transforms(self, image, mask):
    #img = img.resize((wsize, baseheight), PIL.Image.ANTIALIAS)

    image = image.resize((64, 64), PIL.Image.NEAREST)
    mask = mask.resize((64, 64), PIL.Image.NEAREST)
    
    mask = np.array(mask)
    
    image = TF.to_tensor(image)
    
    mask = torch.from_numpy(mask) 
    
    
    return [image, mask]
</code></pre>
<p>Now the mask is actually between zero and one as it should be (the problem for more<br>
indices I still need to solve it of course) but once I try to train  I get this:</p>
<p><code>ValueError: Target size (torch.Size([1, 64, 64])) must be the same as input size (torch.Size([1, 1, 64, 64]))</code></p>
<p>I tried to add in the calculation of the loss:      <code>target_ = torch.empty(batch_size, 1,64,64)  target = target_.to(device)</code></p>
<pre><code class="lang-auto">def calc_loss(pred, target, metrics, bce_weight=0.5):
    target_ = torch.empty(batch_size, 1,64,64)
    target = target_.to(device)
    bce = F.binary_cross_entropy_with_logits(pred, target)
        
    pred = F.sigmoid(pred)
    dice = dice_loss(pred, target)
</code></pre>
<p>This does not crash but I have Nan as output in the loss.  So yes I guess i’m quite confused and trying to read as much but cannot figure out a lot at the moment so whatever help would be great <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> thanks again!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Diauro" data-post="9" data-topic="70979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e68b1a/48.png" class="avatar"> Diauro:</div>
<blockquote>
<p>Basically I have PNG image but already labelled with index 0 , 1 , 2, 3</p>
</blockquote>
</aside>
<p>In that case you don’t need any mapping, as your mask will already contain the class indices.</p>
<aside class="quote no-group quote-modified" data-username="Diauro" data-post="9" data-topic="70979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e68b1a/48.png" class="avatar"> Diauro:</div>
<blockquote>
<p>``</p>
<p>I tried to add in the calculation of the loss: <code>target_ = torch.empty(batch_size, 1,64,64)  target = target_.to(device)</code></p>
</blockquote>
</aside>
<p><code>torch.empty</code> uses uninitialized memory to create the tensor, so it might contain invalid values (e.g. NaNs).<br>
For a multi-class segmentation (each pixel belongs to one class only). you should use <code>nn.CrossEntropyLoss</code> instead of <code>nn.BCEWithLogitsLoss</code>.<br>
The latter criterion can be used for a multi-label classification/segmentation (each pixel can belong to zero, one, or more classes).</p>

username: Diauro
cooked: <p>Thanks for the quick reply!</p>
<p>I must admit that is not clear to me how to create the mask. Once I load the mask with PIL I pass my mask as</p>
<blockquote>
<p>mask = torch.from_numpy(mask)</p>
</blockquote>
<p>but then is not clear how to create this one:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="70979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>but instead contain the class indices in the shape <code>[batch_size, height, width]</code> .</p>
</blockquote>
</aside>
<p>Because I guess once I create this tensor like [batch_size, num_classes, height , width] I guess I can skip this step on the training loop</p>
<aside class="quote no-group" data-username="Diauro" data-post="9" data-topic="70979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/e68b1a/48.png" class="avatar"> Diauro:</div>
<blockquote>
<p><code>target_ = torch.empty(batch_size, 1,64,64)  target = target_.to(device)</code></p>
</blockquote>
</aside>
<p>That gives me the error with the Nan due to what you said about the memory that might contain those values.</p>
<p>Best</p>

username: Diauro
cooked: <p>Hi Songyuc,</p>
<p>Could you post if it’s not a problem (or send me the link to some examples),  of how you create the multiclass masks?</p>
<p>Many thanks!</p>

username: henrique
cooked: <p>Hey Diauro, you can find many examples and pre-trained models at <a href="https://github.com/qubvel/segmentation_models.pytorch" rel="nofollow noopener">https://github.com/qubvel/segmentation_models.pytorch</a><br>
Great resource!</p>

username: songyuc
cooked: <p>Yes, here is an example of tranforming the mask data into train ID,<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/gujingxiao/Lane-Segmentation-Solution-For-BaiduAI-Autonomous-Driving-Competition/blob/master/utils/process_labels.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/gujingxiao/Lane-Segmentation-Solution-For-BaiduAI-Autonomous-Driving-Competition/blob/master/utils/process_labels.py" target="_blank" rel="nofollow noopener">gujingxiao/Lane-Segmentation-Solution-For-BaiduAI-Autonomous-Driving-Competition/blob/master/utils/process_labels.py</a></h4>
<pre><code class="lang-py">"""
Author: Jingxiao Gu
Baidu Account: Seigato
Description: Encode &amp; Decode Labels for Lane Segmentation Competition
NOTE: For 8 classes
"""
import numpy as np

def encode_labels(color_mask):
    encode_mask = np.zeros((color_mask.shape[0], color_mask.shape[1]))
    # 0
    encode_mask[color_mask == 0] = 0
    encode_mask[color_mask == 249] = 0
    encode_mask[color_mask == 255] = 0
    # 1
    encode_mask[color_mask == 200] = 1
    encode_mask[color_mask == 204] = 1
    encode_mask[color_mask == 213] = 0
    encode_mask[color_mask == 209] = 1
    encode_mask[color_mask == 206] = 0
</code></pre>

  This file has been truncated. <a href="https://github.com/gujingxiao/Lane-Segmentation-Solution-For-BaiduAI-Autonomous-Driving-Competition/blob/master/utils/process_labels.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Diauro
cooked: <p>Thanks everyone for the help! following you advices I managed to modify and run correctly the unet (more or less, the loss kind of saturates and does not go particularly down but that might be my dataset).</p>
<p>Quick question is there a way to easely implement dropout in the U-net snippet found here <code>https://github.com/usuyama/pytorch-unet</code> ?</p>
<p>Best and thanks again for the help!</p>