How to convert RGB images with many different colors (not only red, green, blue) into classes for segmentation training?, The mask is linked below

thejas_karkera: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a980cb8fb82c6d65899808e2fe8d846d6e844b98.png" data-download-href="https://discuss.pytorch.org/uploads/default/a980cb8fb82c6d65899808e2fe8d846d6e844b98" title="TCGA-2Z-A9J9-01A-01-TS1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a980cb8fb82c6d65899808e2fe8d846d6e844b98.png" alt="TCGA-2Z-A9J9-01A-01-TS1" data-base62-sha1="obuDmqE9akoFk3kAW37zYZ3Ptnq" width="500" height="500" data-dominant-color="191A1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TCGA-2Z-A9J9-01A-01-TS1</span><span class="informations">1000×1000 43.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/training-semantic-segmentation/49275/4">This code</a> shows an example of the transformation from colors to class indices.</p>

thejas_karkera: <p>Thankyou so much for the reply!, i was also wondering how do i pass these values to nn.CrossEntropy function. Won’t i get an dimensional mismatch?<br>
And also wanted to know after getting the mask from your code, can i directly pass these masks from the <strong>get_item</strong>() function?</p>

ptrblck: <p>Yes, you should be able to transform the color masks to the mask targets containing indices and return them in the <code>__getitem__</code>.</p>
<p>No, <code>nn.CrossEntropyLoss</code> won’t raise an error, if you pass the model outputs in the shape <code>[batch_size, nb_classes, height, width]</code> and the masks as <code>[batch_size, height, width]</code> for a multi-class segmentation use case.</p>

thejas_karkera: <p>Thanks for the reply again. I’m clear how to pass the arguments to <code>nn.CrossEntropyLoss</code> but im not clear how do i pass the masks and images.<br>
What values should i return in my ‘<strong>get__item</strong>’. Can you help me out with this, im new to pytorch.</p>

thejas_karkera: <p>In the linked code actually we have to specify num of classes, i.e  in my case number of different colors. I don’t have that information with me. So, is there any other solution?</p>

ptrblck: <p>That’s a rather uncommon use case. How would you define the last output layer, if the number of classes is unknown?<br>
Are you planning to add an “unknown” class category which the model should use for all additional classes?</p>

thejas_karkera: <p>I’m still working on it, I was thinking ‘Number of Unique Colors= Number of classes’.<br>
Or am I wrong? Is there a different way I could train using this dataset.</p>

ptrblck: <p>Yes, usually the number of <em>all</em> unique colors in the dataset would correspond to the number of classes.</p>
<aside class="quote no-group" data-username="thejas_karkera" data-post="8" data-topic="110102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thejas_karkera/48/33259_2.png" class="avatar"> thejas_karkera:</div>
<blockquote>
<p>Is there a different way I could train using this dataset.</p>
</blockquote>
</aside>
<p>As explained before, you could try to add “new/unknown” colors to a special “unknown” class category, but it really depends on your use case and what you are trying to achieve.</p>

thejas_karkera: <p>Thankyou for the reply. Assuming that there are same number of unique colors in every image in my dataset I will move ahead.<br>
But I’m still not clear how can I pass the mask, after using the code.<br>
Can I know exactly what the code will return. For eg: Shape Or Content.</p>

ptrblck: <p>The <code>__getitem__</code> should return an input tensor in the shape <code>[channels, height, width]</code> and a mask tensor in the shape <code>[height, width]</code> containing class indices in the range <code>[0, nb_classes-1]</code>.<br>
The <code>DataLoader</code> will then add the batch dimension to these tensors, such that the input tensor will have a shape of <code>[batch_size, channels, height, width]</code> while the mask <code>[batch_size, height, width]</code>.</p>

thejas_karkera: <p>Thankyou for the reply, helped me a lot!. Get back to you if something goes wrong while implementing this.</p>

thejas_karkera: <p>Hello Sir thankyou for all the help, is there a code to get the number of unique colors in an image</p>

ptrblck: <p>Yes, my linked code snippet gets all unique colors from the image.</p>
<pre><code class="lang-python">[...]
# Get color codes for dataset (maybe you would have to use more than a single
# image, if it doesn't contain all classes)
target = torch.from_numpy(target)
colors = torch.unique(target.view(-1, target.size(2)), dim=0).numpy()
[...]
</code></pre>

thejas_karkera: <p><strong>Ignore the previous error, I’m using the linked code as shown above, Im using your code as below and im getting an error:"The shape of the mask [3, 1000] at index 0 does not match the shape of the indexed tensor [1000, 1000] at index 0"</strong><br>
**What are the values of h and w i should use, i Highly doubt im using the code properly<br>
**</p>
<p>def <strong>getitem</strong>(self, idx):</p>
<pre><code>    # load images ad masks

    img_path = os.path.join(self.root, "original_images", self.imgs[idx])

    mask_path = os.path.join(self.root, "col", self.masks[idx])

    img = Image.open(img_path).convert("RGB")

    # note that we haven't converted the mask to RGB,

    # because each color corresponds to a different instance

    # with 0 being background

    h, w = 1000, 1000

             

    mask = Image.open(mask_path)

    #mask = np.array(mask)

    # Create mapping
</code></pre>
<h1>Get color codes for dataset (maybe you would have to use more than a single</h1>
<h1>image, if it doesn’t contain all classes)</h1>
<pre><code>    #target = torch.from_numpy(mask)

    target=self.transform(mask) #converts to tensor, resize to 1000x1000

    colors = torch.unique(target.view(-1, target.size(2)), dim=0).numpy()

    target = target.permute(2, 0, 1).contiguous()

    mapping = {tuple(c): t for c, t in zip(colors.tolist(), range(len(colors)))}

    mask = torch.empty(h, w, dtype=torch.long)

    for k in mapping:

# Get all indices for current class

      idx = (target==torch.tensor(k, dtype=torch.uint8).unsqueeze(1).unsqueeze(2))

      validx = (idx.sum(0) == 3)  # Check that all channels match

      mask[validx] = torch.tensor(mapping[k], dtype=torch.long)
      return self.transform(img), (mask)</code></pre>

thejas_karkera: <p>Hello sir is there any example code for this? im getting a ““1only batches of spatial targets supported (3D tensors) but got targets of size: : [1, 1000, 1000, 3]”” error</p>

ptrblck: <p>You could compare my code example to yours and try to find the difference or are you also seeing this error using my code snippet?<br>
If you cannot find the issue, feel free to post an executable code snippet which reproduces the error.</p>

thejas_karkera: <p><strong>This is how i usedyour code:</strong><br>
<em><strong>I dont have that error anymore, but is the proper way to use your code?</strong></em><br>
“This is the current error”<br>
“ValueError: Expected target size (1, 480), got torch.Size([1, 3, 480])”</p>
<p>“Used nn.CrossEntropyLoss(), num_classes=256(randomly initialized, bcz i was getting the len(colors) as 2992”</p>
<p>class Cus_dataset(torch.utils.data.Dataset):</p>
<pre><code>def __init__(self, root,transform,transformm):

    self.root = root

    self.transform =  transform

    self.transformm = transformm

    # load all image files, sorting them to

    # ensure that they are aligned

    self.imgs = list(sorted(os.listdir(os.path.join(root, "original_images"))))

    self.masks = list(sorted(os.listdir(os.path.join(root, "col"))))

                  

def __getitem__(self, idx):

    # load images ad masks

    img_path = os.path.join(self.root, "original_images", self.imgs[idx])

    mask_path = os.path.join(self.root, "col", self.masks[idx])

    img = Image.open(img_path).convert("RGB")

    # note that we haven't converted the mask to RGB,

    # because each color corresponds to a different instance

    # with 0 being background

                    

    mask = Image.open(mask_path)

    mask = np.array(mask)

    #colors = np.unique(mask)

    #mask =self.transform(mask)

    def mask_to_class(mask):

      target = torch.from_numpy(mask)

      h,w = target.shape[0],target.shape[1]

      masks = torch.empty(h, w, dtype=torch.long)

      colors = torch.unique(target.view(-1,target.size(2)),dim=0).numpy()

      target = target.permute(2, 0, 1).contiguous()

      mapping = {tuple(c): t for c, t in zip(colors.tolist(), range(len(colors)))}

      for k in mapping:

        idx = (target==torch.tensor(k, dtype=torch.uint8).unsqueeze(1).unsqueeze(2))

        validx = (idx.sum(0) == 3) 

        masks[validx] = torch.tensor(mapping[k], dtype=torch.long)

      

    masks = mask_to_class(mask)
   return masks</code></pre>

ptrblck: <aside class="quote no-group" data-username="thejas_karkera" data-post="20" data-topic="110102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thejas_karkera/48/33259_2.png" class="avatar"> thejas_karkera:</div>
<blockquote>
<p>“Used nn.CrossEntropyLoss(), num_classes=256(randomly initialized, bcz i was getting the len(colors) as 2992”</p>
</blockquote>
</aside>
<p>Since you don’t know the number of classes, I think you would have to estimate it.<br>
Do 2992 unique colors look right for the processed dataset?</p>
<aside class="quote no-group" data-username="thejas_karkera" data-post="20" data-topic="110102">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thejas_karkera/48/33259_2.png" class="avatar"> thejas_karkera:</div>
<blockquote>
<p>“ValueError: Expected target size (1, 480), got torch.Size([1, 3, 480])”</p>
</blockquote>
</aside>
<p>What shape does the model output have?<br>
If you are working on a multi-class segmentation use case, the output should have the shape <code>[batch_size, nb_classes, height, width]</code>, while the target should have the shape <code>[batch_size, height, width]</code> and contain class indices in the range <code>[0, nb_classes-1]</code>.</p>

thejas_karkera: <p>My input image size is 480x480x3, if i assume it has 256 classes, then my last layer of my model will have a channel size 256.</p>