Multi Class implementation Mask RCNN

FPCRasmus: <p>Dear,</p>
<p>I’m following the tutorial:<br>
TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>
<p>I’m able to train and test it on my own dataset with eyes. The problem I have is that the tutorial only covers 2 classes: background and people. The masks are made:<br>
‘zeros’ = background<br>
‘ones’ = instance number 1<br>
‘twos’ = instance number 2</p>
<p>The last 2 instances in this case are the same class: People.</p>
<p>What do I have to change to make it work for multiple classes. I.e. I want to classify: pupils, iris, and background (3 classes).</p>
<p>Do I have to make separate masks for pupils and irises , i.e.:<br>
<strong>Folder1</strong><br>
‘zeros’ = background<br>
‘ones’ = iris</p>
<p><strong>Folder2</strong><br>
‘zeros’ = background<br>
‘ones’ = pupil</p>
<p>or can I identify them in 1 overall mask:<br>
‘zeros’ = background<br>
‘ones’ = iris<br>
‘twos’ = pupil</p>
<p>The code also mentions: num_classes = 2<br>
changing this to 3 doesn’t really makes sense because I don’t know what the setup should be.</p>
<p>Kind regards<br>
Rasmus</p>

FPCRasmus: <p>No one?</p>
<p>I tried some other frame works which works but I can’t get it to work on this one.</p>

Deeply: <p>replacing</p>
<pre><code class="lang-auto">labels = torch.ones((num_objs,), dtype=torch.int64)
</code></pre>
<p>with the following</p>
<p><code>labels = torch.as_tensor(obj_ids, dtype=torch.int64)</code></p>
<p>will do.<br>
In fact, the replacement even works for two-class instance segmentation.</p>

azer: <p>Hi I tried what you advised however this error is triggered</p>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
3 for epoch in range(num_epochs):<br>
4     # train for one epoch, printing every 10 iterations<br>
----&gt; 5     train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)<br>
6     # update the learning rate<br>
7     lr_scheduler.step()</p>
<p>5 frames<br>
/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/roi_heads.py in fastrcnn_loss(class_logits, box_regression, labels, regression_targets)<br>
40     # the corresponding ground truth labels, to be used with<br>
41     # advanced indexing<br>
—&gt; 42     sampled_pos_inds_subset = torch.nonzero(labels &gt; 0).squeeze(1)<br>
43     labels_pos = labels[sampled_pos_inds_subset]<br>
44     N, num_classes = class_logits.shape</p>
<p>RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered</p>

Deeply: <aside class="quote no-group" data-username="azer" data-post="4" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azer/48/43253_2.png" class="avatar"> azer:</div>
<blockquote>
<p>num_classes = class_logits.shap</p>
</blockquote>
</aside>
<p>Working fine with me.<br>
Yet, <code>copy_if failed to synchronize</code> may be due to incorrect number of classes you used to build the model, and might not be related to the above correction I am suggesting. One reason of this (copy_if failed error), which I have encountered myself, is neglecting the background in the number of classes when building the model. Hence, if you have 10 object categories that you want to detect in your problem (background is neglected), then, you’ll have to build your model to detect 11 classes; that is, the number of classes should be 11.</p>

azer: <p>Hi</p>
<p>This is my model 20 classes + background in total 21.  Still got the error</p>
<p>(box_predictor): FastRCNNPredictor(<br>
(cls_score): Linear(in_features=1024, out_features=21, bias=True)<br>
(bbox_pred): Linear(in_features=1024, out_features=84, bias=True)<br>
)<br>
(mask_roi_pool): MultiScaleRoIAlign()<br>
(mask_head): MaskRCNNHeads(<br>
(mask_fcn1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu1): ReLU(inplace=True)<br>
(mask_fcn2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu2): ReLU(inplace=True)<br>
(mask_fcn3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu3): ReLU(inplace=True)<br>
(mask_fcn4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(relu4): ReLU(inplace=True)<br>
)<br>
(mask_predictor): MaskRCNNPredictor(<br>
(conv5_mask): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2))<br>
(relu): ReLU(inplace=True)<br>
(mask_fcn_logits): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))</p>

Deeply: <aside class="quote no-group" data-username="azer" data-post="6" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/azer/48/43253_2.png" class="avatar"> azer:</div>
<blockquote>
<p>onv2d(256, 21</p>
</blockquote>
</aside>
<p>why don’t you just follow the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#modifying-the-model-to-add-a-different-backbone" rel="noopener nofollow ugc">tutorial - modifying-the-model-to-add-a-different-backbone</a>?</p>

FPCRasmus: <p>And what should my dataset look like? I put 2 examples in my initial question above.</p>

azer: <p>I had followed the tutorial but I don’t need to modified the backbone at this stage. If I cannot get decent AP with fasterRCNN as a baseline there is not point to change the backbone. I  am using Pascal Voc dataset and was expecting to get similar results to the benchmark but it seems that there is something missing.</p>

Deeply: <aside class="quote no-group" data-username="FPCRasmus" data-post="8" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fpcrasmus/48/20771_2.png" class="avatar"> FPCRasmus:</div>
<blockquote>
<p>I put 2 examples in my initial question above</p>
</blockquote>
</aside>
<p>you’ll need to put</p>
<pre><code class="lang-auto">‘zeros’ = background
‘ones’ = iris
‘twos’ = pupil
</code></pre>
<p>into one <code>numpy</code> mask and you are good to go. You can then use and modify the tutorial’s <code>PennFudanDataset</code> class accordingly. In addition to using <code>labels = torch.as_tensor(obj_ids, dtype=torch.int64)</code> suggested above; and use <code>num_classes=3</code>. This will work out of the box.</p>

Deeply: <p>If you don’t want to change the backbone, and opt for using <code>resnet50</code> model described in the tutorial, all should go well if you use <code>labels = torch.as_tensor(obj_ids, dtype=torch.int64)</code>. Unless there is something incorrect in your <code>dataset class</code>. Hence, I suggest that you revise your <code>dataset class</code>, best way to do this is by invoking it separately (a simple program with a <code>main</code> function) and check that everything is correct; better off in the debug mode (via <code>Spyder</code> or <code>PyChram</code>) to trace everything in a step-by-step bases. You might as well go over the rest of your code to make sure everything is correct. I have tried it with 60 different classes and it is working.</p>

FPCRasmus: <p>So I made my dataset the way you described it, I put num_classes to 3 and changed the labels line.</p>
<p>I now get an error in the following part of the code:</p>
<pre><code class="lang-auto"># split the color-encoded mask into a set of binary masks

  masks = mask == obj_ids[:, None, None]
  print(obj_ids[:, None, None])
        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)

        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])
</code></pre>
<p>The error that I get:</p>
<pre><code class="lang-auto">    data.reraise()
  File "anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
    raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in &lt;listcomp&gt;
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "multi_training.py", line 55, in __getitem__
    pos = np.where(masks[i])
TypeError: 'bool' object is not subscriptable
</code></pre>
<p>The obj_ids print statement gives:</p>
<pre><code class="lang-auto">[[[1]]

 [[2]]]
</code></pre>
<p>masks become False instead of a 2d list with false and true</p>
<p>What can I do? or did I made a mistake with the masks?</p>
<p>This is one of the masks: (note, iris = (1,1,1) pupil = (2,2,2))<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c158f106dfcdf39ca4784f73a3f706747164ab7.png" alt="nhai_right" data-base62-sha1="jZf2cmsLRZDJhMd3vV9CNMmIhBZ" width="255" height="255"></p>
<p>The changes are working with the old, 2 class dataset.</p>

Deeply: <aside class="quote no-group quote-modified" data-username="FPCRasmus" data-post="12" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fpcrasmus/48/20771_2.png" class="avatar"> FPCRasmus:</div>
<blockquote>
<p>ypeError: ‘bool’ object is not subscriptable</p>
</blockquote>
</aside>
<p>I think this is a problem related to <code>numpy</code>, not sure which version you’re using. I am using</p>
<pre><code class="lang-auto">np.__version__
'1.17.4'
</code></pre>
<p>and it does not have this problem. To confirm, here’s a snapshot of one of the masks:</p>
<pre><code class="lang-auto">[False, False, False, ..., False, False, False],
False, False, False, ..., False, False, False],
False, False, False, ..., False, False, False]]])
</code></pre>
<pre><code class="lang-auto">type(masks[0][0][0])
&lt;class 'numpy.bool_'&gt;
</code></pre>
<p>the first <code>0</code> is the mask index, and the two other zeros are the <code>xy</code> location in the mask.</p>
<p>if you are using an earlier <code>numpy</code> version, just upgrade it and test again.</p>
<p>NB. Note sure why in the ONE MASK example above your pixel/mask-values show a triplet in (note, <code>iris = (1,1,1) pupil = (2,2,2)</code>)!? Each generated mask should only have one channel .That being said, imagine the original mask that contains the labels (call it the color-encoded-mask), before converting to three binary masks, as a gray-level image with one-channel filed with <code>0s, 1s,</code> and <code>2s</code>. The two masks are generated from the color-encoded-mask by removing the background (the <code>0s</code>) using <code>obj_ids = np.unique(mask)[1:]</code>, followed by  <code>masks = mask == obj_ids[:, None, None]</code>.</p>

FPCRasmus: <p>Thank you very much! The mistake was indeed that I had still RGB images as masks.<br>
Converting them with the following solved it.</p>
<blockquote>
<p>mask = Image.open(mask_path).convert(‘L’)</p>
</blockquote>
<p>I can also now save my masks with:</p>
<pre><code class="lang-auto">img2 = Image.fromarray(prediction[0]['masks'][0, 0].mul(255).byte().cpu().numpy())
img2.save('1.png')

img1 = Image.fromarray(prediction[0]['masks'][1, 0].mul(255).byte().cpu().numpy())
img1.save('2.png')
</code></pre>
<p>Did you find any way to draw the results such as the tutorial <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-the-dataset" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-the-dataset</a><br>
With the bounding boxes and the outline?</p>

Deeply: <aside class="quote no-group" data-username="FPCRasmus" data-post="14" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fpcrasmus/48/20771_2.png" class="avatar"> FPCRasmus:</div>
<blockquote>
<p>Did you find any way to draw the results such as the tutoria</p>
</blockquote>
</aside>
<p>Glad it helped! This <a href="https://www.learnopencv.com/mask-r-cnn-instance-segmentation-with-pytorch/" rel="noopener nofollow ugc">one</a> worked for me</p>

AI-P-K: <p>Do you guys have a sample of your working code? I am dealing with the same problem.</p>

Deeply: <aside class="quote no-group" data-username="AI-P-K" data-post="16" data-topic="70476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ai-p-k/48/41172_2.png" class="avatar"> AI-P-K:</div>
<blockquote>
<p>sample of your working cod</p>
</blockquote>
</aside>
<p>this has been up for sometime; please try something similar to the bellow <em>__getitem___</em></p>
<pre><code class="lang-auto">def __getitem__(self, index):    
        img_file = self.coco.loadImgs(self.imgIds[index] )[0]                               
        image_A = Image.open(self.path2images + img_file['file_name'])        
        annIds = self.coco.getAnnIds(imgIds=img_file['id'], catIds=self.catIds, 
                                     iscrowd=None) # suppose all instances are not crowd        
        
        anns = self.coco.loadAnns(annIds)   
        num_objs=len(anns)
                
        boxes=[]; labels=[]; area=[]               
        masks = np.zeros((num_objs, img_file['height'], img_file['width'] ) ) # just getting the shape of the mask
        for i in range(num_objs):
            labels.append(anns[i]['category_id'])            
            masks[i,:,:] = self.coco.annToMask(anns[i])            
            # boxes.append(anns[i]['bbox']) # seems there is a problem in the boxes 
            # area.append(anns[i]['area']) # and areas
              
        '''  I am calculating the bboxes and areas from the masks
             as they might be incorrect, I've had a nan in maskrcnn's loss, 
             then after checking, seems that the area does not conform 
             with the bounding boxes. 
             But this is a problem of the dataset I'm using. 
            Feel free to skim if you're sure about your dataset.
        '''

        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])        
      
        boxes = torch.as_tensor(boxes, dtype=torch.float32)      
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        
        target = {}                
        target["boxes"]=  boxes
        target["labels"] = torch.as_tensor(labels, dtype=torch.int64) 
        target["masks"] = torch.as_tensor(masks, dtype=torch.uint8)
        target["image_id"] = torch.tensor([index]) 
        target["area"] =  area
        target["iscrowd"] = torch.zeros((num_objs,), dtype=torch.int64) # suppose all instances are not crowd             

        if self.transforms != None:
            image_A = self.transforms(image_A)                
        
        return image_A, target
   
</code></pre>

AI-P-K: <p>I am sorry, i think i am just an idiot if i follow the tutorial from <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.10.0+cu102 documentation</a> where do i have to make changes to add more classes for the mask rcnn model. I have a dataset containing png masks and trying to segment two classes 1. legs, 2. A4 paper. I am spinning in circle for few days to get a more straight forward answer and i can’t really find anything…</p>
<p>P.S the COLAB version from top of the page is not working</p>

Deeply: <p>You can manually check if your dataset class (let it be denoted as <code>TheeDatasetClass</code>) is working correctly before using Mask RCNN, this is what I usually do. Just create an instance of the dataset, and verify if you are getting the correct image, boxes, etc. Something like:</p>
<pre><code class="lang-auto">TheeDatasetClass x
zz = x[1]
</code></pre>
<p>So, if you are <code>Spyder</code> or <code>PyCharm</code>, you can check if <code>zz</code> has what you wanted.</p>
<p><strong>Tip</strong>: Always use debugging to make sure your code is working as you want it to, and this is why <code>PyTorch</code> is very handy, among other things. For example, if your dataset is causing a problem due to some out of your will reason, just hoover a for-loop over the entire dataset and check which indices are causing the problem.</p>

AI-P-K: <p>import os<br>
import numpy as np<br>
import torch<br>
import torch.utils.data<br>
from PIL import Image</p>
<p>class PennFudanDataset(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self, root, transforms=None):<br>
self.root = root<br>
self.transforms = transforms</p>
<h1>
<a name="load-all-image-files-sorting-them-to-1" class="anchor" href="#load-all-image-files-sorting-them-to-1"></a>load all image files, sorting them to</h1>
<h1>
<a name="ensure-that-they-are-aligned-2" class="anchor" href="#ensure-that-they-are-aligned-2"></a>ensure that they are aligned</h1>
<p>self.imgs = list(sorted(os.listdir(os.path.join(root, “Images”))))<br>
self.masks = list(sorted(os.listdir(os.path.join(root, “Masks”))))</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root, "Images", self.imgs[idx])
    mask_path = os.path.join(self.root, "Masks", self.masks[idx])
    img = Image.open(img_path).convert("RGB")
    # note that we haven't converted the mask to RGB,
    # because each color corresponds to a different instance
    # with 0 being background
    mask = Image.open(mask_path)

    mask = np.array(mask)
    # instances are encoded as different colors
    obj_ids = np.unique(mask)
    # first id is the background, so remove it
    obj_ids = obj_ids[1:]
    #print(obj_ids)

    # split the color-encoded mask into a set
    # of binary masks
    masks = mask == obj_ids[:, None, None]

    # get bounding box coordinates for each mask
    num_objs = len(obj_ids)
    boxes = []
    for i in range(num_objs):
        pos = np.where(masks[i])
        xmin = np.min(pos[1])
        xmax = np.max(pos[1])
        ymin = np.min(pos[0])
        ymax = np.max(pos[0])
        boxes.append([xmin, ymin, xmax, ymax])

    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    # there is only one class
    # labels = torch.ones((num_objs,), dtype=torch.int64)
    **labels = torch.as_tensor(obj_ids, dtype=torch.int64)**
    #print('Labels Before',labels)
    **labels[labels == 3] = 2**
    #print('Labels After',labels)
    masks = torch.as_tensor(masks, dtype=torch.uint8)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
  
    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["masks"] = masks
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
        img, target = self.transforms(img, target)

    return img, target

def __len__(self):
    return len(self.imgs)
</code></pre>
<p>So as far as i understood to train Mask RCNN with multiple classes there are two things to change in the code.</p>
<ol>
<li>In the dataset class replace the existent labels with → labels = torch.as_tensor(obj_ids, dtype=torch.int64)</li>
<li>Set the num_classes = number of clasess + 1(background).<br>
e.g, if we have two classes Pedestrian, Car than num_classes = 3</li>
</ol>
<p>Is this correct?</p>