How to define the labels for multi-class segmentation in TORCHVISION OBJECT DETECTION FINETUNING TUTORIAL

username: Deeply
cooked: <p>I am trying the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="nofollow noopener">Object Detection Finetuing tutorial</a>, which is very nice, smooth and helpful. I think there is a little bug in the <code>labels</code>, as they should mimic " <code>labels (Int64Tensor[N])</code> : the label for each bounding box", or more plausibly, " <code>labels (Int64Tensor[N])</code> : the label for each object". Clearly, the code works well with the Fudan dataset as it only has one object, ie person.  If I am correct, then</p>
<pre><code class="lang-auto">labels = torch.ones((num_objs,), dtype=torch.int64)
</code></pre>
<p>should be replaced with the following:</p>
<p><code>labels = torch.as_tensor(obj_ids, dtype=torch.int64)</code></p>
<p>in</p>
<p><code>__getitem__</code> of <code>class PennFudanDataset(object)</code></p>

username: ptrblck
cooked: <p><code>num_objs</code> is defined as <code>num_objs = len(obj_ids)</code>, while <code>obj_ids</code> is created via:</p>
<pre><code class="lang-python">obj_ids = np.unique(mask)
# first id is the background, so remove it
obj_ids = obj_ids[1:]
</code></pre>
<p>so I assume your suggestion should work for your described use case.<br>
Would you mind creating an issue <a href="https://github.com/pytorch/tutorials/issues">here</a> and describe the problem? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Deeply
cooked: <p>Now I think what I am proposing is also incorrect, as <code>labels</code> should contain, probably, the <code>object_id</code> or <code>label_id</code> and the number times each object is repeated in the mask. For example, having two classes with 3 objects of the first class and 5 object of the second class, one might need to use the following structure (background is dropped of course).</p>
<pre><code class="lang-auto">[1 1 1; 2 2 2 2 2]
</code></pre>
<p>I guess such labeling structure is used for evaluation using <code>pycocotools</code>. I have tried both the original form and the one I posted above and they both work, but I am concerned that they are both incorrect for my specific problem having 60 difference classes.</p>
<p>Hence, before reporting an issue, I am going to edit the title of the question and see if someone else has come up with an idea of how to deal with the labels.</p>

username: Stihl
cooked: <p>Hi<br>
Have you solved this issue?<br>
Can’t uderstand how data should look like if I have several objects for one class<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/torchvision/models.html?highlight=fasterrcnn_resnet50_fpn#torchvision.models.detection.fasterrcnn_resnet50_fpn" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/torchvision/models.html?highlight=fasterrcnn_resnet50_fpn#torchvision.models.detection.fasterrcnn_resnet50_fpn" target="_blank" rel="noopener nofollow ugc">torchvision.models — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<ul>
<li>boxes (<code>FloatTensor[N, 4]</code>): the ground-truth boxes in <code>[x1, y1, x2,y2]</code> format, with values of <code>x</code> between <code>0</code> and <code>W</code> and values of <code>y</code> between <code>0</code> and <code>H</code>
</li>
<li>labels (<code>Int64Tensor[N]</code>): the class label for each ground-truth box<br>
So the model should take N boxes of size 4, where N is the number of classes, I can’t understand how to pass several objects of one class. Please help</li>
</ul>

username: Deeply
cooked: <aside class="quote no-group" data-username="Stihl" data-post="4" data-topic="74360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/4af34b/48.png" class="avatar"> Stihl:</div>
<blockquote>
<p>the class label for each ground-truth b</p>
</blockquote>
</aside>
<p>This worked for me:</p>
<pre><code class="lang-auto"># instances are encoded as different colors
        obj_ids = np.unique(mask)[1:] # first id is the background, so remove it     
        masks = mask == obj_ids[:, None, None] # split the color-encoded mask into a set of binary masks

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)                       
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        # convert everything into torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)      
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
</code></pre>

username: PaulOmo
cooked: <p>So what happened to this path of the code?</p>
<pre><code>    masks = torch.as_tensor(masks, dtype=torch.uint8)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

    target = {}
    target["boxes"] = boxes
    target["labels"] = labels
    target["masks"] = masks
    target["image_id"] = image_id
    target["area"] = area
    target["iscrowd"] = iscrowd

    if self.transforms is not None:
        img, target = self.transforms(img, target)

    return img, target</code></pre>

username: Deeply
cooked: <p>My bad not to post the rest (although I think your code snippet is correct).</p>
<pre><code class="lang-auto">        # instances are encoded as different colors
        obj_ids = np.unique(mask)[1:] # first id is the background, so remove it     
        masks = mask == obj_ids[:, None, None]  # split the color-encoded mask into a set of binary masks
        num_objs = len(obj_ids)                       
        boxes = []
        # find the boxes
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        # convert everything into torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)      
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        
        target = {}
        target["boxes"] = boxes
        target["labels"] = torch.as_tensor(obj_ids, dtype=torch.int64) # corrected by Rawi
        target["masks"] = torch.as_tensor(masks, dtype=torch.uint8) #uint8
        target["image_id"] = torch.tensor([index]) 
        target["area"] = area
        target["iscrowd"] = torch.zeros((num_objs,), dtype=torch.int64) # suppose all instances are not crowd

        if self.transforms is not None:
              img, target = self.transforms(img, target)

        return img, target

</code></pre>

username: PaulOmo
cooked: <p>Thanks so much. I gonna try this now and let you know. Thanks again. Between I have about 38 classes excluding the background(black) and not all images have all 38 classes in them. Does still work?</p>
<p>Thanks.</p>

username: Deeply
cooked: <aside class="quote no-group" data-username="PaulOmo" data-post="8" data-topic="74360">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a698b9/48.png" class="avatar"> PaulOmo:</div>
<blockquote>
<p>I have about 38 classes excluding the background(black) and not all images have all 38 classes in them</p>
</blockquote>
</aside>
<p>Absolutely. This is the case in most instance segmentation cases. Good luck.</p>

username: PaulOmo
cooked: <p>Thanks a lot. You are gem. Will update you.</p>

username: PaulOmo
cooked: <p>Sorry to trouble you. Would you by any chance know why am getting this error. Image below.<br>
Step I took.</p>
<ol>
<li>
<p>I used the cold as you suggested.</p>
</li>
<li>
<p>I called the function and passed in the path to the dataset and mask.<br>
dataset_sample = PersonDataset(‘C:/Users/LENOVO/measurement_model_dev/Train’)</p>
</li>
<li>
<p>tried to print the first data using:<br>
img, target = dataset_sample[1]</p>
</li>
</ol>
<p>I ended up with this error. see image.</p>
<p>Any Help will appreciated.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/414f6e931f414485c47e7814793f9edde6fffdcd.png" data-download-href="https://discuss.pytorch.org/uploads/default/414f6e931f414485c47e7814793f9edde6fffdcd" title="error_image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414f6e931f414485c47e7814793f9edde6fffdcd_2_690x303.png" alt="error_image" data-base62-sha1="9jLduSgDHaAMJsBByKp8R4HBbGJ" width="690" height="303" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414f6e931f414485c47e7814793f9edde6fffdcd_2_690x303.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414f6e931f414485c47e7814793f9edde6fffdcd_2_1035x454.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414f6e931f414485c47e7814793f9edde6fffdcd_2_1380x606.png 2x" data-dominant-color="FBFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_image</span><span class="informations">2456×1080 75.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Deeply
cooked: <p>This is due to Numpy version. Works fine with me, mine is</p>
<pre><code class="lang-auto">np.__version__  
Out[13]: '1.17.4'
</code></pre>
<p>So, seems <code>np.where</code> will not subscript <code>bool</code> types in the future (or even now as happening with you).</p>
<p>Now, you don’t need to play with <code>numpy</code> package. Just replace</p>
<p><code>masks = mask == obj_ids[:, None, None] </code></p>
<p>with</p>
<p><code>masks = (mask == obj_ids[:, None, None]).astype(int)</code></p>
<p>and you’ll be good to go.</p>
<p>I’ve tried it, and I got the same result.</p>

username: PaulOmo
cooked: <p>Thanks a lot for all the help.</p>
<ol>
<li>
<p>I tired the code but having this error:<br>
AttributeError: ‘bool’ object has no attribute ‘astype’ as shown in the first image.</p>
</li>
<li>
<p>I also downgraded numpy to version 1.17.4 but had the same error.</p>
</li>
<li>
<p>I am thinking, should I even try to print the object or try to get the values (img or target) from img, target = dataset_sample[7]?</p>
</li>
<li>
<p>From my understanding, this line of code</p>
</li>
</ol>
<p>masks = mask == obj_ids[:, None, None]</p>
<p>turns multi-colored mask image into a True or False binary mask. And then uses this line:</p>
<p>pos = np.where(masks[i])</p>
<p>to do selection based on the boolean value.</p>
<ol start="5">
<li>I also did some reading on the np.where here: <a href="https://numpy.org/doc/stable/reference/generated/numpy.where.html" class="inline-onebox" rel="noopener nofollow ugc">numpy.where — NumPy v1.26 Manual</a></li>
</ol>
<p>and though np.where() uses 3 inputs I can’t relate it to the this:</p>
<p>pos = np.where(masks[i])</p>
<p>which seem to take in one value.</p>
<ol start="6">
<li>My thought<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a627993e9860ef7dcaf06ccde7910e9ad6a7807.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a627993e9860ef7dcaf06ccde7910e9ad6a7807" title="error_image2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a627993e9860ef7dcaf06ccde7910e9ad6a7807_2_690x303.png" alt="error_image2" data-base62-sha1="jKcS5Hny3IUFK0Q7xh6b3dLvL71" width="690" height="303" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a627993e9860ef7dcaf06ccde7910e9ad6a7807_2_690x303.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a627993e9860ef7dcaf06ccde7910e9ad6a7807_2_1035x454.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/a/8a627993e9860ef7dcaf06ccde7910e9ad6a7807_2_1380x606.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_image2</span><span class="informations">2456×1080 129 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  is to finish the entire code and see if I can train the model, maybe I am thinking it too much by trying to see what the data is inside the return from:<br>
img, target = dataset_sample[7].</li>
</ol>
<p>Any suggestions? or help will be appreciated.</p>
<p>And I do really appreciate the all time taken to reply and help.</p>
<p>Thank you.</p>

username: Deeply
cooked: <p>There was no need to downgrade <code>numpy</code>. Anyways, try this:</p>
<pre><code class="lang-auto">x=np.asarray([True, False]); 
print(x); x= x.astype(int); print(x)
[ True False]
[1 0]
</code></pre>
<p>It should work. I’ve managed to reproduce your errors using this naïve test:<br>
A)</p>
<pre><code class="lang-auto">x=True; np.where(x[0])
Traceback (most recent call last):

  File "&lt;ipython-input-42-5a36c5ff4d25&gt;", line 1, in &lt;module&gt;
    x=True; np.where(x[0])

TypeError: 'bool' object is not subscriptable
</code></pre>
<p>B)</p>
<pre><code class="lang-auto">x=True; np.astype(x)
Traceback (most recent call last):

  File "&lt;ipython-input-43-de5d8813d34d&gt;", line 1, in &lt;module&gt;
    x=True; np.astype(x)

AttributeError: module 'numpy' has no attribute 'astype'
</code></pre>
<p>I think there’s a problem with the type and size of your masks.<br>
Could you please check the type and size of your masks?</p>

username: PaulOmo
cooked: <p>Hmmm. I think I know where the issues is from your code explanation, but don’t know how to solve it yet.</p>
<p>The type of the mask is ‘bool’ should be a list of ‘bool’; like in your<br>
<code>x = [True, False]</code><br>
but in my case it’s just a ‘bool’.</p>
<p>I have an error when I tried to get the size with <code>.size</code>:</p>
<blockquote>
<p>AttributeError: ‘bool’ object has no attribute ‘size’</p>
</blockquote>
<p>and that is true as the type is bool and not a list.</p>
<p>I printed the image with pyplot as seen in the image below.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77684ac565772fecbea6e75f2ac3b936930da95a.png" data-download-href="https://discuss.pytorch.org/uploads/default/77684ac565772fecbea6e75f2ac3b936930da95a" title="error_image1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77684ac565772fecbea6e75f2ac3b936930da95a_2_545x499.png" alt="error_image1" data-base62-sha1="h2kglKtKy3GnV4Hph3Tp5SomNfA" width="545" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77684ac565772fecbea6e75f2ac3b936930da95a_2_545x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77684ac565772fecbea6e75f2ac3b936930da95a_2_817x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/7/77684ac565772fecbea6e75f2ac3b936930da95a_2_1090x998.png 2x" data-dominant-color="F0EFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_image1</span><span class="informations">1234×1132 70.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Deeply
cooked: <p>Not sure what and how 're you doing it. If you have a png image, you can get the mask from the alpha channel, then, convert it to a numpy array. Something like:</p>
<pre><code class="lang-auto">mask = image.getchannel('A') # getting the mask from the alpha channel (png image), 255 for foreground, 0 otherwise
mask = numpy.asarray(mask) 
mask = np.asarray(mask, dtype= 'int') # or uint8 
mask=mask/255 # 0 to 1 scale
</code></pre>

username: PaulOmo
cooked: <p>Hi,</p>
<p>Thanks a lot. I figured it with your help. It was the image, they were not in the right format. I had to pre-process them using this tutorial here.</p>
<p><a href="https://www.bulentsiyah.com/preprocessing-rgb-image-masks-to-segmentation-masks" rel="noopener nofollow ugc">https://www.bulentsiyah.com/preprocessing-rgb-image-masks-to-segmentation-masks</a></p>
<p>Thanks again.</p>

username: Anshumaan_Dash
cooked: <p>Hey,<br>
Were you trying to do multi class semantic segmentation or instance segmentation. If you were doing semantic segmentation, could you please explain what is the purpose of getting the bounding boxes!</p>
<p>Thanks.</p>

username: monster
cooked: <p>Hi, my question is quite different .</p>
<p>I am using detectron2 for model training with MaskRcnn and it needs the data in coco-json file so I also want to add images for training without any ground truth to reduce the false detection so how can I do it?</p>
<p>currently I am using labelme tool for image labelling</p>