Loading with datasets.CocoDetection - how to get binary mask?

Nicholas_Wickman: <p>torchvision.datasets.CocoDetection returns tensors for images a list of tensors for the segmentations in each image. I’m struggling to understand how to work with this for semantic segmentation training.</p>
<p>I think I want to convert this list of segmentations into binary masks, but I’m having trouble figuring out how.</p>
<p>Can somebody help me?</p>

mderakhshani: <p>You can use from <a href="https://github.com/cocodataset/cocoapi" rel="nofollow noopener">coco api</a> to load mask related to each annotation using <a href="https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py#L426" rel="nofollow noopener">annToMask</a> function.</p>

Nicholas_Wickman: <p>Do you know how I can use that in conjunction with the dataloader to get a batch of masks?</p>

mderakhshani: <p>It is so simple.You can just add your snippet to the <strong>getitem</strong> function in the dataloader. It will bachify for you!</p>

7029279: <p>I tried to use annToMask like this below. I modified my model that originally used VOC Pascal dataset. Now I am using COCO 2014.</p>
<pre><code class="lang-auto">def __getitem__(self, index):
    raw_img, anno_class_img = self.pull_item(index)
    return raw_img, anno_class_img

def pull_item(self, index):
    coco = self.coco
    img = coco.loadImgs(self.id_list[index])[0]
    image_file_path = "./{}2014-2/{}".format(self.phase, img["file_name"])
    raw_img = Image.open(image_file_path)
    raw_img = raw_img.convert('RGB')   

    cat_ids = coco.getCatIds() 
    anns_ids = coco.getAnnIds(imgIds=img['id'], catIds=cat_ids, iscrowd=None)
    anns = coco.loadAnns(anns_ids)
    
    mask = coco.annToMask(anns[0])
    for i in range(len(anns)):
        mask += coco.annToMask(anns[i])
    anns_img = Image.fromarray(mask)
        
    raw_img = self.transform(raw_img)
    anns_img = self.transform(anns_img)
    return raw_img, anns_img
</code></pre>
<p>Below is parts of my training function</p>
<pre><code class="lang-auto">for images, labels in dataloaders_dict[phase]:
                if images.size()[0] == 1:
                    continue
                images = images.to(device)
                labels = torch.squeeze(labels)
                labels = labels.to(device)
                if (phase == 'train') and (count == 0):
                    optimizer.step()
                    optimizer.zero_grad()
             
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = net(images)
                    loss = criterion(outputs, labels.long())
</code></pre>
<p>But this is not working as the loss is way too low and when I do inference on test pictures, it classify all pixels as black, which I guess is non-object or background. I have no idea what to do. Any suggestion?<br>
|| Loss: 0.2654 || 10iter: 45.0074 sec.<br>
|| Loss: 0.0304 || 10iter: 34.7093 sec.<br>
|| Loss: 0.0018 || 10iter: 53.4208 sec.<br>
|| Loss: 0.0001 || 10iter: 34.5700 sec.</p>
<pre><code class="lang-auto">
        for category in anns:
            seg_rle = category['segmentation']
            tmp = decode(frPyObjects(seg_rle, raw_img.size[1], raw_img.size[0]))
            if tmp.ndim == 3:
                tmp = np.sum(tmp, axis=2, dtype=np.uint8)
            category['segmentation'] = tmp
        for category in anns:
            pilImg = Image.fromarray(category['segmentation'])
            anns_img = pilImg.resize((raw_img.size[1], raw_img.size[0]), resample=Image.NEAREST)
</code></pre>
<p>I tried this variation for my dataloader but with no luck,</p>