Multi-class Multi-label image segmentation for beginners

username: Paleheart
cooked: <p>So I’m a self-admitted noob but with computer vision it looks like I dived straight into the deep end and I think I touched bottom</p>
<p>I started a couple of weeks ago with lobe.ai but quickly realized I needed way more. I moved to detectron2 and just got that working last night (using mask-rcnn to generate image segmentation masks) but I realized again that I need a little more.</p>
<p>Since it looks like detectron2 leans on pytorch, I figured Id ask here to</p>
<p>Is there a model out there that can do multi-label image segmentation? If there isnt is there a way to take mask-rcnn and change its classification layers to return multiple classes per mask region?</p>
<p>I could build and run multiple trained models side by side (like where I started with lobe) but the detectron2 models are kinda chunky and I dont want to load a few gigs of models to generate output</p>
<p>I know I’m asking questions way over my head, be gentle <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue_winking_eye.png?v=12" title=":stuck_out_tongue_winking_eye:" class="emoji" alt=":stuck_out_tongue_winking_eye:"></p>

username: KFrank
cooked: <p>Hi Andrew!</p>
<aside class="quote no-group" data-username="Paleheart" data-post="1" data-topic="144306" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/c68b51/48.png" class="avatar"> Paleheart:</div>
<blockquote>
<p>Is there a model out there that can do multi-label image segmentation? If there isnt is there a way to take mask-rcnn and change its classification layers to return multiple classes per mask region?</p>
</blockquote>
</aside>
<p>In short:  Yes, you can perform multi-label, multi-class image segmentation,<br>
and image-segmentation architectures can relatively easily be modified to<br>
do so (or already do so).</p>
<p>Let me draw a distinction between <em>instance segmentation</em> (which is what<br>
Mask R-CNN does), the easier problem of <em>semantic segmentation</em> (such<br>
as performed by U-Net), and <em>object detection</em> (which is “simpler” in that<br>
it doesn’t assign specific pixels to specific classes or instances).</p>
<p>In instance segmentation, you assign each pixel to a specific instance of<br>
an object of a given class.  Thus, you might say that this pixel belongs to<br>
the second person in the image and that pixel belongs to the third dog in<br>
the image.  So instance segmentation can naturally be multi-class (that<br>
is person vs. dog vs. pelican).  But it is less naturally multi-label.  That is,<br>
a given pixel would not typically be assigned to more than one class (not<br>
both a person and a dog).  (I could imagine inventing such a use case, but<br>
it would seem contrived.)</p>
<p>Note, that as it stands, Mask R-CNN can and does generate overlapping<br>
“mask regions” (by which I assume that you mean bounding boxes).  So<br>
bounding box 1 could be “person” and it could overlap with bounding box<br>
2, which could be “dog.”  The masks within the bounding boxes are then<br>
essentially independent of one another.  Although you would hope it<br>
wouldn’t, it could well turn out that the predicted mask for the person<br>
in bounding box 1 shares some pixels with the predicted mask for the<br>
dog in bounding box 2.  So, although not its typical use case, in some<br>
sense Mask R-CNN is already multi-label.</p>
<p>In the case of semantic segmentation (for which again a multi-label use<br>
case would be somewhat contrived), it is perfectly straightforward to take<br>
something like a (multi-class) U-Net, interpret its final per-pixel class<br>
predictions as multi-label predictions, train it against multi-label target<br>
(ground-truth) data, typically replacing a <code>CrossEntropyLoss</code> loss<br>
criterion with <code>BCEWithLogitsLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Paleheart
cooked: <p>Hi, K. Frank!</p>
<p>Thanks for writing back. I appreciate you putting up with my excited noob questions (I very much know that pain) and for really breaking things down. I didnt even realize the difference in instance vs semantic segmentation.</p>
<p>Originally I had thought of running multiple models against the image collecting the masks and combining them (kind of like I originally did with lobe.ai’s basic image classification). The models are kind of chunky though so that kind of turns into its own pain.</p>
<p>Thats why I asked if it was possible to modify like the model itself, since having one model do it would definitely be the simplest solution<br>
(as an aside though one of my predictions from my quick and dirty concept did have some slightly overlapping regions, but the second label for the zone never got predicted at all. Im using detectron2 and now Im wondering if it did some kind of data cleanup in there somewhere to remove the identical training polygons, I wonder if I could use random to jiggle them a tiny amount)</p>
<p>Maybe I could use a heavy model to predict a region and then run a multi-label object detection model over top of it to get additional features and see which region they mostly overlapped with.</p>
<p>Either way thanks for the great explanation! I very much appreciate the reply</p>

username: Paleheart
cooked: <p>Followup</p>
<p>Hey guess what K. Frank, detectron2 IS doing some kinda of data pruning, I jiggled my polygon a bit and it successfully predicted two regions on top of each other from one model.</p>
<p>It’s not exactly multi-label, but its close enough I can combine the regions and get the results I’m after I think!</p>