Instance segmentation with torchvision

AndriPi: <p>Hi,</p>
<p>I would like to quickly build an instance segmentation model on a dataset I received, and I would like to try <code>torchvision</code> out, since it looks like the most user-friendly CV framework in PyTorch. Which models are available for the instance segmentation task in <code>torchvision</code>? It looks like the only one is</p>
<p><a href="https://pytorch.org/vision/stable/models.html#mask-r-cnn" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/models.html#mask-r-cnn</a></p>
<p>Is there a tutorial to use Mask-RCNN in <code>torchvision</code>? In particular, I need to know how to build the <code>DataLoader</code>. Which format must the annotations have?</p>

KFrank: <p>Hi Andri!</p>
<aside class="quote no-group" data-username="AndriPi" data-post="1" data-topic="112988" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/71c47a/48.png" class="avatar"> AndriPi:</div>
<blockquote>
<p>Is there a tutorial to use Mask-RCNN in <code>torchvision</code>?</p>
</blockquote>
</aside>
<p>The <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial</a> should give you<br>
what you need.</p>
<p>From memory, it’s a bit imperfect – maybe a little out of date with some<br>
minor inconsistencies – but torchvision’s <code>maskrcnn_resnet50_fpn</code><br>
does work, although you will have to do some some relatively<br>
straightforward work to adapt it to your use case.</p>
<p>Note that you get a pre-trained version “for free,” so you need a lot less<br>
data than if you were training it from scratch.</p>
<p>Best.</p>
<p>K. Frank</p>

AndriPi: <p>Hi K. Frank!</p>
<p>Thanks a lot for the pointer. I saw the tutorial, but given the title, I thought it would be on object detection, not on instance segmentation, and I didn’t read it. I’ll dig deep into the Colab version. Before I try that, however, are there any fundamentals I should learn? I have no previous experience with <code>torchvision</code> and PyTorch, but I have some experience with <code>fastai</code>, and quite a bit of experience with Keras and Tensorflow, so I don’t exactly start from scratch. What I did up to now was to pull the <a href="https://ngc.nvidia.com/" rel="noopener nofollow ugc">NGC</a> Docker image for PyTorch, and spin up the container. This is the beginning of my PyTorch journey <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

KFrank: <p>Hi Andri!</p>
<aside class="quote no-group" data-username="AndriPi" data-post="3" data-topic="112988" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/71c47a/48.png" class="avatar"> AndriPi:</div>
<blockquote>
<p>I have some experience with <code>fastai</code>, and quite a bit of experience with Keras and Tensorflow</p>
</blockquote>
</aside>
<p>At a general conceptual level, pytorch and kera/tensorflow do pretty<br>
much the same thing, so that’s a big part of what you need to learn<br>
and understand.  There are, however, significant differences in the<br>
architecture and the details of implementation.</p>
<p>You could just jump into the deep end with Mask-RCNN, and work<br>
through the documentation and ask questions here when issues arise.</p>
<p>If you like tutorials, you could work though pytorch’s <a href="https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html" rel="noopener nofollow ugc">A 60 Minute Blitz</a><br>
tutorial (and other tutorials and introductory documentation).  Whether<br>
or not you follow a tutorial, I would suggest that you build and train a<br>
simple toy-problem network in pytorch – just to get a feel for the<br>
pytorch way of doing things – before jumping straight to your real<br>
Mask-RCNN use case.</p>
<p>Best.</p>
<p>K. Frank</p>