Change backbone in MaskRCNN

Bernd: <p>Hello<br>
I have a Mask RCNN using ResNet50, that works fine, except that it very slow and very big.<br>
It runs out of GPU Memory as soon as I set the batch_size to more than 2(!).</p>
<p>Since I do not have 1000 different classes to detect &amp; classify (like ImageNet), but only 50, I was wondering if a smaller backbone would not be a better fit!?<br>
So I want to test different backbones.</p>
<p>In the pedestrian tutorial I find a part that explains how to exchange the backbone:<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#modifying-the-model-to-add-a-different-backbone" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#modifying-the-model-to-add-a-different-backbone</a><br>
This sample does the modification for FasterRCNN (doc error?), but it seems works fine for Mask RCNN.<br>
But doesn’t it need a mask_roi_pooler too? If so how?</p>
<p>I tried the same adaptations for AlexNet and VGG16, but I failed.<br>
Could anyone help me to get AlexNet or VGG in there?</p>
<p>Regards,<br>
Bernd</p>

bsridatta: <p>Why not just replace resnet 50 layers with 18. Also, try PyTorch mixed-precision, it reduces the memory used a lot, maybe by half.</p>

Bernd: <p>Yes, I tried it and resnet18 works fine <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
And I will try the mixed precision, if I find some howto…</p>
<p>I still would love to experiment with very small or even self made backbones.<br>
Its not only that my use case has 50 instead of 1000 classes, its also that I have very few samples only.<br>
So it is essiential that I reduce the number of weights/parameters to a minimum, to have a fast learning  model.</p>
<p>I have tried the 50 classes without detection (just as cutout images) and could train them with AlexNet or VGG10 with 15 samples each class (and a lot of image augmentation) with 80-85% accuracy. And now I would like to do the same with the Mask RCNN around it to detect the objects first.</p>
<p>I had the impression that the backbone can easily be exchanged, but it seems the adaptation is very specific and not obvious. Therefore I need some help here.</p>

bsridatta: <p>There is an official example for PyTorch’s amp.</p>
<p>If I were in your place, I shall stick to the larger model itself. They are have already learned a lot of patterns and even though they take more time per epoch, they only need a few of them to give good results. I would strongly suggest you first try freezing all blocks except the last and finetune on your dataset. That’s the best way forward. This should be much much faster to train too.</p>
<p>Irrespective of number of classes, the models should learn a ton of features and should be able to generalize. I would say only a small portion of the last layers would be focusing on the class level patterns.</p>
<p>I hope this helps.</p>

Bernd: <p>AMP helped a lot here!<br>
It reduced size and inreased performance.<br>
And since I can now use batch_size 8 it really speeds up everything!<br>
Thanks a lot for this advise! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Still, if anyone has ever succeded in getting AlexNet or VGG to run inside MaskRCNN, please show me how.<br>
Even if it does not help, it would help me to convince my boss that this is not a solution.</p>

bsridatta: <p>Thanks for the update, great to know.</p>