Instance segmentation mask R-CNN change backbone - fine tuning

username: ser17
cooked: <p>Hi, I’m new in Pytorch and I’m using the torchvision.models to practice with semantic segmentation and instance segmentation.<br>
I have used mask R-CNN with backbone ResNet50 FPN ( <code>torchvision.models.detection.</code> <code>maskrcnn_resnet50_fpn</code>) for instance segmentation to find mask of images of car, and everything works well.</p>
<p>I thought that with a different backbone maybe I could reach better result, so I’m trying to change the backbone of Mask R-CNN with MobileNet v2 or ResNext pre-trained, following the instruction on this Pytorch documentation (<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#defining-your-model" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>), but with bad result, producing something like this<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f302e552d453afb1a616584bb2cc0155ec06f0e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9f302e552d453afb1a616584bb2cc0155ec06f0e" title="mask%20RCNN-resnet50-1-img1-wrong"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f302e552d453afb1a616584bb2cc0155ec06f0e_2_662x500.jpeg" alt="mask%20RCNN-resnet50-1-img1-wrong" data-base62-sha1="mIf8ZJ6iAnm48qLbeG6EgpbBZ70" width="662" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f302e552d453afb1a616584bb2cc0155ec06f0e_2_662x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f302e552d453afb1a616584bb2cc0155ec06f0e_2_993x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f302e552d453afb1a616584bb2cc0155ec06f0e.jpeg 2x" data-dominant-color="9EC1B0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">mask%20RCNN-resnet50-1-img1-wrong</span><span class="informations">1142×862 635 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I don’t know if either the models are incompatible (the weight of MobileNet v2/ResNext doesn’t match with mask R-CNN architecture) or I did something wrong in the implementation.<br>
This is the code I have used for instancing the models and backbone:</p>
<pre><code class="lang-auto"># import necessary libraries
from PIL import Image
import matplotlib.pyplot as plt
import torch
import torchvision.transforms as T
import torchvision
import torch
import numpy as np
import cv2
import random

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

from torchvision.models.detection import MaskRCNN
from torchvision.models.detection.rpn import AnchorGenerator

# load a pre-trained model for classification and return only the features
backbone = torchvision.models.mobilenet_v2(pretrained=True).features

# MaskRCNN needs to know the number of output channels in a backbone. For mobilenet_v2, it's 1280, so we need to add it here
backbone.out_channels = 1280

# let's make the RPN generate 5 x 3 anchors per spatial location, with 5 different sizes and 3 different aspect ratios. We have a Tuple[Tuple[int]] because each feature map could potentially have different sizes and aspect ratios
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                   aspect_ratios=((0.5, 1.0, 2.0),))

# let's define what are the feature maps that we will use to perform the region of interest cropping, as well as the size of the crop after rescaling. if your backbone returns a Tensor, featmap_names is expected to be [0]. More generally, the backbone should return an OrderedDict[Tensor], and in featmap_names you can choose which feature maps to use.

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)

# put the pieces together inside a MaskRCNN model
model = MaskRCNN(backbone,
                   num_classes=2,
                   rpn_anchor_generator=anchor_generator,
                   box_roi_pool=roi_pooler)

model.eval()
</code></pre>
<p>After there are function for color mask, get prediction and instance segmentation …<br>
and this is the code to preprocess and transform the image</p>
<pre><code class="lang-auto">img = Image.open('car1.jpg')

from PIL import Image
from torchvision import transforms

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
input_tensor = preprocess(img)
input_batch = input_tensor.unsqueeze(0) # create a mini-batch as expected by the model

img = preprocess(img)
pred = model([img])
</code></pre>
<p>Hope someone can help me solving this problem,<br>
thanks in advance!</p>

username: WaterKnight
cooked: <p>I think that you are missing this <a class="mention" href="/u/ser17">@ser17</a> :</p>
<pre><code class="lang-auto">roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                              output_size=7,
                                              sampling_ratio=2)

mask_roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                              output_size=14,
                                                              sampling_ratio=2)

model = MaskRCNN(backbone,
                          num_classes=2,
                         rpn_anchor_generator=anchor_generator,
                          box_roi_pool=roi_pooler,
                          mask_roi_pool=mask_roi_pooler)

</code></pre>

username: Jeffrey_Danowitz
cooked: <p>Question - In the tutorial<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial" rel="noopener nofollow ugc">Tutorial on adding backbones to RCNN</a><br>
I do not see a mention to the image normalizing:<br>
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])</p>
<p>Does this occur automatically somewhere in the data loader that was built for the PennFudanDataset or somewhere else? Do I need to add it manually somewhere? If someone can explain where this is, I’d appreciate it.</p>
<p>Thanks,<br>
Jeff</p>

username: vmirly1
cooked: <p>I think when you build a MaskRCNN object, the GeneralizedRCNNTransformation will be added as the first block of the model:</p>
<pre><code class="lang-auto">  (transform): GeneralizedRCNNTransform(                                              
      Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])        
      Resize(min_size=(800,), max_size=1333, mode='bilinear')                                              
  ) 
  (backbone): Sequential(
    (0): Sequential(
      (0): ConvBNActivation(
        (0): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): FrozenBatchNorm2d(16)
        (2): Hardswish()
      )
      (1): InvertedResidual(
        (block): Sequential(
...
</code></pre>
<p>This is added by the <code>MaskRCNN()</code>, as the backbone before passing to MaskRCNN() did not have this transform block.</p>