Questions about Mask RCNN

username: M_S
cooked: <p>Hi, I want to use <code>torchvision.models.detection.maskrcnn_resnet50_fpn</code></p>
<p>I have a use case of two classes (and background), so I’m invoking it with arguments <code>pretrained=False, num_classes=3, pretrained_backbone=True</code>.</p>
<p>My questions are:</p>
<ol>
<li>
<p>If the backbone is pretrained, does this mean I need to transform my input images to have the same distribution as the images that the original model was trained on?<br>
1.1 If so, how do I use it? What are the means and the std’s? Do I need to transform the data before forwarding, during the dataset creation (as with the PyTorch ResNet FCN model for semantic segmentation, pretrained on ImageNet, with <code>mean = [0.485, 0.456,0.406]</code> and <code>std = [0.229, 0.224, 0.225]</code>), or do I need to pass arguments <code>image_mean, image_std</code> to the MaskRCNN class?</p>
</li>
<li>
<p>From the docs,</p>
</li>
</ol>
<blockquote>
<ul>
<li>boxes ( <code>FloatTensor[N, 4]</code> ): the ground-truth boxes in <code>[x1, y1, x2, y2]</code> format, with values between <code>0</code> and <code>H</code> and <code>0</code> and <code>W</code></li>
</ul>
</blockquote>
<p>Shouldn’t it be <code>H - 1</code> and <code>W - 1</code>?</p>
<ol start="3">
<li>From the docs,</li>
</ol>
<blockquote>
<ul>
<li>masks ( <code>UInt8Tensor[N, H, W]</code> ): the segmentation binary masks for each instance</li>
</ul>
</blockquote>
<p>By binary mask the meaning is a matrix of zeros, and ones where the mask of instance exists? If so, why do we need to also pass boxes? Maybe I misunderstood, but can’t the boxes be calculated by the minimal and maximal coordinates of ones, per dimension? And also, wouldn’t it be safer to be a Boolean tensor?</p>
<p>Thanks.</p>

username: spanev
cooked: <p>Hi <a class="mention" href="/u/m_s">@M_S</a>,</p>
<ol>
<li>The backbone is ResNet50 pretrained on ImageNet and the samples need to be normalized as you said. You should transform the data as it is done <a href="https://github.com/pytorch/examples/blob/ee964a2eeb41e1712fe719b83645c79bcbd0ba1a/imagenet/main.py#L202" rel="nofollow noopener">here</a>.</li>
<li>Yes , [0, H) and [0, W).</li>
<li>There are N masks of the size of the image. Binary mask means that the pixel values are 0 everywhere except where the relevant instance is, where they are 1.<br>
<code>UInt8Tensor</code> is basically a boolean.</li>
</ol>
<p>Hope this answers your questions!</p>

username: M_S
cooked: <p>Thanks <a class="mention" href="/u/spanev">@spanev</a></p>
<ol>
<li>So what are the arguments <code>image_mean, image_std</code> used for?</li>
<li>Why should we pass boxes if we already pass segmentation masks? The boxes can be calculated from the maps. Am I missing something?</li>
</ol>

username: spanev
cooked: <ol>
<li>Looking at the code again, the transform is already defined in the FasterRCNN from which M-RCNN inherits. As you suggested, you only need to provide the right <code>image_mean</code> and <code>image_std</code> when instantiating <code>maskrcnn_resnet50_fpn</code> ( <code>image_mean = [0.485, 0.456,0.406]</code> and <code>image_std = [0.229, 0.224, 0.225]</code>).<br>
You can see that the transform is declared <a href="https://github.com/pytorch/vision/blob/09823951fec09215f9efc5d5d31456763da5ae04/torchvision/models/detection/faster_rcnn.py#L231" rel="nofollow noopener">here</a> and used <a href="https://github.com/pytorch/vision/blob/09823951fec09215f9efc5d5d31456763da5ae04/torchvision/models/detection/generalized_rcnn.py#L47" rel="nofollow noopener">here</a>.</li>
</ol>
<p>You can see it when printing the network:</p>
<pre><code class="lang-auto">MaskRCNN(
  (transform): GeneralizedRCNNTransform()
  [...]
</code></pre>
<ol start="2">
<li>We pass masks of size HxW (where H is the height of the image and W is the width of the image). There is no information about the bbox in the mask. MaskRCNN applies a RoIAlign to the full mask during the target sampling phase to get the actual RoI mask.</li>
</ol>

username: M_S
cooked: <p>You say there is no information of the bbox in the mask, but isn’t the bbox essentially defined by the minimal and maximal values of x and y containing 1 in the mask?</p>

username: spanev
cooked: <p>Sure, if you crop a mask w.r.t. to its corresponding bbox, you will have the mask.</p>
<p>During the training, the RPN proposes RoIs that are not necessarily aligned with the bbox (RoIs have a fixed squared size and we’re trying to get match them with regression*), but we have to crop the mask of the instance we are learning anyway: hence the full size HxW masks for each instance.</p>
<p>* Find more info about the bbox regression in the <a href="https://arxiv.org/pdf/1504.08083.pdf" rel="nofollow noopener">Fast-RCNN paper</a>, Section 2.3 and Figure 1.</p>

username: M_S
cooked: <p>I agree completely but that’s not what I meant.<br>
I mean there is a simple algorithm to calculate a bbox from a segmentation map:<br>
x1 = smallest x that has 1 in map<br>
y1 = smallest y that has 1 in map<br>
x2 = highest x that has 1 in map<br>
y2 = highest y that has 1 in map<br>
so I don’t understand why we pass bboxes during training. We don’t need to pass them. They are easily calculated from the maps</p>

username: M_S
cooked: <p>Hi, asking again.<br>
What’s the point of passing bboxes if we already pass masks?</p>

username: tom
cooked: <p>It’s more efficient to calculate them in advance for training. I think maskrcnn uses them before predicting the mask.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Deeply
cooked: <p>I think this gives an option to the user to save the bboxes(s) in file(s) and load them along with the mask, hence there is no need to calculate bboxes, and that will save some computation time.</p>