Confusion re: Faster-RCNN Mobilenet Example

Mertens: <p>Hi all, I have some confusion regarding the mobilenet example provided in the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py" rel="nofollow noopener">F-RCNN Code</a>.</p>
<p>In particular, copying the code as given in the example:</p>
<pre><code class="lang-auto">import torch
import torchvision
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection.rpn import AnchorGenerator
# load a pre-trained model for classification and return
# only the features
backbone = torchvision.models.mobilenet_v2(pretrained=False).features

# FasterRCNN needs to know the number of
# output channels in a backbone. For mobilenet_v2, it's 1280
# so we need to add it here
backbone.out_channels = 1280

# let's make the RPN generate 5 x 3 anchors per spatial
# location, with 5 different sizes and 3 different aspect
# ratios. We have a Tuple[Tuple[int]] because each feature
# map could potentially have different sizes and
# aspect ratios
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                   aspect_ratios=((0.5, 1.0, 2.0),))

# let's define what are the feature maps that we will
# use to perform the region of interest cropping, as well as
# the size of the crop after rescaling.
# if your backbone returns a Tensor, featmap_names is expected to
# be [0]. More generally, the backbone should return an
# OrderedDict[Tensor], and in featmap_names you can choose which
# feature maps to use.
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)

#put the pieces together inside a FasterRCNN model
model = FasterRCNN(backbone, num_classes=2, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

model.eval()
x = [torch.rand(3, 300, 400), torch.rand(3, 500, 400)]
predictions = model(x)
</code></pre>
<p>And running the code from a fresh docker (built last night) , gives the following error trace:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "mobileNetExample.py", line 38, in &lt;module&gt;
    predictions = model(x)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/models/detection/generalized_rcnn.py", line 71, in forward
    detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/models/detection/roi_heads.py", line 756, in forward
    box_features = self.box_roi_pool(features, proposals, image_shapes)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 539, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/ops/poolers.py", line 188, in forward
    self.setup_scales(x_filtered, image_shapes)
  File "/opt/conda/lib/python3.7/site-packages/torchvision/ops/poolers.py", line 161, in setup_scales
    lvl_min = -torch.log2(torch.tensor(scales[0], dtype=torch.float32)).item()
IndexError: list index out of range
</code></pre>
<p>Making either of the following changes lets the evaluate at least run through:</p>
<pre><code class="lang-auto">model = FasterRCNN(backbone, num_classes=2, rpn_anchor_generator=anchor_generator) #using the default roi_pooler
</code></pre>
<p>OR</p>
<pre><code class="lang-auto">roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                output_size=7,
                                                sampling_ratio=2)
</code></pre>
<p>Looking at the typing notes for the <a href="https://github.com/pytorch/vision/blob/master/torchvision/ops/poolers.py" rel="nofollow noopener">pooler</a>, it seems like the input should be a list[str], and it’s not clear to me how passing featmap_names=[0] is supposed to be handled.</p>
<p>Ultimately, I’m not sure how to build an F-RCNN following the mobile net example (e.g. using a backbone that returns a tensor rather than a dict). Is the example provided in the code actually correct? If not, would setting featmap_names=[‘0’] provide correct behavior, or is it necessary to wrap the outputs up into a singleton dict?</p>
<p>Thanks for any insight you might have!</p>

Mertens: <p>Just a quick update:</p>
<p>It looks as though at some point the MultiScaleRoIAlign constructor moved from using a list of indices to a list of keys for featmap_names, while the docstrings weren’t updated to reflect this.</p>
<p>This has been corrected in <a href="https://github.com/pytorch/vision/commit/bbaa6f3b19ad374ebefcf0efa30a07802e2d7723" rel="nofollow noopener">this commit</a>. The correct behavior was being emulated by</p>
<pre><code class="lang-auto">roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                output_size=7,
                                                sampling_ratio=2)
</code></pre>

chhaya_kumar_das: <p>Can you tell the version of Pytorch. Is it 1.5?</p>

Mertens: <p>I’d have to go back and double check the dates of the various versions to be sure, at the time I was using the latest stable build of both pytorch and pytorch vision. But, I believe this was using pytorch 1.5 and pytorch vision .4?</p>

chhaya_kumar_das: <p>Were you able to get the correct results out of the model? Because as it seems that the targets are always modified with simple forward pass. <a href="https://discuss.pytorch.org/t/torchvision-fasterrcnn-bounding-box-error/80985/6">Check this .</a> If possible can you tell me am I doing something wrong or is it a bug? You can comment on that post. Please excuse me for any mistakes. I am new to this.</p>

111317: <p>maybe you can choose torch1.4.0 and torchvision0.5.0 . this is a url for download:<a href="https://download.pytorch.org/whl/torch_stable.html" rel="nofollow noopener">https://download.pytorch.org/whl/torch_stable.html</a></p>

chhaya_kumar_das: <p>Will give it a try. Thanks bud</p>

bulatnv: <p>Thank you, Mertens.<br>
This definitely works for torch 1.7</p>

douglasrizzo: <p>I’m on PyTorch 1.9.0 and Torch Vision 0.10.0 and <code>[0]</code> (which is still in the tutorial) doesn’t work, while <code>['0']</code> does.</p>