Index error using custom backbone on FasterRCNN

AdilZouitine: <p>Hello everyone, I’m following the object detection tutorial you’ll find <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#modifying-the-model-to-add-a-different-backbone" rel="noopener nofollow ugc">here</a>.</p>
<p>However I want to use in backbone a vgg11/Resnet18 instead of the MobileNet v2 of the tutorial.</p>
<p>For the VGG11 I replace :</p>
<pre data-code-wrap="py"><code class="lang-py">backbone = torchvision.models.mobilenet_v2(pretrained=True).features
backbone.out_channels = 1280
</code></pre>
<p>by:</p>
<pre data-code-wrap="py"><code class="lang-py">backbone = torchvision.models.vgg11_bn(pretrained=True, progress=True).features
backbone.out_channels = 512
</code></pre>
<p>For the ResNet18:</p>
<pre data-code-wrap="py"><code class="lang-py">resnet18 = torchvision.models.resnet18(pretrained=True)
backbone = torch.nn.Sequential(nn.Sequential(*list(resnet18.children())[:-2]))
backbone.out_channels = 512
</code></pre>
<p>I have the same error for both backbones:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/5/154acd4cf553c01dc3871f47dc669082e55f33df.png" data-download-href="https://discuss.pytorch.org/uploads/default/154acd4cf553c01dc3871f47dc669082e55f33df" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/154acd4cf553c01dc3871f47dc669082e55f33df.png" alt="image" data-base62-sha1="32mhWbs5aTu8FaX5Mm0TmjgtDeD" width="690" height="368" data-dominant-color="3F4042"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">969×518 26.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fef03561a817926550a9f3cf5beff73519524b1d.png" data-download-href="https://discuss.pytorch.org/uploads/default/fef03561a817926550a9f3cf5beff73519524b1d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fef03561a817926550a9f3cf5beff73519524b1d.png" alt="image" data-base62-sha1="AnhV9HeVu7CDtG6OHv0oJG9kY9f" width="690" height="204" data-dominant-color="3F4042"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">892×264 13.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you have an explanation, I’m interested .</p>
<p>Torch : 1.5.0<br>
Torchvision : 0.6.0</p>
<p>Thank you.</p>

emcp: <p>here is how I have been getting through with custom backbone networks for resnet 101</p>
<pre><code class="lang-auto">
class ModelResnet101FasterRCNN(FasterRCNN):
    def __init__(self, data_conf, model_conf):
        print("Creating model")
        backbone_nn = torchvision.models.__dict__[model_conf["hyperParameters"]["net"]](pretrained=True)
        # This line above yields the equivalent of ...
        # backbone_nn = torchvision.models.resnet101(pretrained=True)
        # OR
        # backbone_nn = torchvision.models.wide_resnet101_2(pretrained=True)

        modules = list(backbone_nn.children())[:-1]  # delete the last fc layer.
        backbone_nn = nn.Sequential(*modules)
        for param in backbone_nn.parameters():
            param.requires_grad = False

        # FasterRCNN needs to know the number of
        # output channels in a backbone. For resnet101, it's 2048
        backbone_nn.out_channels = 2048

</code></pre>

AdilZouitine: <p>Hello, I tried your solution but it doesn’t work.<br>
Here is the link google collab where I used your solution: <a href="https://colab.research.google.com/drive/1AeGt5ia1aOAQDMRGtjAjXIKV32Ni_fPx" rel="nofollow noopener">here</a></p>
<p>Thank you for your help.</p>

emcp: <p>looks like you are not done yet… when using a custom backbone neural network the tutorial states you need to also instantiate the pooling components and attach them to your model… I am not seeing that in your notebook</p>
<p>from the tutorial you need…</p>
<pre><code class="lang-auto"># FasterRCNN needs to know the number of
# output channels in a backbone. For mobilenet_v2, it's 1280
# so we need to add it here
backbone.out_channels = 1280

# let's make the RPN generate 5 x 3 anchors per spatial
# location, with 5 different sizes and 3 different aspect
# ratios. We have a Tuple[Tuple[int]] because each feature
# map could potentially have different sizes and
# aspect ratios
anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                   aspect_ratios=((0.5, 1.0, 2.0),))

# let's define what are the feature maps that we will
# use to perform the region of interest cropping, as well as
# the size of the crop after rescaling.
# if your backbone returns a Tensor, featmap_names is expected to
# be [0]. More generally, the backbone should return an
# OrderedDict[Tensor], and in featmap_names you can choose which
# feature maps to use.
roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)

# put the pieces together inside a FasterRCNN model
model = FasterRCNN(backbone,
                   num_classes=2,
                   rpn_anchor_generator=anchor_generator,
                   box_roi_pool=roi_pooler)
</code></pre>

Jimmy_Hall: <p>I got the same error following the tutorial exactly, but the FasterRCNN documentation:</p>
<p><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/detection/faster_rcnn.py</a></p>
<p>suggests that  the featmap_names in the MultiScaleROIAlign should be a character, ‘0’, not an integer. I replaced</p>
<pre><code class="lang-auto">roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)
</code></pre>
<p>with</p>
<pre><code class="lang-auto">roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=['0'],
                                                output_size=7,
                                                sampling_ratio=2)
</code></pre>
<p>and the error on foward pass went away. I haven’t verified that the model finetunes appropriately though.</p>
<p>Hope this helps.</p>

emcp: <p>My work is here <a href="https://github.com/JRGEMCP/bootstrap-pytorch-torchvision-fasterrcnn" rel="nofollow noopener">https://github.com/JRGEMCP/bootstrap-pytorch-torchvision-fasterrcnn</a></p>
<p>I PRd a hint in the documentation regarding that exact problem… but I am still not getting good results</p>

AdilZouitine: <p>Hey, this is the solution.<br>
This should be notified in the documentation.</p>
<p>Thanks to all of you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

emcp: <p>the PR is waiting here <a href="https://github.com/pytorch/tutorials/pull/979" rel="nofollow noopener">https://github.com/pytorch/tutorials/pull/979</a></p>