Creating and training a custom network based on pretrained Faster-RCNN

username: Arda_Tumay
cooked: <p>I’m trying to create a custom network with pretrained fasterrcnn_resnet50_fpn from torchvision. The aim is to insert new layers between fpn and rpn. For this, I create a new nn.Module class and divide the original model into two as shown in below snippet.</p>
<pre><code class="lang-auto">from torchvision.models.detection import fasterrcnn_resnet50_fpn

fasterRcnn = fasterrcnn_resnet50_fpn(pretrained=False, progress=True, num_classes=15, pretrained_backbone=True)

......

class CustomFasterRcnn(nn.Module):

  def __init__(self):
    super(CustomFasterRcnn, self).__init__()

    self.resnet50WithFpn = nn.Sequential(*list(fasterRcnn.children())[0:2])
    self.RPN = nn.Sequential(*list(fasterRcnn.children())[2:])

  def forward(self, x):

    x = self.resnet50WithFpn(x)
    x = self.RPN(x)

    return x
</code></pre>
<p>As long as what I saw from internet, its forward method takes 2 arguments, self and x. However, in official docs of pytorch, <a href="https://pytorch.org/vision/stable/models.html#id57" class="inline-onebox" rel="noopener nofollow ugc">torchvision.models — Torchvision 0.11.0 documentation</a>, it is written that model takes two arguments during training which are images, targets.</p>
<p>Currently, it is giving error becase of lack of argument which is target. So, should I change the forward method to get 3 arguments, self, x and targets? If so, where should I pass targets argument in method?</p>
<p>Note: I am going to initialize new layers in <strong>init</strong> method and use these layers in the <strong>forward</strong> method in between <code>self.resnet50WithFpn(x)</code> and <code>self.RPN(x)</code></p>

username: ptrblck
cooked: <p>Splitting a model into <code>nn.Sequential</code> containers would work for simple models, which are initializing and using all submodules in a sequential way without any usage of the functional API.<br>
For quite simple models such as ResNets this approach would already miss <a href="https://github.com/pytorch/vision/blob/1cbd9cde35eb56129f29e45f6bad94fddc5eb8be/torchvision/models/resnet.py#L277">this</a> <code>torch.flatten</code> operation.<br>
More advanced models such as <code>FasterRCNN</code> will most likely break using this approach.<br>
You can find the <code>forward</code> definition of the base class <a href="https://github.com/pytorch/vision/blob/1cbd9cde35eb56129f29e45f6bad94fddc5eb8be/torchvision/models/detection/generalized_rcnn.py#L50-L119">here</a>.</p>

username: Arda_Tumay
cooked: <p>So are you saying that there is no easy approach to modify <code>FasterRCNN</code> to split its parts and add new layers this way?</p>
<p>It seems I need to inherit from its base class to modify its behavior. Am I right?</p>
<p>Or can you suggest a new way to do such a thing?</p>
<p>What I actually want to achieve is to add new <code>conv2d-relu</code> layers in between <code>fpn</code> and <code>rpn</code> to get proposals to be classified by head.</p>

username: ptrblck
cooked: <p>Yes, I don’t think splitting the entire model into <code>nn.Sequential</code> container would easily work (certain, “simple” submodules could work).</p>
<aside class="quote no-group" data-username="Arda_Tumay" data-post="3" data-topic="135906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arda_tumay/48/43092_2.png" class="avatar"> Arda_Tumay:</div>
<blockquote>
<p>It seems I need to inherit from its base class to modify its behavior. Am I right?</p>
</blockquote>
</aside>
<p>Yes, I believe this would be the proper way. To do so, check the second link, which points to the base class.</p>
<aside class="quote no-group" data-username="Arda_Tumay" data-post="3" data-topic="135906">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arda_tumay/48/43092_2.png" class="avatar"> Arda_Tumay:</div>
<blockquote>
<p>What I actually want to achieve is to add new <code>conv2d-relu</code> layers in between <code>fpn</code> and <code>rpn</code> to get proposals to be classified by head.</p>
</blockquote>
</aside>
<p>Assuming these layers won’t change the activation shape (which would thus allow to add them without modifying other layers too), you could try to replace the <code>fpn</code> with an <code>nn.Sequential</code> module containing the <code>fpn</code> as well as the additional layers.</p>