Feature extracting from Resnet pretrained on COCO

username: Sam_Lerman
cooked: <p>I need a Resnet pretrained on COCO, so I opted to use:</p>
<p><code>models.detection.fasterrcnn_resnet50_fpn(pretrained=True)</code></p>
<p>However, all I need are the Resnet components for simple classification. How do I isolate that model without all of the extra FasterRCNN parts?</p>

username: Sam_Lerman
cooked: <p>Would just using that model’s <code>.backbone.body</code> work?</p>

username: ptrblck
cooked: <p>Yeah, but not out of the box.<br>
Note that the FasterRCNN ResNet backbone uses <code>FrozenBatchNorm</code> layers, while you would most likely want to use the trainable batchnorm layers.<br>
Also, the last linear layer is missing and the output is different.</p>
<p>However, there might be a hacky way to grab the backbone’s trained parameters.</p>
<p>First you would have to check for all <code>FrozenBatchNorm</code> layers and replace them with their equivalent <code>nn.BatchNorm2d</code> layers.<br>
Then small fixes in the architecture naming (<code>model.bn1.bn1</code> in particular) and finally you could create a reference <code>resnet50</code> model and load the trained parameters from the FasterRCNN backbone.</p>
<p>Here is the code for this work flow:</p>
<pre><code class="lang-python"># Create the model
model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
resnet = model.backbone.body

# Check for all FrozenBN layers
bn_to_replace = []
for name, module in resnet.named_modules():
    if isinstance(module, torchvision.ops.misc.FrozenBatchNorm2d):
        print('adding ', name)
        bn_to_replace.append(name)


# Iterate all layers to change
for layer_name in bn_to_replace:
    # Check if name is nested
    *parent, child = layer_name.split('.')
    # Nested
    if len(parent) &gt; 0:
        # Get parent modules
        m = resnet.__getattr__(parent[0])
        for p in parent[1:]:    
            m = m.__getattr__(p)
        # Get the FrozenBN layer
        orig_layer = m.__getattr__(child)
    else:
        m = resnet.__getattr__(child)
        orig_layer = copy.deepcopy(m) # deepcopy, otherwise you'll get an infinite recusrsion
    # Add your layer here
    in_channels = orig_layer.weight.shape[0]
    bn = nn.BatchNorm2d(in_channels)
    with torch.no_grad():
        bn.weight = nn.Parameter(orig_layer.weight)
        bn.bias = nn.Parameter(orig_layer.bias)
        bn.running_mean = orig_layer.running_mean
        bn.running_var = orig_layer.running_var
    m.__setattr__(child, bn)
    
    
print(resnet)

# Fix the bn1 module to load the state_dict
resnet.bn1 = resnet.bn1.bn1

# Create reference model and load state_dict
reference = models.resnet50()
reference.load_state_dict(resnet.state_dict(), strict=False)
</code></pre>
<p>Note that I haven’t properly tested this use case, so let me know, if that works for you.</p>

username: Sam_Lerman
cooked: <p>Thank you so much. About to test it now. Does this account for every resnet parameter, or will it leave some layers untrained? Is the final fc layer the only one that’s not accounted for by this?</p>

username: Sam_Lerman
cooked: <p>It seems to work! Now, I would like to remove the <code>avgpool</code>, so I defined an <code>Identity</code> module and replaced it with that. However, now I don’t know what input size to define the subsequent linear layer with. Before, I could just do <code>fc.in_features</code>.</p>
<p>Is there a way to get the output shape of <code>layer4[2].conv3</code>?</p>

username: Sam_Lerman
cooked: <p>Got it to work with <code>num_ftrs = 2048 * 7 * 7</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Thank you!</p>

username: ptrblck
cooked: <p>Awesome, good to hear you got it working!<br>
Let me know, if you see any issues with it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Sam_Lerman
cooked: <p>Just checking, could any of this interfere with loading and saving state_dicts? I’ve been having a weird bug with that.</p>

username: ptrblck
cooked: <p>What kind of error are you seeing?<br>
Note that I’ve used <code>strict=False</code> while loading the <code>state_dict</code> to the reference model, as the FasterRCNN ResNet is missing the last linear layer.</p>