Using feature extraction layers from pre-trained FRCNN

Aviv_Shamsian: <p>Hi all,<br>
I have trained FRCNN using <code>torchvision.models.detection.fasterrcnn_resnet50_fpn</code> and now I want to use it’s feature extraction layers for something else.<br>
To do so I first printed <code>frcnn.modules()</code> and see that the model has 4 major components:<br>
0) GeneralizedRCNNTransform</p>
<ol>
<li>BackboneWithFPN</li>
<li>RPN</li>
<li>ROI Heads</li>
</ol>
<p>I tried first to use both first and second modules but ran into an error this is what I tried:</p>
<pre><code class="lang-auto">feature_net = nn.Sequential(*list(frcnn.modules()[:2])
a = torch.rand(4,3,256,256)
out = feature_net(a)
</code></pre>
<p>and got the following error:</p>
<pre><code class="lang-auto">TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple
</code></pre>
<p>After looking at <code>GeneralizedRCNNTransform</code> module I have noticed it just holds some metadata about the model, image mean, image std etc.</p>
<p>so I tried to do the same but with the (1) module only (BackboneWithFPN)</p>
<pre><code class="lang-auto">feature_net = nn.Sequential(*list(frcnn.modules()[1])
</code></pre>
<p>Then passing <code>a</code> worked fine but when trying to print <code>out.shape</code>  raised the following error:</p>
<pre><code class="lang-auto">AttributeError: 'collections.OrderedDict' object has no attribute 'shape'
</code></pre>
<p>out isn’t tensor as expected.<br>
How can it be solved?</p>
<p>Thank you in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>The output of the unmodified model is an <code>OrderedDict</code> containing the <code>boxes, labels, and scores</code>.<br>
Could you check your current output for these keys?</p>

Aviv_Shamsian: <p>Hi Patrick!<br>
I forgot to mention this I already did this check the keys of the output are ‘0’,‘1’,‘2’,‘pool’<br>
Maybe this is related to nested Sequential modules?</p>
<p>Thank you for helping!</p>

ptrblck: <p>It might be related to the wrapping in an <code>nn.Sequential</code> module.<br>
<code>nn.Sequential</code> modules are used for quite simple models and I assume the currently used one might not work out of the box using this approach.<br>
If you want to use some activations, I would recommend to use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> and process these activations further.</p>

Aviv_Shamsian: <p>I just want to use one specific children of the model (the one which response to feature extraction). I wrote by mistake <code>nn.modules</code> I tried to use <code>nn.children</code> and tried to use the second children in the list.</p>
<p>BTW, this model will be used for inference only W/O any training process.<br>
What is the right way to do so?</p>
<p>BR,<br>
Aviv</p>

ptrblck: <p>Which child module do you want to use?</p>
<pre><code class="lang-python">list(model.children())[1]
</code></pre>
<p>would return <code>BackboneWithFPN</code>. Do you want to use this module as a standalone model?</p>

Aviv_Shamsian: <p>Yup <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Shouldn’t I wrap it up with sequential module?</p>

ptrblck: <p>You don’t necessarily need to wrap it in an <code>nn.Sequential</code> module, as it is already a working module.<br>
However, since the <code>FeaturePyramidNetwork</code> is used internally, you will get the <code>OrderedDict</code> as the output as seen in <a href="https://github.com/pytorch/vision/blob/bb5af1d77658133af8be8c9b1a13139722315c3a/torchvision/ops/feature_pyramid_network.py#L138">this line of code</a>.<br>
Which means, your previous code would work for this submodule.</p>

Aviv_Shamsian: <p>Now it make sense!!! each value in the <code>Dict</code> represents the different resolution feature maps!</p>

Aviv_Shamsian: <p>BTW <a class="mention" href="/u/ptrblck">@ptrblck</a> another thing, lets say I want to save this <code>OrderedDict</code> to  <code>h5py</code> file. How can I do so? Is there any built in function for that?</p>

ptrblck: <p>I’m not sure, what the easiest way would be to store the output in an HDF5 file.<br>
However, you could just use <code>torch.save(out_dict, 'name.pth')</code> to save it.<br>
Would that work or do you need to use h5py?</p>

Aviv_Shamsian: <p>I want it to be compressed and fast if I will use <code>dataloader</code> to read these files.</p>

Dipam_Vasani: <p>I want to extract the model till the FPN and not the layers after that (pooling and other things), I am doing the following:</p>
<pre><code class="lang-python">all_children = list(model.backbone.children())
second_child = list(all_children[1].children())[0]
new_children = [all_children[0]] + [second_child]
nn.Sequential(*new_children)
</code></pre>
<p>My model is defined as follows:</p>
<pre><code class="lang-auto">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = pretrained)
num_classes = len(set(cls2idx.values())) + 1
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
</code></pre>
<p>When I pass images to my model, I get an error saying forward expects one positional argument but 2 were given. I am only doing model(images). What am I doing wrong?</p>
<p>How do I get the feature maps so that I can use them for another application?</p>