How to take inputs through pre-trained CNNs, one layer at time?

mn01: <p>Hello - I am new to pytorch &amp; these forums, apologies if I haven’t directed this question to the right place.  Before I begin, I am using Python 3.7 and torch==1.5.0 incase either of these matter.</p>
<p>I am struggling to take inputs forward through an arbitrary pre-trained CNN, one layer/module at a time. I understand that forward hooks can be used to capture intermediate features. What I am hoping to find is a generalized way to replay intermediate features through subsequent layers, without requiring a full forward pass.  Is there a best practice for this in PyTorch?</p>
<p><strong>[edit]</strong> perhaps simplest form of my question is how to call</p>
<pre><code class="lang-auto">x = net.features(x)
y = net.classifier(x)
</code></pre>
<p>When I call all net.children() in sequence I either get an output shape that does not match what’s produced by full forward pass, or else execution errors passing intermediate features to the next layer.  Calling net.features() followed by net.classifier() also produces same/similar size mismatch errors described above.</p>
<p>I am not looking for fix specific to densenet161, but use it here as an example:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as tmodels
net = tmodels.densenet161(pretrained=True)
net.eval()

x = torch.randn(1, 3, 227, 227)
y = net(x).data.numpy()
print('y full:', y.shape)

for child in net.children():
  x = child(x)
y = x.data.numpy()
print('y part:', y.shape)
</code></pre>
<p>With this I get size mismatch errors in the net.children() loop:</p>
<pre><code class="lang-auto">$ python ./test.py 
y full: (1, 1000)
Traceback (most recent call last):
  File "./test.py", line 11, in &lt;module&gt;
    x = child(x)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
    output = input.matmul(weight.t())
RuntimeError: size mismatch, m1: [15456 x 7], m2: [2208 x 1000] at ../aten/src/TH/generic/THTensorMath.cpp:41
</code></pre>
<p>Insights/help much appreciated.  Thanks!</p>

ptrblck: <p>You could apply all child modules, if the model would generally work in an <code>nn.Sequential</code> container.<br>
The model would thus have to define only <code>nn.Modules</code> in its <code>__init__</code> method and apply these modules sequentially.<br>
However, this is often not the case and the <code>forward</code> method often uses functional calls such as <code>x.view()</code> etc., especially if the model architecture is more complicated then a simple stack of layers.</p>
<p>The <code>densenet</code> you are using is for example applying <a href="https://github.com/pytorch/vision/blob/1fe1e110677ab22f8512293987939d31916b7a8b/torchvision/models/densenet.py#L193-L195">these calls</a> from the functional API, which are lost if you just try to call the modules.</p>

mn01: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> - thanks!  I could tell that <code>net.classifier(net.features(x))</code> was not enough, but not quite understand why.</p>
<p>It seems a shame that for torchvision models at least, there isn’t simple API definition around having either <code>net.features()</code> -or- <code>net.classifier()</code> do the formatting. Or perhaps another method, say <code>net.format_features()</code>.</p>
<p>Thanks again for your helpful reply!</p>