Visualizing intermediate layers with pretrained model with nn.ModuleList

HansBambel: <p>I am trying to visualize some intermediate (attention) layers from a network that I created myself.<br>
I already tried the approaches from <a href="https://discuss.pytorch.org/t/accessing-intermediate-layers-of-a-pretrained-network-forward/12113" class="inline-onebox">Accessing intermediate layers of a pretrained network forward?</a> and <a href="https://discuss.pytorch.org/t/extract-features-from-models-made-with-nn-modulelist/10427" class="inline-onebox">Extract Features from models made with nn.ModuleList</a> but to no avail.</p>
<p>Somehow I am not possible to iterate over my <code>nn.ModuleList</code>:</p>
<pre><code class="lang-auto">class Attention_Maps(nn.Module):
    def __init__(self):
        super(Attention_Maps, self).__init__()
        # Load my model
        model = myModelClass
        model = model.load_from_checkpoint("&lt;path_to_my_model&gt;.ckpt")
        # With vgg_modules it works, despite being the same type
        vgg_modules = list(vgg16(pretrained=True).features)
        image_modules = list(model.children())
        self.modules = nn.ModuleList(image_modules)

    def forward(self, x):
        results = []
        # TypeError here in enumerate
        for i, model in enumerate(self.modules):
            x = model(x)
            if i in {1, 3, 5, 7, 9}:
                results.append(x)
        return results
</code></pre>
<p>This results in the following error:<br>
<code>TypeError: 'method' object is not iterable</code></p>
<p>The weird thing is that <code>vgg_modules</code> and <code>image_modules</code> are the same type, so I don’t expect them to behave differently.<br>
Am I doing something wrong?</p>

chetan_patil: <p>Could you replace <code>nn.ModuleList()</code> with <code>nn.Sequential()</code> and then try ?</p>

HansBambel: <p>I tried that already as well. I need to do <code>self.modules = nn.Sequential(*image_modules)</code>, but I get the same TypeError. This method also does not work with <code>vgg_modules</code> as <code>nn.ModuleList</code>.</p>

chetan_patil: <p>Hi, I tried using <code>nn.Sequential()</code> and in my case for <code>vgg_modules</code> works,</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import torch.nn as nn

vgg_modules = list(models.vgg16(pretrained=True).features)
modules = nn.Sequential(*vgg_modules)
result = []
x = torch.randn(1,3,42,42)

for i, mod in enumerate(modules):
    x = mod(x)
    # print("Passed layer {}".format(i))
    if i in {1, 3, 5, 7, 9}:
        result.append(x)
        if i&gt;10:
            break

print("done.!!", len(result))
</code></pre>

HansBambel: <p>That is interesting! The code works when it is not in a class for me. So, unlike the example from the linked discussions that put the loaded model in a separate class and used the <code>forward</code>-method, your solution did the trick for me!</p>
<p>This works for me now:</p>
<pre><code class="lang-auto">model = model.load_from_checkpoint(f"&lt;path-to-my-model&gt;.ckpt")
image_modules = list(model.children())
modules = nn.Sequential(*image_modules)

results = []
for i, model in enumerate(modules):
    x = model(x)
    if i in {1, 3, 5, 7, 9}:
        results.append(x)
        if i &gt;= 9:
            break
print("Done!", len(results))
</code></pre>
<p>It now also works with</p>
<pre><code class="lang-auto">image_modules = list(model.children())
modules = nn.ModuleList(image_modules)
</code></pre>
<p>Thanks!</p>