Same activation at different layers

username: jmlb
cooked: <p>I want to extract the feature maps at different layers of a VGG model.<br>
The VGG model is implemented as a nested sequence of blocks. Each elementary block is a sequence of layers. For example:</p>
<pre><code class="lang-auto">block3 = Sequence(
                  Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
                 ReLU(inplace)
                 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 ReLU(inplace)
                 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
                 ReLU(inplace)
</code></pre>
<p>To extract the activations in <code>block3</code> for instance, i used the approach detailed <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a> with <code>get_activation(name)</code> function.</p>
<pre><code class="lang-auto">for name, module in model._modules.items():
    if name == 'encoder':
        for nname, submodule in module._modules.items():
            if nname == 'block3':
                for layer_name, layer in submodule._modules.items():
                    print(nname, layer)
                    model.encoder.block3.register_forward_hook(get_activation(layer_name))

output = model(x_input)
print(activation.keys())   # dict_keys(['1', '3', '4', '0', '2', '5'])
</code></pre>
<p>So far, so good. I can visualize the activations. However, they look exactly the same: when I compute the difference between the output activation of <code>block3.0</code> and of <code>block3.4</code>, I get <code>0</code>:</p>
<pre><code class="lang-auto">block3_0 = activation['0'].numpy()[0]
block3_4 = activation[4'].numpy()[0]
print(np.sum(block3_0 - block3_4))  # -&gt; 0
</code></pre>
<p>Did I setup the forward hook properly so it “attached” to different layers?<br>
Thank you.</p>

username: ptrblck
cooked: <p>Could you print <code>layer_name</code> inside the loop and check, if the same name is passed to <code>get_activation</code>, thus overwriting the old activations?</p>

username: jmlb
cooked: <p>Thanks for the reply.<br>
Here is the result of <code>print(layer_name, layer)</code> inside the last loop:</p>
<pre><code class="lang-auto">0 Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
1 ReLU(inplace)
2 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
3 ReLU(inplace)
4 Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
5 ReLU(inplace)
</code></pre>
<p>The 5 layers are listed (also shown in the list of keys of <code>activations</code>).</p>

username: spanev
cooked: <p>You are hooking the output of the whole <code>block3</code> for each activation, I think this is what you want to do:</p>
<pre><code class="lang-auto">for layer_name, layer in module._modules.items():
     print(layer_name, layer)
     model.encoder.block3[int(layer_name)].register_forward_hook(get_activation(layer_name))
</code></pre>

username: ptrblck
cooked: <p>Good catch! That should be the reason for the same activations.</p>

username: jmlb
cooked: <p><a class="mention" href="/u/spanev">@spanev</a> , I replaced the nested loop by the loop that you provided. It prints only 3 layers of <code>block3</code>:</p>
<pre><code class="lang-auto">0 Conv2d(256, 128, kernel_size=(2, 2), stride=(1, 1))
1 ReLU(inplace)
2 Conv2d(128, 21, kernel_size=(1, 1), stride=(1, 1))
</code></pre>
<p>But the architecture of the block is:</p>
<pre><code class="lang-auto">self.block3 = nn.Sequential(nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
                                    nn.ReLU(inplace=True),
                                    nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
                                    nn.ReLU(inplace=True),
                                    nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
                                    nn.ReLU(inplace=True))
</code></pre>
<p>On the good side, if I compare block3_0 and block3_2, the sum of the difference is &gt; 0.</p>

username: spanev
cooked: <p>Hmm that’s weird, I just tried it with the block you provided and that’s what I get:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x = torch.rand(1, 128, 64, 64)
&gt;&gt;&gt; block3 = nn.Sequential(nn.Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
...                  nn.ReLU(),
...                  nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
...                  nn.ReLU(),
...                  nn.Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)),
...                  nn.ReLU())
&gt;&gt;&gt; activation = {}
&gt;&gt;&gt; for layer_name, layer in block3._modules.items():
...     print(layer_name, layer)
...     block3[int(layer_name)].register_forward_hook(get_activation(layer_name))
...
0 Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
&lt;torch.utils.hooks.RemovableHandle object at 0x101aa9ba8&gt;
1 ReLU()
&lt;torch.utils.hooks.RemovableHandle object at 0x10a1452e8&gt;
2 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
&lt;torch.utils.hooks.RemovableHandle object at 0x101aa9ba8&gt;
3 ReLU()
&lt;torch.utils.hooks.RemovableHandle object at 0x10a1452e8&gt;
4 Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
&lt;torch.utils.hooks.RemovableHandle object at 0x101aa9ba8&gt;
5 ReLU()
&lt;torch.utils.hooks.RemovableHandle object at 0x10a1452e8&gt;
&gt;&gt;&gt; block3(x).shape
torch.Size([1, 256, 64, 64])
&gt;&gt;&gt; activation.keys()
dict_keys(['0', '1', '2', '3', '4', '5'])

</code></pre>

username: spanev
cooked: <p>Can you check if this is the code you have now:</p>
<pre><code class="lang-auto">for name, module in model._modules.items():
    if name == 'encoder':
        for nname, submodule in module._modules.items():
            if nname == 'block3':
                for layer_name, layer in submodule._modules.items():
                    print(nname, layer) 
                    model.encoder.block3[int(layer_name)].register_forward_hook(get_activation(layer_name))
</code></pre>

username: jmlb
cooked: <p>Great! It works now. All layers of <code>block3</code> are printed out (I had to restart the notebook kernel).<br>
Thanks a lot!</p>
<pre><code class="lang-auto">this_block = model.encoder.block3
for layer_name, layer in this_block._modules.items():
    print(layer_name, layer)
    model.encoder.block3[int(layer_name)].register_forward_hook(get_activation(layer_name))
</code></pre>