How to access intermediate layer in a pretrained model and sequence layers?

username: Johnson_Mark
cooked: <p>I have a model code that defined as (model_baseline)</p>
<pre><code class="lang-auto">        modules = []
        for block in blocks:
            for bottleneck in block:
                modules.append(
                    unit_module(bottleneck.in_channel,
                                bottleneck.depth,
                                bottleneck.stride))
        self.body = Sequential(*modules)

        self._initialize_weights()

    def forward(self, x):
        x = self.input_layer(x)
        x = self.body(x)
        x = self.output_layer(x)

        return x
</code></pre>
<p>Link: <a href="https://github.com/ZhaoJ9014/face.evoLVe.PyTorch/blob/d5e31893f7e30c0f82262e701463fd83d9725381/backbone/model_irse.py#L156" rel="noopener nofollow ugc">https://github.com/ZhaoJ9014/face.evoLVe.PyTorch/blob/d5e31893f7e30c0f82262e701463fd83d9725381/backbone/model_irse.py#L156</a></p>
<p>where blocks is</p>
<pre><code class="lang-auto">blocks = [
            get_block(in_channel=64, depth=64, num_units=3),
            get_block(in_channel=64, depth=128, num_units=13),
            get_block(in_channel=128, depth=256, num_units=30),
            get_block(in_channel=256, depth=512, num_units=3)
        ]
</code></pre>
<p>I want to access the output of block 0 in the <code>body</code> during training (<code>get_block(in_channel=64, depth=64, num_units=13),</code>) . How can I do it in pytorch? Note that, the network is loaded from a pretrained model</p>
<p>This is what I tried</p>
<pre><code class="lang-auto">        modules_0 = []
        modules = []
        for block in blocks[0:1]:
            for bottleneck in block:
                modules_0.append(
                    unit_module(bottleneck.in_channel,
                                bottleneck.depth,
                                bottleneck.stride))
        self.body_0 = Sequential(*modules_0)

        modules = []
        for block in blocks[1:]:
            for bottleneck in block:
                modules.append(
                    unit_module(bottleneck.in_channel,
                                bottleneck.depth,
                                bottleneck.stride))
        self.body = Sequential(*modules)


        self._initialize_weights()

    def forward(self, x):
        x = self.input_layer(x)
        x = self.body_0(x)
        x = self.body(x)
        x = self.output_layer(x)
</code></pre>
<p>However, the above method cannot use weight from pretrained model that trained on the model_baseline architecture</p>

username: ptrblck
cooked: <p>You can use forward hooks to get the output of a specific module.<br>
<a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">This post</a> gives you an example. Let me know, if you get stuck.</p>

username: Johnson_Mark
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Great for your help. I have tried your approach but it does not work. This is my full code<br>
<a href="https://gist.github.com/John1231983/aeba806e92ed62052e842f4de74049b1#file-model-py-L177" rel="noopener nofollow ugc">https://gist.github.com/John1231983/aeba806e92ed62052e842f4de74049b1#file-model-py-L177</a></p>

username: ptrblck
cooked: <p>You are registering the hook after the forward pass was already performed.<br>
Could you use the same approach I’ve used in the linked post or explain what exactly is not working?</p>

username: Johnson_Mark
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="99524">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>explain</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> My aim is to get attention (CAM) at a selected layer during forwarding. Is it possible to use your method in my case?</p>

username: ptrblck
cooked: <p>Yes, you should be able to get the intermediate activations, if you stick to my code example.<br>
As explained before, you are registering the hook after the forward pass was already performed, so that you won’t be able to get this activation anymore.</p>