How to use Hooks to obtain layer outputs

username: oasis
cooked: <p>Beginner question: I was trying to use PyTorch Hook to get the layer output of pretrained model. I’ve tried two approaches both with some issues:</p>
<p>method 1:</p>
<pre><code class="lang-auto">net = EfficientNet.from_pretrained('efficientnet-b7')

visualisation = {}

def hook_fn(m, i, o):
  visualisation[m] = o 

def get_all_layers(net):
  for name, layer in net._modules.items():
    #If it is a sequential, don't register a hook on it
    # but recursively register hook on all it's module children
    if isinstance(layer, nn.Sequential):
      pass
    else:
      # it's a non sequential. Register a hook
      layer.register_forward_hook(hook_fn)

get_all_layers(net)

  
out = net(torch.randn(2,3,224,224))

# Just to check whether we got all layers
visualisation.keys()
</code></pre>
<p>This is from the <a href="https://blog.paperspace.com/pytorch-hooks-gradient-clipping-debugging/" rel="nofollow noopener">tutorial</a>. However, there’re two issues:</p>
<ol>
<li>the result only has 8 keys/items - was expecting it to have a lot more layers.</li>
<li>EfficientNet seems to require batch input, but say if I want to store the last but second layer’s output for 10 different images respectively in a list or array, how can I do so?</li>
</ol>
<p>method 2:<br>
I was trying to understand how to use the Hook class to extract a last but second layer output from a bunch of images without saving every layer, and if I’m passing input to a pre-trained model, do I still need a backward step?</p>
<pre><code class="lang-auto">class Hook():
    def __init__(self, module, backward=False):
        if backward==False:
            self.hook = module.register_forward_hook(self.hook_fn)
        else:
            self.hook = module.register_backward_hook(self.hook_fn)
    def hook_fn(self, module, input, output):
        self.input = input
        self.output = output
    def close(self):
        self.hook.remove()
</code></pre>
<p>Thanks in advance.</p>

username: albanD
cooked: <p>Hi,</p>
<blockquote>
<p>the result only has 8 keys/items - was expecting it to have a lot more layers.</p>
</blockquote>
<p>First of all, I wouldn’t use the <code>._modules</code> private attribute but the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.children"><code>.children()</code></a> public API.<br>
As you can see in the doc, it only gives the direct submodules for the current one. And so if you want to get all of them, you will need to call <code>.children()</code> on each submodule you see as well!</p>
<blockquote>
<p>EfficientNet seems to require batch input, but say if I want to store the last but second layer’s output for 10 different images respectively in a list or array, how can I do so?</p>
</blockquote>
<p>I’m not what you mean by that sentence.<br>
You can access any value you want there, so just get the samples you want from the batch?</p>

username: oasis
cooked: <p>Thank you so much for your answer! I wasn’t familiar with the difference between modules vs children - will try children instead</p>
<p>for the next part, I mean if only passing in one image, the Net won’t work and will return ’ [Error: Expected more than 1 value per channel when training]’</p>

username: albanD
cooked: <aside class="quote no-group" data-username="oasis" data-post="3" data-topic="93440">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/a698b9/48.png" class="avatar"> oasis:</div>
<blockquote>
<p>for the next part, I mean if only passing in one image, the Net won’t work and will return ’ [Error: Expected more than 1 value per channel when training]’</p>
</blockquote>
</aside>
<p>I guess this is because you have a batchnorm or similar normalization layer in there that can only work in training mode when you pass in more than 1 sample?</p>

username: oasis
cooked: <p>if I run for name, layer in efficientNet.children() I will get ‘TypeError: cannot unpack non-iterable Conv2dStaticSamePadding object’, and if I run <code>children().items()</code> instead I get ‘AttributeError: ‘generator’ object has no attribute ‘items’’. Still puzzled by how to get an output layer from efficientNet or how to refer to it by its name.</p>

username: albanD
cooked: <p>You can check the doc. This function returns just the module. So you cannot do <code>name, mod in XXX.children()</code>. As you can see in the doc, if you want names, you can use <code>.named_children()</code>.</p>