Forward hooks on sequential nodes giving same output

builder173: <p>I’m working on visualizing the convolutions in my model. I use the following approach to add forward hooks to each module. These hooks record the input and output to two dicts.</p>
<pre><code class="lang-auto">node_out = {}
node_in = {}

#function to generate hook function for each module
def get_node_out(name):
  def hook(model, input, output):
    node_in[name] = input[0][0].detach()
    node_out[name] = output[0].detach()
  return hook

hook_handles = {}
for name, module in model.named_modules():
    hook_handles[name] = module.register_forward_hook(get_node_out(name))
</code></pre>
<p>The model above was already trained, in eval mode, and a prediction made on a single sample. The model starts with the following double convolution. I noticed that the output from the first BatchNorm1d layer was all non-negative and then confirmed that it was actually identical to the subsequent ReLU layer. I pulled all the necessary parameters (running mean, running std etc.) to do the BatchNorm1d manually and confirmed that it should have negative values given the node_in values to that layer. Is there something invalid about the way I’m generating/registering the hooks here that would cause this? Is there another approach recommended?</p>
<pre><code class="lang-auto">nn.Sequential(
        nn.Conv1d(in_c, out_c, padding=padding, kernel_size=kernel_size, bias=False),
        nn.BatchNorm1d(out_c),
        nn.ReLU(inplace=True),
        nn.Conv1d(out_c, out_c, padding=padding, kernel_size=kernel_size, bias=False),
        nn.BatchNorm1d(out_c),
        nn.ReLU(inplace=True))
</code></pre>
<p>I tried to follow the examples below in my implementation:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://web.stanford.edu/~nanbhas/blog/forward-hooks-pytorch/">
  <header class="source">
      <img src="https://web.stanford.edu/~nanbhas/assets/images/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://web.stanford.edu/~nanbhas/blog/forward-hooks-pytorch/" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 17 August 2020">Nandita Bhaskhar – 17 Aug 20</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://web.stanford.edu/~nanbhas/blog/forward-hooks-pytorch/" target="_blank" rel="noopener nofollow ugc">Intermediate Activations — the forward hook</a></h3>

  <p>Table of contents</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote" data-post="2" data-topic="43347">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-register-forward-hooks-for-each-module/43347/2">How to register forward hooks for each module</a> 
  </div>
  <blockquote>
    You could use model.named_modules() to iterate all submodules including their names and store the hooks in a dict: 
self.hooks = {}
for name, module in model.named_modules():
    hooks[name] = module.register_forward_hook(self,hook_fn)
  </blockquote>
</aside>

<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="builder173" data-post="1" data-topic="161334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/builder173/48/51744_2.png" class="avatar"> builder173:</div>
<blockquote>
<p>Is there something invalid about the way I’m generating/registering the hooks here that would cause this?</p>
</blockquote>
</aside>
<p>No, your hooks look correct.</p>
<aside class="quote no-group" data-username="builder173" data-post="1" data-topic="161334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/builder173/48/51744_2.png" class="avatar"> builder173:</div>
<blockquote>
<p>Is there another approach recommended?</p>
</blockquote>
</aside>
<p>Yes, you could disable <code>inplace</code> layers, since they are changing the activations inplace.<br>
In your case, the output of batchnorm will be changed by the <code>ReLU(inplace=True)</code> layer directly, so that the output of batchnorm and the input/output of the relu layer are all the same tensor after the relu was applied on it.</p>