<< extracting stats from activated layers - training >>

marcin: <p>Hey!<br>
I would like to make sure I am doing this right and make some improvements.<br>
It’s kinda lengthy, but I want to be as specific as possible.</p>
<p>The train and test code is in this <a href="https://github.com/kuangliu/pytorch-cifar" rel="nofollow noopener">repo</a>, I am making some minor changes, but nothing major. Just adding stuff to these examples.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.0.0'
</code></pre>
<p>I am running experiments on CIFAR-10 using various architectures, for this example let’s use VGG11.<br>
<strong>I would like to record stats of all activated layers while training, every epoch or every n-th epoch</strong>.</p>
<p>Result of a below print statement.</p>
<pre><code class="lang-auto">print(list(net.named_modules()))
</code></pre>
<pre><code class="lang-auto">[('', VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace)
    (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (4): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (5): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (6): ReLU(inplace)
    (7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (8): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (10): ReLU(inplace)
    (11): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (12): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (13): ReLU(inplace)
    (14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (15): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (16): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (17): ReLU(inplace)
    (18): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (19): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (20): ReLU(inplace)
    (21): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (22): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (23): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (24): ReLU(inplace)
    (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (26): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (27): ReLU(inplace)
    (28): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (29): AvgPool2d(kernel_size=1, stride=1, padding=0)
  )
  (classifier): Linear(in_features=512, out_features=10, bias=True)
)), ('features', Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU(inplace)
  (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (4): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (5): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (6): ReLU(inplace)
  (7): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (8): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (9): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (10): ReLU(inplace)
  (11): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (12): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (13): ReLU(inplace)
  (14): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (15): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (16): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (17): ReLU(inplace)
  (18): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (19): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (20): ReLU(inplace)
  (21): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (22): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (23): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (24): ReLU(inplace)
  (25): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (26): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (27): ReLU(inplace)
  (28): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (29): AvgPool2d(kernel_size=1, stride=1, padding=0)
)), 
('features.0', Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), 
('features.1', BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.2', ReLU(inplace)), 
('features.3', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1,ceil_mode=False)), 
('features.4', Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), 
('features.5', BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.6', ReLU(inplace)), 
('features.7', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)), 
('features.8', Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), 
('features.9', BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.10', ReLU(inplace)), 
('features.11', Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), ('features.12', BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.13', ReLU(inplace)), 
('features.14', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)), 
('features.15', Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), ('features.16', BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.17', ReLU(inplace)), 
('features.18', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), ('features.19', BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.20', ReLU(inplace)), 
('features.21', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)), 
('features.22', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), 
('features.23', BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.24', ReLU(inplace)), 
('features.25', Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))), 
('features.26', BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)), 
('features.27', ReLU(inplace)), 
('features.28', MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)), 
('features.29', AvgPool2d(kernel_size=1, stride=1, padding=0)), 
('classifier', Linear(in_features=512, out_features=10, bias=True))]
</code></pre>
<p>Using hooks that you’re mentioned <a href="https://discuss.pytorch.org/t/extract-features-from-layer-of-submodule-of-a-model/20181/2">here</a> I am able extract stats from the layers like this:</p>
<pre><code class="lang-auto">activations = defaultdict(list)
def save_activation(name):
    def hook(model, input, output):
        activations[name].append((epoch,
        np.float(output.min().detach().data), 
        np.float(output.max().detach().data), 
        np.float(output.mean().detach().data), 
        np.float(output.std().detach().data)))
    return hook
net.module.register_forward_hook(save_activation('features.2'))
net.module.register_forward_hook(save_activation('features.6'))
net.module.register_forward_hook(save_activation('features.10'))
net.module.register_forward_hook(save_activation('features.13'))
</code></pre>
<p>Here is litany of my questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">:</p>
<ol>
<li>
<p>Given the output of net.named_modules, am I getting the right indices for activated layers? <code>(2): ReLU(inplace)</code><br>
this is how I am accessing hooked information<br>
<code>net.module.register_forward_hook(save_activation('features.2'))</code><br>
If this is not right, please advise on how can I access activated layers explicitly and efficiently.</p>
</li>
<li>
<p>The dictionary collects stats of all batches - 391 to be exact, this is kind of spammy. How could I access the net stats of activated layers at the epoch end?</p>
</li>
<li>
<p>Is default dict the right data structure for collecting all that information?</p>
</li>
</ol>
<p>I’ve seen fast.ai did some work on that, but I don’t really understand how to use it, so I would prefer to stick to this version of training network.<br>
The best analogy that comes to mind is tf.summary in TensorFlow. That collects info to visualise in TensorBoard. To be clear, I want to do more than visualisation, hence stats needed.</p>
<p>Looking forward to hearing your thoughts on this.</p>

ptrblck: <ol>
<li>You would have to call <code>register_forward_hook</code> on the particular layer, not the complete model/module. Based on your code it looks like you are working with a <code>DataParallel</code> model, since you are accessing the <code>.module</code> attribute. If that’s the case, you would need to address each layer separately:</li>
</ol>
<pre><code class="lang-python">net.module.features[2].register_forward_hook(save_activation('features.2'))
net.module.features[6].register_forward_hook(save_activation('features.6'))
net.module.features[10].register_forward_hook(save_activation('features.10'))
net.module.features[13].register_forward_hook(save_activation('features.13'))
</code></pre>
<p>The <code>name</code> passed to <code>save_activation</code> is just for your own convenience. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
<ol start="2">
<li>You could just register the hook for the last epoch, pass some flag to <code>save_activations</code> if you are currently dealing with the last epoch, or just overwrite the activations using:</li>
</ol>
<pre><code class="lang-python">def save_activation(name):
    def hook(model, input, output):
        activations[name] = [(epoch,
        np.float(output.min().detach().data), 
        np.float(output.max().detach().data), 
        np.float(output.mean().detach().data), 
        np.float(output.std().detach().data))]
    return hook
</code></pre>
<ol start="3">
<li>It depends on your use case etc. I personally like to use <code>dicts</code> as you can make sure which activation you are currently working with. You could use whatever data structure fits your needs, but I’m quite paranoid about using plain <code>lists</code> as I cannot be sure to look at the right layer (e.g. what if the list was somehow extended). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">
</li>
</ol>
<p>If you want to visualize some stats of your model using Tensorboard, have a look at this <a href="https://github.com/pytorch/ignite/blob/master/examples/contrib/mnist/mnist_with_tensorboard_logger.py" rel="nofollow noopener">Ignite example</a>, where some handlers are used to do this.</p>

marcin: <p>Awesome, thanks for you help with this.</p>
<p>Just one more stupid question:<br>
Any layer described like below, refers to the activation layer and index at which could be accessed right?</p>
<pre><code class="lang-auto">(2): ReLU(inplace)
</code></pre>
<p>In above example</p>
<pre><code class="lang-auto">net.module.register_forward_hook(save_activation('features.2'))
</code></pre>
<p>I understand that correctly, right? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I know stupid, but really want to be sure <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>Thanks once more <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>The <code>(2)</code> corresponds to the layer name.<br>
If you just pass the layers into an <code>nn.Sequential</code> module, the name will correspond to the index in this sequential block.<br>
However, you could also pass custom names:</p>
<pre><code class="lang-python">model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))
    
print(model)
&gt; Sequential(
  (conv1): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))
  (relu1): ReLU()
  (conv2): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1))
  (relu2): ReLU()
)
print(id(model.conv1) == id(model[0]))
&gt; True
</code></pre>
<p>In your example code, you just register the hook to the complete model, so that <code>input</code> will correspond to the input data and <code>output</code> to the result of your last layer.</p>

marcin: <p>Apologies my bad, pasted wrong thing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Ment to paste your way</p>
<pre><code class="lang-auto">net.module.features[2].register_forward_hook(save_activation('features.2'))
</code></pre>
<p>like that I am accessing tensor of this particular activated layer right? So I can compute stats.</p>

ptrblck: <p>You are indexing into the <code>nn.Sequential</code> module, which is defined as <code>features</code>.<br>
<code>print(net.module.features[2])</code> will give you that particular <em>layer</em>, not the activation tensor.<br>
This layer will register the hook, which will get the activation tensor in <code>def hook</code>.</p>

marcin: <p>and that <em>hooked</em> tensor will correspond to the specified layer (accessed by indexing or naming)?</p>
<p>So, by printing DataParallel model like above <code>list(net.named_modules())</code>, I will know indices of all layers including activations, which will be named with the following pattern (in this example, could be named differently elsewhere):</p>
<pre><code class="lang-auto">(2): ReLU(inplace)
(6): ReLU(inplace)
...
(27): ReLU(inplace)
</code></pre>
<p>Then by registering the hooks with right indices, using your method,  I can compute stats for activated layers.<br>
Apologies for layman lingo, but I am not as technical as you are ;).</p>
<p>And big thanks for being patient with my questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <aside class="quote no-group" data-username="marcin" data-post="7" data-topic="45880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> marcin:</div>
<blockquote>
<p>and that <em>hooked</em> tensor will correspond to the specified layer (accessed by indexing or naming)?</p>
</blockquote>
</aside>
<p>Yes, the tensor in your <code>hook</code> function will correspond to the specified layer (accessed by indexing or naming).</p>
<aside class="quote no-group" data-username="marcin" data-post="7" data-topic="45880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> marcin:</div>
<blockquote>
<p>So, by printing DataParallel model like above <code>list(net.named_modules())</code> , I will know indices of all layers including activations</p>
</blockquote>
</aside>
<p>Yes, if the activations are created as modules. The alternative way would be to use the functional API for the activation functions, e.g. as done in <a href="https://github.com/pytorch/vision/blob/249cfbf5fef41ee30f380d895ad451b0d16d2947/torchvision/models/densenet.py#L120">DenseNet</a>.<br>
If you encounter such a model, you might want to override the <code>forward</code> method and store the activations manually or replace them with the corresponding modules.</p>
<aside class="quote no-group" data-username="marcin" data-post="7" data-topic="45880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> marcin:</div>
<blockquote>
<p>Then by registering the hooks with right indices, using your method, I can compute stats for activated layers.</p>
</blockquote>
</aside>
<p>Yes, your code looks fine to achieve this.</p>
<aside class="quote no-group" data-username="marcin" data-post="7" data-topic="45880">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/958977/48.png" class="avatar"> marcin:</div>
<blockquote>
<p>Apologies for layman lingo, but I am not as technical as you are ;).</p>
<p>And big thanks for being patient with my questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Haha, no need to apologize. Your post are perfectly clear! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

marcin: <p>Thank you, good human! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
For all clarifications, now time to get to work <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>