How to fully unpack the module._modules.items()

username: MANIK_SHARMA_ed15b02
cooked: <p>def forward_pass_on_convolutions(self, x):<br>
“”"<br>
Does a forward pass on convolutions, hooks the function at given layer<br>
“”"</p>
<pre><code>    l1 = ['conv0','norm0','relu0','pool0','norm5']
    l2 = ['denseblock1','denseblock2', 'denseblock3', 'denseblock4']
    l3 = ['transition1', 'transition2', 'transition3']

    l_all = l1+l2+l3

    conv_output = None
    for module_pos, module in self.model.features._modules.items():
        print ('m', module_pos)
        if module_pos in l1:
            x = module(x)  # Forward
            if module_pos == self.target_layer:
                x.register_hook(self.save_gradient)
                conv_output = x  # Save the convolution output on that layer

        if module_pos in l2:
            for smodule_pos, smodule in module._modules.items():
                print ('sm', smodule_pos)
                x = smodule(x)  # Forward
                if smodule_pos == self.target_layer:
                    x.register_hook(self.save_gradient)
                    conv_output = x  # Save the convolution output on that layer

        if module_pos in l3:
            for smodule_pos, smodule in module._modules.items():
                print ('sm', smodule_pos)
                x = smodule(x)  # Forward
                if smodule_pos == self.target_layer:
                    x.register_hook(self.save_gradient)
                    conv_output = x  # Save the convolution output on that layer

    return conv_output, x
</code></pre>
<p>it gives the error:</p>
<p>Traceback (most recent call last):<br>
File “main.py”, line 651, in <br>
main(args)<br>
File “main.py”, line 635, in main<br>
visualize(args)<br>
File “main.py”, line 526, in visualize<br>
cam = grad_cam.generate_cam(prep_img, S, target_class)<br>
File “./pytorch-cnn-visualizations/src\gradcam_.py”, line 99, in generate_cam<br>
conv_output, model_output = self.extractor.forward_pass(input_image, cell_type)<br>
File “./pytorch-cnn-visualizations/src\gradcam_.py”, line 71, in forward_pass<br>
conv_output, x = self.forward_pass_on_convolutions(x)<br>
File “./pytorch-cnn-visualizations/src\gradcam_.py”, line 49, in forward_pass_on_convolutions<br>
x = smodule(x)  # Forward<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\models\densenet.py”, line 48, in forward<br>
bottleneck_output = cp.checkpoint(bn_function, *prev_features)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\checkpoint.py”, line 155, in checkpoint<br>
return CheckpointFunction.apply(function, preserve, *args)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\utils\checkpoint.py”, line 74, in forward<br>
outputs = run_function(*args)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torchvision\models\densenet.py”, line 23, in bn_function<br>
bottleneck_output = conv(relu(norm(concated_features)))<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 547, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\modules\batchnorm.py”, line 81, in forward<br>
exponential_average_factor, self.eps)<br>
File “C:\Users\shmak\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\functional.py”, line 1656, in batch_norm<br>
training, momentum, eps, torch.backends.cudnn.enabled<br>
RuntimeError: running_mean should contain 48 elements not 144</p>
<p>But if i run:</p>
<pre><code>def forward_pass_on_convolutions(self, x):
    """
        Does a forward pass on convolutions, hooks the function at given layer
    """
    conv_output = None
    for module_pos, module in self.model.features._modules.items():
        print ('m', module_pos)
        x = module(x)  # Forward
        if module_pos == self.target_layer:
            x.register_hook(self.save_gradient)
            conv_output = x  # Save the convolution output on that layer
    return conv_output, x
</code></pre>
<p>It works fine</p>
<p>Also i might add, my motive is to access the denselayers modules and if possible conv1 and conv2 filters of the denselayer.</p>

username: ptrblck
cooked: <p>The error is raised, if you pass an input with the wrong number of channels to a batchnorm layer as shown here:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(144)
x = torch.randn(2, 48, 24, 24)
out = bn(x)
&gt; RuntimeError: running_mean should contain 48 elements not 144
</code></pre>
<p>Make sure that the incoming activations has the expected number of channels as specified in the batchnorm layer initialization.</p>

username: MANIK_SHARMA_ed15b02
cooked: <p>Hi, thank you for your reply.</p>
<p>But my problem is that:<br>
when i iterate over modules like:</p>
<pre><code class="lang-auto">for module_pos, module in self.model.features._modules.items():
    x = module(x)
</code></pre>
<p>It does not throw any error</p>
<p>But when iterate like this:</p>
<pre><code class="lang-auto">for module_pos, module in self.model.features._modules.items():
    for smodule_pos, smodule in module._modules.items():
        x  = smodule(x)
</code></pre>
<p>It throws error, therefore there is some sub_module which is gettting skipped.</p>
<p>How can I fully get the lowest nested Modules (i think in a DFGS manner?)?</p>
<p>Thanks again for your reply.</p>

username: ptrblck
cooked: <p>I’m not sure how your model is implemented, but by iterating all modules and submodules, you will skip all functional API calls from the original <code>forward</code> method.<br>
E.g. if a flatten operation was used via <code>torch.flatten</code> or <code>out = out.view(out.size(0), -1)</code>, this operation won’t show up in the modules and you might end up with shape mismatch errors.</p>
<p>Could you check the <code>forward</code> implementation and search for these functional calls?</p>

username: MANIK_SHARMA_ed15b02
cooked: <p>This is the standard pytorch implementation of memory-efficient Densenet-161 I am using.<br>
And as I checked here, <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/densenet.py</a></p>
<p>There are indeed such functional calls.<br>
Is there are any workaround for this?</p>
<p>Also i think it is important to add - This is an implementation of GradCAM. Therefore it is necessary to access the submodule if its gradients are to be visualized.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MANIK_SHARMA_ed15b02" data-post="5" data-topic="80144">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manik_sharma_ed15b02/48/11813_2.png" class="avatar"> MANIK_SHARMA_ed15b02:</div>
<blockquote>
<p>Is there are any workaround for this?</p>
</blockquote>
</aside>
<p>You could add these functional calls as new <code>nn.Modules</code>, but I wouldn’t recommend it.</p>
<aside class="quote no-group" data-username="MANIK_SHARMA_ed15b02" data-post="5" data-topic="80144">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/manik_sharma_ed15b02/48/11813_2.png" class="avatar"> MANIK_SHARMA_ed15b02:</div>
<blockquote>
<p>Therefore it is necessary to access the submodule if its gradients are to be visualized.</p>
</blockquote>
</aside>
<p>I’m not familiar with the necessary steps for GradCAM.<br>
If you need to get gradients of parameters, you could either loop all parameters and access their <code>param.grad</code> attribute or register hooks for it.<br>
Changing the forward pass shouldn’t change the gradient access, should it?</p>

username: MANIK_SHARMA_ed15b02
cooked: <p>There is nothing from in iterating over modules, as long as you get the last nested ones properly.<br>
The problem here is the Architecture of DenseNet where the next denselayer requires the input from all the previous layers.<br>
Someone implementing GradCAM has to take care for that fact.</p>