Visualizing the outputs of kernels, instead of the outputs of filters

username: hamedB
cooked: <p>Hello</p>
<p>As I have seen, in many posts in the forum, peoples have explained about visualizing featuermaps(i.e., the outputs of the filters.)<br>
As the mentioned featuremaps are the summation of all outputs of kernels, my question is how we can access to these featuremaps outputted from kernels, right before summing up to produce final featuremaps?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Each “featuremap” is created by a single kernel. I.e. a conv layer using the setup <code>nn.Conv2d(in_channels=3, out_channels=10, kernel_size=3)</code>, the output activation (featuremap) will have the shape <code>[batch_size, out_channels=10, height, width]</code>. The <code>out_channels</code> dimension corresponds to the number of filters/kernels in the layer (in the default setup) and each filter creates one activation map.</p>
<aside class="quote no-group" data-username="hamedB" data-post="1" data-topic="137266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>right before summing up to produce final featuremaps?</p>
</blockquote>
</aside>
<p>I guess this refers to the applied operation of each filter on the input activation. If you want to split the channels of the filter and apply them separately, you could use the <code>groups</code> argument during the instantiation of the conv layer.</p>

username: hamedB
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you for your explanations.<br>
In my explanations, I assume kernels as the axes/dimension for each filter.<br>
Based on your example, each one of 10 featuremaps refers to a filter which contains three kernels(because of the input dimension.)<br>
In other words, during the Conv. operation, 30 featuremaps are produces, afterwards they are summarized to 10 featuremaps.<br>
I meant, instead of accessing 10 featuremaps using hook function, accessing those 30 featuremaps.<br>
Is <em><strong>groups argument</strong></em> that you mentioned a solution to what I want(<em><strong>30 featuremaps</strong></em>). If so, is it possible to apply such operation on pre-trained CNN such as VGG19? or shall I train it from scratch with the Conv. layers including groups argument?</p>
<p>Thank you very much.</p>

username: ptrblck
cooked: <p>Filters and kernels are usually referring to the same object, but if I understand you correctly you are calling a 3D tensor a filter and each 2D slice a kernel?</p>
<p>If so, it seems you would like to get the activations after each “filter” was applied to the corresponding input activation channel and after the spatial dimensions (height and width) were already reduced, BUT before the channel dimension is reduced?</p>
<p>It would be possible to achieve this using grouped convolutions, but you would need to perform a lot of reshaping and would need to know how the conv kernels are applied internally.<br>
For grouped convs, take a look at <a href="https://discuss.pytorch.org/t/convolution-operator-with-groups-for-taking-each-convolved-result/714/9">this post</a> first as it gives you some information about the order the kernels are applied.</p>
<p>Let’s assume you are working with an input activation in the shape <code>[batch_size=1, channels=4, height=24, width=24]</code> and a standard conv layer with a <code>weight</code> tensor in the shape <code>[out_channels=8, in_channels=4, height=3, width=3]</code>. To keep the spatial dimension equal, let’s also use <code>padding=1</code> and <code>stride=1</code>. Let’s also set the <code>bias</code> to <code>None</code> to make it a bit easier.<br>
Now, you could use a grouped convolution via <code>nn.Conv2d(4, 8*4, 3, 1, 1, groups=4)</code>, however you would need to be careful about the order how these filters are applied.</p>
<p>The main issue are the needed permutations needed in the filters as the grouped conv is applied sequentially on each input channel as seen in the linked post, while the standard conv applies a “full” kernel with all its channels to the input, so <em>all</em> input channels will be used on each filter.<br>
It’s a bit hard to describe in words and although my drawing skills are not perfect, here is an illustration:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/adf32e382273800dbda282c878eba5db1628ea89.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/adf32e382273800dbda282c878eba5db1628ea89" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/adf32e382273800dbda282c878eba5db1628ea89_2_467x500.jpeg" alt="image" data-base62-sha1="oOPC5mZs0BAxHe3i8908CBsbQyR" width="467" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/adf32e382273800dbda282c878eba5db1628ea89_2_467x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/adf32e382273800dbda282c878eba5db1628ea89_2_700x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/adf32e382273800dbda282c878eba5db1628ea89.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">767×821 89.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here the corresponding code:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

N, C, H, W = 1, 4, 24, 24

for _ in range(100):
    N, C, H, W = torch.randint(1, 100, (4,))
    
    x = torch.rand(N, C, H, W)
    
    out_channels = 8
    kw = 3
    conv = nn.Conv2d(C, out_channels, kw, 1, 1, bias=False)
    out = conv(x)
    #print(out.shape)
    # &gt; torch.Size([1, 8, 24, 24])
    
    conv_grouped = nn.Conv2d(C, C*out_channels, kw, 1, 1, groups=C, bias=False)
    with torch.no_grad():
        conv_grouped.weight.copy_(conv.weight.permute(1, 0, 2, 3).reshape(C*out_channels, 1, kw, kw))
    out_grouped = conv_grouped(x)
    #print(out_grouped.shape)
    # &gt; torch.Size([1, 32, 24, 24])
    
    out_grouped = out_grouped.view(N, C, out_channels, H, W).permute(0, 2, 1, 3, 4).reshape(N, C*out_channels, H, W)
    
    # manually reduce
    idx = torch.arange(out_channels)
    idx = torch.repeat_interleave(idx, C)
    idx = idx[None, :, None, None].expand(N, -1, H, W)
    out_grouped_reduced = torch.zeros_like(out)
    out_grouped_reduced.scatter_add_(dim=1,index=idx, src=out_grouped)
    
    # check error
    print(torch.allclose(out_grouped_reduced, out, atol=5e-6), (out_grouped_reduced - out).abs().max())
</code></pre>
<p>As you can see, the outputs are equal after the “manual” reduction.</p>
<aside class="quote no-group" data-username="hamedB" data-post="3" data-topic="137266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>If so, is it possible to apply such operation on pre-trained CNN such as VGG19?</p>
</blockquote>
</aside>
<p>You could manipulate the pretrained VGG19 by replacing all conv layers with their grouped equivalent. I think writing a custom conv layer implementation would be beneficial given the complexity of the code.</p>

username: hamedB
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>First of all, I am thankful for your explanations and providing the links and image which really helped me to understand how grouped option works.</p>
<p>Kindly, there is still something unclear for me.<br>
If I am not wrong, based on your scheme, it is possible that channels from different filter are grouped. Is that right? Because in the standard one each filter has 4 channel with size 3,3, while in the grouped one each filter has 8 channel with size 3,3.<br>
could you please take a look in this <a href="https://towardsdatascience.com/a-comprehensive-introduction-to-different-types-of-convolutions-in-deep-learning-669281e58215" rel="noopener nofollow ugc">link</a>, where the an example of convolutional operation is explained.<br>
Here is part of that link:<br>
<strong>"Here the input layer is a 5 x 5 x 3 matrix, with 3 channels. The filter is a 3 x 3 x 3 matrix. First, each of the kernels in the filter are applied to three channels in the input layer, separately. Three convolutions are performed, which result in 3 channels with size 3 x 3. Then these three channels are summed together (element-wise addition) to form one single channel (3 x 3 x 1). This channel is the result of convolution of the input layer (5 x 5 x 3 matrix) using a filter (3 x 3 x 3 matrix)."</strong></p>
<p>Then, based on your standard convolutional layer, each filter has shape [4,3,3] such that each [1,3,3] is separately applied on each [1,24,24] input activation. When convoultional operation is done, the output will be [4,24,24] for one filter, afterwards, they are summed across channel dimension to output activation with shape [1,24,24]. Am I right?<br>
and what I need is the activations for each filter with shape [4,24,24] instead of [1,24,24].</p>

username: ptrblck
cooked: <p>Your explanation sounds right and fits into the standard conv approach shown in the attached picture.</p>
<aside class="quote no-group" data-username="hamedB" data-post="5" data-topic="137266">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamedb/48/35498_2.png" class="avatar"> hamedB:</div>
<blockquote>
<p>and what I need is the activations for each filter with shape [4,24,24] instead of [1,24,24].</p>
</blockquote>
</aside>
<p>My code snippet provides exactly this with additionally the channel dim reduction to verify that the outputs are equal.</p>

username: hamedB
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>A few months ago, I created this topic.<br>
I re-implemented your code snippet for replacing the standard <code>conv. layer</code> in Alexnet with the new one discussed in this topic. However, when I use <code>torch.allclose</code> function (implemented in your code) to figure out whether the output of the standard <code>conv. layer</code> and <code>modified one</code> is equal or not, the result is always <code>False</code>, even the program is run without any error. I mean the input is passed through all layer without error but the output of <code>torch.allclose</code> is always False. The difference between your code and my code is that in your code you considered bias parameter to be false for simplification. My code also considers the bias values.<br>
I provide my code also. I really appreciate it if you could please help out. What is wrong with my code that the out puts are different?</p>
<p>Thank you.</p>
<p>The code snippet for modifying the conv layer.</p>
<blockquote>
<p>for name, module in alex_net._modules.items():<br>
if name == ‘features’:<br>
for n,m in module._modules.items():<br>
if type(m) == torch.nn.modules.conv.Conv2d:<br>
alex_net._modules[name][int(n)] = custom_conv(alex_net._modules[name][int(n)])</p>
<p>print(alex_net)</p>
</blockquote>
<p>output: I keep the <code>standard conv. layer</code> in the <code>custom_conv class</code> for comparing their outputs.</p>
<blockquote>
<p>AlexNet(<br>
(features): Sequential(<br>
(0): custom_conv(<br>
(original_conv): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))<br>
(conv_grouped): Conv2d(3, 192, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2), groups=3)<br>
)<br>
(1): ReLU(inplace=True)<br>
(2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(3): custom_conv(<br>
(original_conv): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))<br>
(conv_grouped): Conv2d(64, 12288, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), groups=64)<br>
)<br>
(4): ReLU(inplace=True)<br>
(5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
(6): custom_conv(<br>
(original_conv): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(conv_grouped): Conv2d(192, 73728, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192)<br>
)<br>
(7): ReLU(inplace=True)<br>
(8): custom_conv(<br>
(original_conv): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(conv_grouped): Conv2d(384, 98304, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384)<br>
)<br>
(9): ReLU(inplace=True)<br>
(10): custom_conv(<br>
(original_conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(conv_grouped): Conv2d(256, 65536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256)<br>
)<br>
(11): ReLU(inplace=True)<br>
(12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(6, 6))<br>
(classifier): Sequential(<br>
(0): Dropout(p=0.5, inplace=False)<br>
(1): Linear(in_features=9216, out_features=4096, bias=True)<br>
(2): ReLU(inplace=True)<br>
(3): Dropout(p=0.5, inplace=False)<br>
(4): Linear(in_features=4096, out_features=4096, bias=True)<br>
(5): ReLU(inplace=True)<br>
(6): Linear(in_features=4096, out_features=1000, bias=True)<br>
)<br>
)</p>
</blockquote>
<p>Custom conv. layer implementations.</p>
<blockquote>
<p>class custom_conv(nn.Module):<br>
def <strong>init</strong>(self, conv_module):<br>
super(custom_conv, self).<strong>init</strong>()<br>
self.original_conv = conv_module<br>
self.in_channels = conv_module.in_channels<br>
self.out_channels = conv_module.out_channels<br>
self.kernel_size = conv_module.kernel_size<br>
self.stride = conv_module.stride<br>
self.padding = conv_module.padding<br>
self.conv_grouped = nn.Conv2d(self.in_channels, self.in_channels*self.out_channels,self.kernel_size,<br>
self.stride, self.padding, groups=self.in_channels, bias=True)<br>
with torch.no_grad():<br>
self.conv_grouped.weight.copy_(conv_module.weight.permute(1, 0, 2, 3).<br>
reshape(self.in_channels * self.out_channels, 1, self.kernel_size[0], self.kernel_size[1]))</p>
<pre><code>def __call__(self, x):
    out = self.original_conv(x)
    #print('original output ',out.shape)
    N = out.shape[0]
    H = out.shape[2]
    W = out.shape[3]
    out_grouped = self.conv_grouped(x)
    #print(out_grouped.shape)
    out_grouped = out_grouped.view(N, self.in_channels , self.out_channels, H, W).\
        permute(0, 2, 1, 3, 4).reshape(N, self.in_channels*self.out_channels, H, W)
    idx = torch.arange(self.out_channels)
    idx = torch.repeat_interleave(idx, self.in_channels)
    idx = idx[None, :, None, None].expand(N, -1, H, W)
    out_grouped_reduced = torch.zeros_like(out)
    out_grouped_reduced.scatter_add_(dim=1, index=idx, src=out_grouped)
    print(torch.allclose(out_grouped_reduced, out, atol=5e-6), (out_grouped_reduced - out).abs().max())

    return out_grouped_reduced, out
</code></pre>
</blockquote>
<p>I pass a random input with shape <code>[1,3,224,224]</code> and the output of my code is as follow:</p>
<blockquote>
<p>layer index:  0<br>
False tensor(0.2228, grad_fn=)<br>
original shape :  torch.Size([1, 64, 55, 55])<br>
grouped shape:  torch.Size([1, 64, 55, 55])<br>
layer index:  1<br>
original shape :  torch.Size([1, 64, 55, 55])<br>
grouped shape:  torch.Size([1, 64, 55, 55])<br>
layer index:  2<br>
original shape :  torch.Size([1, 64, 27, 27])<br>
grouped shape:  torch.Size([1, 64, 27, 27])<br>
layer index:  3<br>
False tensor(2.6001, grad_fn=)<br>
original shape :  torch.Size([1, 192, 27, 27])<br>
grouped shape:  torch.Size([1, 192, 27, 27])<br>
layer index:  4<br>
original shape :  torch.Size([1, 192, 27, 27])<br>
grouped shape:  torch.Size([1, 192, 27, 27])<br>
layer index:  5<br>
original shape :  torch.Size([1, 192, 13, 13])<br>
grouped shape:  torch.Size([1, 192, 13, 13])<br>
layer index:  6<br>
False tensor(10.5771, grad_fn=)<br>
original shape :  torch.Size([1, 384, 13, 13])<br>
grouped shape:  torch.Size([1, 384, 13, 13])<br>
layer index:  7<br>
original shape :  torch.Size([1, 384, 13, 13])<br>
grouped shape:  torch.Size([1, 384, 13, 13])<br>
layer index:  8<br>
False tensor(10.4743, grad_fn=)<br>
original shape :  torch.Size([1, 256, 13, 13])<br>
grouped shape:  torch.Size([1, 256, 13, 13])<br>
layer index:  9<br>
original shape :  torch.Size([1, 256, 13, 13])<br>
grouped shape:  torch.Size([1, 256, 13, 13])<br>
layer index:  10<br>
False tensor(10.4024, grad_fn=)<br>
original shape :  torch.Size([1, 256, 13, 13])<br>
grouped shape:  torch.Size([1, 256, 13, 13])<br>
layer index:  11<br>
original shape :  torch.Size([1, 256, 13, 13])<br>
grouped shape:  torch.Size([1, 256, 13, 13])<br>
layer index:  12<br>
original shape :  torch.Size([1, 256, 6, 6])<br>
grouped shape:  torch.Size([1, 256, 6, 6])<br>
layer index:  12<br>
original shape :  torch.Size([1, 256, 6, 6])<br>
grouped shape:  torch.Size([1, 256, 6, 6])<br>
layer index:  12<br>
original shape :  torch.Size([1, 1000])<br>
grouped shape:  torch.Size([1, 1000])</p>
</blockquote>

username: ptrblck
cooked: <p>I don’t know how you are processing the <code>bias</code> but you would either have to scale it with <code>1/C</code> if you are using the <code>bias</code> directly in the grouped conv or you could add it after the reduction as seen here:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

N, C, H, W = 1, 4, 24, 24

for _ in range(100):
    N, C, H, W = torch.randint(1, 100, (4,))
    
    x = torch.rand(N, C, H, W)
    
    out_channels = 8
    kw = 3
    conv = nn.Conv2d(C, out_channels, kw, 1, 1, bias=True)
    out = conv(x)
    #print(out.shape)
    # &gt; torch.Size([1, 8, 24, 24])
    
    conv_grouped = nn.Conv2d(C, C*out_channels, kw, 1, 1, groups=C, bias=True)
    with torch.no_grad():
        conv_grouped.weight.copy_(conv.weight.permute(1, 0, 2, 3).reshape(C*out_channels, 1, kw, kw))
        conv_grouped.bias.copy_(conv.bias.repeat(C) / C) # scale bias, since it's added after the channel reduction in the plain conv
    out_grouped = conv_grouped(x)
    #print(out_grouped.shape)
    # &gt; torch.Size([1, 32, 24, 24])
    
    out_grouped = out_grouped.view(N, C, out_channels, H, W).permute(0, 2, 1, 3, 4).reshape(N, C*out_channels, H, W)
    
    # manually reduce
    idx = torch.arange(out_channels)
    idx = torch.repeat_interleave(idx, C)
    idx = idx[None, :, None, None].expand(N, -1, H, W)
    out_grouped_reduced = torch.zeros_like(out)
    out_grouped_reduced.scatter_add_(dim=1, index=idx, src=out_grouped)
    
    # alternatively add the bias after the reduction
    #out_grouped_reduced = out_grouped_reduced + conv.bias[None, :, None, None].expand_as(out_grouped_reduced)
    
    # check error
    print(torch.allclose(out_grouped_reduced, out, atol=5e-6), (out_grouped_reduced - out).abs().max())
</code></pre>

username: hamedB
cooked: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>  for your reply.<br>
If I use the bias in this way for modifying pre-trained CNNs such as Alex on Image-Net, then I would be expected to have the same results for both models(i.e., the standard model and modified model )?</p>
<p>Thank you</p>

username: hamedB
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I tried your solution and it worked. the output of<code> torch.allclose</code> function became True.</p>
<p>If you don’t mind, I have one more question. In your code, you use the block<br>
<code>with torch.no_grad():</code><br>
for defining new conv. layer. it means that the parameter of this new conv.layer is not trainable?<br>
When I remove <code>with torch.no_grad():</code> I got error. If I want to make it trainable how I should implemented it?</p>
<p>Thank you for your attention given to my issue.</p>

username: ptrblck
cooked: <p>I use the <code>no_grad</code> guard to copy the parameters from the plain conv to the grouped conv layer. The grouped conv layer will still be trainable, but the parameter copy should not be tracked by Autograd (as it’ll raise an error as you’ve already seen).</p>