Convolution that only take channel-wise summation?

Paralysis: <p>I basically want to do element-wise product between a filter and the feature map, but only take summation channel-wise. That is, I have a k*k*c filter, for each sliding widow, by summing only channel-wise, I get a k*k resulting map. The total result is a k*k*n feature map, where n is the number of sliding windows.</p>
<p>Can I do this in the current PyTorch package? Or I need to write my own “Convolution” layer?</p>
<p>Thanks!</p>

ptrblck: <p>Have a look at the group argument of <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d">nn.Conv2d</a>.<br>
If you want to use a set of filters for each input channel, you can set <code>groups=in_channels</code></p>

Paralysis: <p>Not exactly… I want the resulting feature map to be the same spatial size as the filter, not the input. And I only have ONE (<em>k*k*c</em>) filter.</p>
<p>Say we have the filter of (<em>k*k*c</em>), then for each sliding window. We first take element-wise product between the filter and a (<em>k*k*c</em>) region in the input feature map.</p>
<p>Then, we only sum over the channel, which result in a (<em>k*k</em>) matrix (while in the real convolution, we do both spatial and channel summation, resulting in just a single value).</p>
<p>Next, we move the filter to the next (<em>k*k*c</em>) region in the input feature map, getting another (<em>k*k</em>) matrix.</p>
<p>This procedure goes on till the sliding window covers all the input feature map. We concatenate all the resulting (<em>k*k</em>) matrices to be a (<em>k*k*N</em>) tensor, where <em>N</em> is the number of sliding windows.</p>
<p>Is this doable in current PyTorch framework?</p>
<p>Thank you!</p>

ptrblck: <p>Thanks for the detailed explanation of the procedure.<br>
It’s possible and I created a small example (probably you could still optimize something):</p>
<pre><code class="lang-python">batch_size = 1
channels = 5
h, w = 12, 12
image = torch.randn(batch_size, channels, h, w) # input image

kh, kw = 3, 3 # kernel size
dh, dw = 3, 3 # stride

filt = torch.randn(channels, kh, kw) # filter (create this as nn.Parameter if you want to train it)

patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
print(patches.shape)
&gt; torch.Size([1, 5, 4, 4, 3, 3]) # batch_size, channels, h_windows, w_windows, kh, kw

patches = patches.contiguous().view(batch_size, channels, -1, kh, kw)
print(patches.shape) 
&gt; torch.Size([1, 5, 16, 3, 3]) # batch_size, channels, windows, kh, kw

# Now we have to shift the windows into the batch dimension.
# Maybe there is another way without .permute, but this should work
patches = patches.permute(0, 2, 1, 3, 4)
patches = patches.view(-1, channels, kh, kw)
print(patches.shape)
&gt; torch.Size([16, 5, 3, 3]) # windows * batch_size, channels, kh, kw

# Now we can use our filter and sum over the channels
patches = patches * filt
patches = patches.sum(1)
print(patches.shape)
&gt; torch.Size([16, 3, 3]) # batch_size * windows, kh ,kw
</code></pre>
<p>I think this should do it.<br>
The shapes are a bit complicated so I tried to comment all steps.<br>
Let me know, if that’s your use case.</p>

Hmrishav_Bandyopadhy: <p>Sorry for pulling at this old thread, but can you explain how this is done…</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="21240">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code> filter (create this as nn.Parameter if you want to train it)</code></p>
</blockquote>
</aside>
<p>Also, does implementing this whole in a forward function of an nn.Module class ensures proper flow of gradients and proper update of weights?</p>

ptrblck: <p>You could wrap the <code>filt</code> in an <code>nn.Parameter</code>, which would create gradients in it after a <code>backward()</code> call:</p>
<pre data-code-wrap="python"><code class="lang-python">filt = nn.Parameter(torch.randn(channels, kh, kw))

patches = image.unfold(...)
...

patches.sum().backward()
print(filt.grad)
</code></pre>
<aside class="quote no-group" data-username="Hmrishav_Bandyopadhy" data-post="6" data-topic="21240">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hmrishav_bandyopadhy/48/24065_2.png" class="avatar"> Hmrishav_Bandyopadhy:</div>
<blockquote>
<p>Also, does implementing this whole in a forward function of an nn.Module class ensures proper flow of gradients and proper update of weights?</p>
</blockquote>
</aside>
<p>This should be the case. The small example shows, that the computation graph is properly created.</p>

Hmrishav_Bandyopadhy: <p>Just for the sake of reassurance, this would generate random filters, which are different for different channels, which is not what happens in <code>nn.conv2d</code> , but is done here for demonstration?</p>

Usama_Hasan: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, just for the sake of knowledge, performing a 1*1 convolutional (Network in network ) over a feature map would do channel-wise summation? Right. If not can you elaborate please.</p>

ptrblck: <p><code>nn.Conv2d</code> initializes the <code>weight</code> randomly in the shape <code>[out_channels, in_channels, height, width]</code>.<br>
In my example <code>filt</code> is a single filter, so <code>out_channels</code> would correspond to 1.</p>

ptrblck: <p>Non-grouped convolution all perform a channel-size summation as the last step.<br>
The <code>1x1</code> kernel would multiply each channel “pixel” with the corresponding weight and apply the summation afterwards.</p>

Hmrishav_Bandyopadhy: <p>Initializing convolutions in that effect with <code>out_channels=1</code> would mean that random values would be set up per <code>in_channel</code> row. In other words, we would see that different randomized filters are being convolved with different input channels.</p>
<p>Like Filter A is used in channel A and Filter B is used in channel B and all of them are summed up. On the other hand, having <code>out_channels</code> as 1 means that we are using a single filter for all channels and are adding them up channel-wise.<br>
If i use:</p>
<pre><code class="lang-auto">out_channels=1
in_channels=2
height=3
width=3
print(torch.rand([out_channels,in_channels,height,width]))
</code></pre>
<p>This gives me</p>
<pre><code class="lang-auto">tensor([[[[0.8728, 0.5557, 0.4790],
          [0.8028, 0.7029, 0.3034],
          [0.6854, 0.5710, 0.0458]],

         [[0.7219, 0.7814, 0.6355],
          [0.2254, 0.4197, 0.9144],
          [0.7522, 0.8480, 0.2152]]]])
</code></pre>
<p>As you see two filters are generated and they will be used in the two channels. But as <code>out_channels=1</code> we should have a single filter which is convolved with the two channels and the outputs of those convolutions are added to create a single filter.</p>
<p>This however seems to give the required filter:</p>
<pre><code class="lang-auto">print(torch.rand([out_channels,height,width]).unsqueeze(1).repeat(1, in_channels, 1,1))
</code></pre>
<p>The output:</p>
<pre><code class="lang-auto">tensor([[[[0.4394, 0.8586, 0.6954],
          [0.2975, 0.1665, 0.0886],
          [0.5199, 0.8727, 0.7965]],

         [[0.4394, 0.8586, 0.6954],
          [0.2975, 0.1665, 0.0886],
          [0.5199, 0.8727, 0.7965]]]])
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Hmrishav_Bandyopadhy" data-post="12" data-topic="21240">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hmrishav_bandyopadhy/48/24065_2.png" class="avatar"> Hmrishav_Bandyopadhy:</div>
<blockquote>
<p>As you see two filters are generated and they will be used in the two channels. But as <code>out_channels=1</code> we should have a single filter which is convolved with the two channels and the outputs of those convolutions are added to create a single filter.</p>
</blockquote>
</aside>
<p>We might be using a different terminology, since I see</p>
<pre data-code-wrap="python"><code class="lang-python">out_channels=1
in_channels=2
height=3
width=3
print(torch.rand([out_channels,in_channels,height,width]))
</code></pre>
<p>as a single filter with multiple input channels.<br>
What you’ve described is how each channel of this single filter is interacting with the channels of the input activation. These are generally not considered to be different filters.</p>
<p><a href="https://cs231n.github.io/convolutional-networks/#conv">CS231n - Convolution layer</a> gives a good description of the kernel shapes and how they are used.</p>

Hmrishav_Bandyopadhy: <p>Okay I’m terribly sorry. It seems i was gravely mistaken. Anyone walking down my path, please look <a href="https://ai.stackexchange.com/questions/5769/in-a-cnn-does-each-new-filter-have-different-weights-for-each-input-channel-or" rel="nofollow noopener">here</a> for an in-depth analysis. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> I would probably have carried this mistake long enough to make more serious ones!</p>

ptrblck: <p>No need to be sorry and I’m glad we figured it out. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If you’re coming from the “classical” computer vision domain, filters with multiple channels sound a bit weird at the beginning.</p>

Hmrishav_Bandyopadhy: <p>Yeah exactly! I started out with Image processing and there they had filters with single channels which is why i got thrown off and subconsciously started believing that the same single channel filter is multiplied with all the input channels.</p>

YOGESH_SANKU: <p>Hi, sorry to disturb you.<br>
Overview: I want to apply convolution operation only on a specific portion of the image ( some ROI)<br>
Summary: let’s say the original image is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f434f312f76e8d0e8e29491bf80f6322b6793c04.jpeg" alt="lrsdO" data-base62-sha1="yQlWGGknzVP9Z5VJd2qTHDhp5lO" width="320" height="256"><br>
and the image removing unwanted area is<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aa4a91cdc05f8a9b3eed5a926c838460e0e13dff.jpeg" alt="lrsdM11" data-base62-sha1="oisVWbrKtis3La6isC5h3pWwo3B" width="320" height="256"><br>
I want to run the detection network on the modified image with convolution operation applied on only a specific part to actually speed up the network. Can you help me with this?<br>
Or source code of how pytorch implemented convolutions so that I can write my own convolution operation.<br>
Thank you</p>

ptrblck: <p>You could implement your own custom convolution using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html"><code>unfold</code></a> (the docs give an example code snippet for it). To do so you could mask the operation, which would create your desired operation, but wouldn’t yield any speedup. I’m also unsure how you would speedup access and processing of such a “random” pattern.</p>

YOGESH_SANKU: <p>we get all the windows using unfold right?. So, I will try to remove the windows whose center is not in the interested region so that in matrix multiplication I will have a matrix of lesser size. which leads to an increase in the speedup. If I am wrong please correct me.</p>
<p>I have written the below code for normal convolution from scratch using previous discussions in this thread. And I am thinking if I am able to decrease the number of windows created then I would be able to decrease the running time of matrix multiplication. Please correct me if I am wrong</p>
<pre><code class="lang-auto">import cv2
import matplotlib.pyplot as plt 
%matplotlib inline
import numpy as np
import torch
import torch.nn as nn
class Myconv2d(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        
        #required variables
        batch_size, channels , h, w = input.shape 
        kh, kw = weight.shape # kernel size
        dh, dw = 1, 1 # stride
        ph, pw = 0, 0 # padding
        
        #create all windows, final size of patches : (batch_size*windows per batch , kh*kw*channels)
        patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
        patches = patches.contiguous().view(batch_size, channels, -1, kh, kw)
        patches = patches.permute(0, 2, 1, 3, 4)
        patches = patches.view(-1, channels, kh, kw)
        windows = patches.shape[0]
        patches = torch.reshape(patches, (windows,-1))
        
        #flatten the weight
        filters = weight.contiguous().view((-1,))
        
        #matrix multiplication
        patches = filters @ patches.T
        
        #output size calculation
        oh = (int)((h-kh+2*ph)/dh+1)
        ow = (int)((w-kw+2*pw)/dw+1)
        
        #back to normal image view
        patches = patches.reshape(batch_size,channels, oh,ow)
        
        return patches
    
    @staticmethod
    def backward(ctx, grad_output):
        input, weight, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias

#model
class Myconv2D(torch.nn.Module):
    def __init__(self, input_features, output_features, bias = None):
        super(Myconv2D, self).__init__()
        self.input_features = input_features
        self.output_features = output_features
        
        sobel = torch.tensor([ [  1.0,  6,  15,  20,  15,  6,  1],
                                [  2,  12, 30,  40, 30, 12,  2],
                                [  3, 18, 45,  60, 45, 18,  3],
                                [  0,  0,  0,   0,  0,  0,  0],
                                [ -3, -18,-45,-60,-45,-18, -3],
                                [ -2, -12, -30,-40,-30,-12, -2],
                                [ -1, -6, -15, -20, -15, -6, -1] ])
        self.weight = nn.Parameter(sobel)
        if bias:
            self.bias = nn.Parameter(torch.empty(output_features))
        else:
            self.register_parameter('bias', None)
        
        if self.bias is not None:
            self.bias.data.uniform_(-0.1, 0.1)
            
    def forward(self, input):
        return Myconv2d.apply(input, self.weight, self.bias)
    
if __name__ == "__main__":
    path = "../input/motorroll-tracking/0001.jpg"
    img = cv2.imread(path)
    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    batch_size = 1
    channels = 1
    h, w = img.shape
    image = torch.from_numpy(img/255.0).float() # input image
    image = image.reshape(batch_size,channels,h,w)
    mymodel = Myconv2D(1,1)
    output = mymodel.forward(image)
    for i in range(batch_size):
        plt.imshow(output[i,0,:,:].detach().numpy(),cmap='gray')
</code></pre>

ptrblck: <p>Yes, if you are able to reduce the matrix size used in the matmul, you would save computations and could thus speed it up. Depending on the actual layout, the logic to remove parts of the matrix etc. you might be able to see a performance gain.</p>

m.sravya_Reddy: <p>Hi, I am also trying something similar. Did this work for you? Were you able to boost the performance? If yes, could you please share the code here? It would be really helpful for my work. Thanks in advance.</p>