Convolving a 2D Kernel on each channel?

username: soulslicer
cooked: <p>I have a single 2D Kernel of size [3,3], and a Tensor of size [B, 64, H, W]</p>
<p>How can I apply this 2D kernel on each channel?</p>
<p>How should i reshape/repeat the kernel?</p>
<p>I tried to repeat my Kernel, as follows:</p>
<p><code>kernel = kernel.repeat((B, 64, 1, 1))</code></p>
<p>But when I apply it the tensor size changes</p>
<p><code>[1, 64, H, 1]</code></p>
<p>I am forced to do this for now but it is inefficient:</p>
<pre><code class="lang-auto">    for b in range(0, dpv_permuted.shape[0]):
        for c in range(0, dpv_permuted.shape[1]):
            dpv_permuted[b,c,:,:] = F.conv2d(dpv_permuted[b,c].unsqueeze(0).unsqueeze(0), **spread_kernel).squeeze(0).squeeze(0)
</code></pre>

username: Nikronic
cooked: <p>Hi,</p>
<p>Here is the issue</p>
<aside class="quote no-group" data-username="soulslicer" data-post="1" data-topic="87328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soulslicer/48/9454_2.png" class="avatar"> soulslicer:</div>
<blockquote>
<p>kernel = kernel.repeat((B, 64, 1, 1))</p>
</blockquote>
</aside>
<p>If your input size is <code>[64, h, w]</code> (batch_size does not matter), and if you want to have outputs with size <code>[64, new_h, new_w]</code>, then you need 64 filters which each one is [64, 3, 3]. Literally a <code>[64, 64, 3, 3]</code> filter is needed.</p>
<pre><code class="lang-auto">
b = 2
c = 64
h, w = 15, 15

x = torch.randn(b, c, h, w)
kernel = (1/9)*torch.tensor([[1, 1, 1], 
                             [1, 1, 1], 
                             [1, 1, 1]])
kernel = kernel.repeat((64, 64, 1, 1))
print(kernel.shape)  # torch.Size([64, 64, 3, 3])
res = F.conv2d(input=x, weight=kernel, padding=1)
print(res.shape)  # torch.Size([2, 64, 15, 15])
</code></pre>
<p>Bests</p>

username: soulslicer
cooked: <p>I tried your code, but it gives me completely different results. Here are my two functions:</p>
<pre><code class="lang-auto">spread_kernel = None
def spread_dpv(dpv, N=5):
    global spread_kernel
    dpv_permuted = dpv.permute(0, 3, 2, 1)

    kernel = torch.Tensor(np.zeros((N, N)).astype(np.float32))
    kernel[int(N / 2), :] = 1 / float(N)
    kernel = kernel.repeat((dpv_permuted.shape[1], dpv_permuted.shape[1], 1, 1)).to(dpv_permuted.device)

    dpv_permuted = F.conv2d(input=dpv_permuted, weight=kernel, padding=N // 2)

    dpv = dpv_permuted.permute(0, 3, 2, 1)
    tofuse_dpv = dpv / torch.sum(dpv, dim=1).unsqueeze(1)
    return tofuse_dpv

spread_kernel = None
def spread_dpv_hack(dpv, N=5):
    # torch.Size([128, 384])
    # torch.Size([1, 1, 5, 5])
    global spread_kernel
    dpv_permuted = dpv.permute(0, 3, 2, 1)
    if spread_kernel is None:
        kernel = torch.Tensor(np.zeros((N, N)).astype(np.float32))
        kernel[int(N / 2), :] = 1.
        print(kernel)
        # kernel[2,2] = 1.
        kernel = kernel.unsqueeze(0).unsqueeze(0)
        kernel = kernel.repeat((1, 1, 1, 1))
        print(kernel.shape)
        kernel = {'weight': kernel.to(dpv_permuted.device), 'padding': N // 2}
        spread_kernel = kernel.copy()

    for b in range(0, dpv_permuted.shape[0]):
        for c in range(0, dpv_permuted.shape[1]):
            dpv_permuted[b,c,:,:] = F.conv2d(dpv_permuted[b,c].unsqueeze(0).unsqueeze(0), **spread_kernel).squeeze(0).squeeze(0)

    dpv = dpv_permuted.permute(0, 3, 2, 1)
    tofuse_dpv = dpv / torch.sum(dpv, dim=1).unsqueeze(1)
    return tofuse_dpv
</code></pre>
<p>They are suppose to be equivalent in theory. But both give completely different output</p>

username: Nikronic
cooked: <p>Sorry for late answer, here is the idea.<br>
If you look up the definition of multi-channel cross-correlation which is also available in <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html" rel="noopener nofollow ugc">Conv2d docs</a>, you can see below formula:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01e36d545ec25518b4523087d36b8bc5cc2ca940.png" alt="image" data-base62-sha1="gHJJMNP8fSrTe7wfqoAPXTHkac" width="634" height="128"></p>
<p>It says, for each output channel, you need to combine correlation results using sum. In your code, you have removed the correlation between different input channels.<br>
Let’s talk intuitively.<br>
If we have an input tensor <code>X</code> with size <code>[in_channel, h, w]</code> and we want to have 3 output channels for the result <code>[3, h, w]</code>, then we need to convolve <code>x</code> with a <code>[3, k, k]</code> kernel 3 times and concatenate the results. But here is the idea, if kernel is same for all the three times, should not the result also be same as both <code>x</code> and <code>kernel</code> are identical for each output channel? The answer would be yes which can be achieved by that summation, if we remove that we break this.</p>
<p>Here is your code with some modifications:</p>
<pre><code class="lang-auto">
torch.manual_seed(0)

spread_kernel = None
kernel_ = None
def spread_dpv(dpv, N=5):
    global spread_kernel
    global kernel_
    
    dpv_permuted = dpv.clone()
    kernel = torch.Tensor(np.zeros((N, N)).astype(np.float32))
    kernel[int(N / 2), :] = 1.
    kernel = kernel.repeat((dpv_permuted.shape[1], dpv_permuted.shape[1], 1, 1)).to(dpv_permuted.device)
    kernel_ = kernel
    dpv_permuted = F.conv2d(input=dpv_permuted, weight=kernel, padding=N // 2)

    dpv = dpv_permuted
    return dpv

spread_kernel = None
def spread_dpv_hack(dpv, N=5):
    global spread_kernel
    dpv_permuted = dpv.clone()
    if spread_kernel is None:
        kernel = torch.Tensor(np.zeros((N, N)).astype(np.float32))
        kernel[int(N / 2), :] = 1.
        kernel = kernel.unsqueeze(0).unsqueeze(0)
        kernel = kernel.repeat((1, 1, 1, 1))
        kernel = {'weight': kernel.to(dpv_permuted.device), 'padding': N // 2}
        spread_kernel = kernel.copy()

    for b in range(0, dpv_permuted.shape[0]):
        for c in range(0, dpv_permuted.shape[1]):
            dpv_permuted[b,c,:,:] = F.conv2d(dpv_permuted[b:b+1,c:c+1], **spread_kernel)

    dpv = dpv_permuted
    return dpv


x = torch.randint(1, 3, (1, 3, 7, 7)).float()
s = spread_dpv(x)
sh = spread_dpv_hack(x)
</code></pre>
<p>modifications:</p>
<ol>
<li>PyTorch uses channel first conv, so you should remove <code>.permute</code> lines unless explicitly tell conv that is channel last.</li>
<li>For simplicity I removed normalization parts.</li>
<li>You were creating different kernel for first method by dividing <code>/ float(N)</code> which had not been used for second method.</li>
</ol>
<p>If you take <code>sh</code> which is the output of your method, then sum wrt channels you will get a output channel for first dimension of <code>s</code>.</p>
<pre><code class="lang-auto">np.sum(sh.numpy(), axis=(0,1))
</code></pre>
<p>Also, we can express it by summing the way I initalized kernel:</p>
<pre><code class="lang-auto">
F.conv2d(x[:,0:1], kernel_[0:1, 0:1], padding=2) + F.conv2d(x[:,1:2], kernel_[0:1, 1:2], padding=2) + F.conv2d(x[:,2:3], kernel_[0:1, 2:3], padding=2)
</code></pre>

username: soulslicer
cooked: <p>I tried your modified code and it still doesnt work. It needs to be a drop in replacement for my spread_dpv_hack method. I dont want there to be any correlation between input channels.</p>
<p>Lets simply my problem. Imagine you have a 100 images that you want to apply a 3x3 blur filter. And those 100 images are in a batch of 6. So you have [6,100, r, c]. I just want to apply the 3x3 blur filter across batches and channels. Why is this so difficult</p>

username: chetan_patil
cooked: <p>Hi,<br>
If I’m not wrong, you have an input of <code>[6,100,h,w]</code> and you want to apply 100 such 3x3 kernels on them such that each Kernel will be applied on only  one channel, this leading to output size of <code>[6,100,h,w]</code>.<br>
Right ?<br>
If so, you can use this<br>
<code>nn.Conv2d(in_channels=100, out_channels=100, groups=100, kernel_size= (3,3))</code></p>

username: Nikronic
cooked: <p>Ok,<br>
I thought you still looking to hold correlation between channels, now I understand. In this case <a class="mention" href="/u/chetan_patil">@chetan_patil</a> 's explanation is the correct way. Sorry about my misinterpretation.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/7791437bb1edccec7277fc0b3221f4f0841b1de8.png" alt="image" data-base62-sha1="h3K2MiTpEOiWSgXnhma9AKaVlwY" width="669" height="225"></p>

username: soulslicer
cooked: <aside class="quote no-group" data-username="chetan_patil" data-post="6" data-topic="87328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chetan_patil/48/25122_2.png" class="avatar"> chetan_patil:</div>
<blockquote>
<p>Conv2d</p>
</blockquote>
</aside>
<p>Can you please provide some example code</p>

username: chetan_patil
cooked: <aside class="quote no-group" data-username="chetan_patil" data-post="6" data-topic="87328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chetan_patil/48/25122_2.png" class="avatar"> chetan_patil:</div>
<blockquote>
<p>nn.Conv2d(in_channels=100, out_channels=100, groups=100, kernel_size= (3,3))</p>
</blockquote>
</aside>
<p>Hello <a class="mention" href="/u/soulslicer">@soulslicer</a>, here it is.</p>
<pre><code class="lang-auto">ip = torch.randn(6, 100, 8, 10)
conv = nn.Conv2d(in_channels=100, out_channels=100, groups=100, kernel_size= (3,3))
op = conv(ip)
print(op.shape)
torch.Size([6, 100, 6, 8])


</code></pre>