How to implement multiple different kernel shapes

username: Chuong_Vo
cooked: <p>Hello. I’m currently working on spherical convolutional network topic. Right now I’m trying to develop a new kind of kernel used for the convolutional layer.<br>
The usual kernel is 3x3 matrix. But for spherical images, after being projected onto a plane using equirectangular projection, there will be distortion. So I want to define the kernel as a spherical cap and project it on plane according to its position.<br>
For example, the kernel at different positions of the sphere perspective to the panorama pictures will look like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1003b38d943f21833c298ee5817281efc858cf3c" title="Positions"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_690x172.png" alt="Positions" data-base62-sha1="2hFzjUkSEbSOewOFynYpwEHP828" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_1380x344.png 2x" data-dominant-color="FBFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Positions</span><span class="informations">1680×420 14.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any way to determine the shape of the kernels in these ways? I have already had the whole coordinate of the points in every case. I would very appreciate any help and information.<br>
Thank you guys very much!</p>

username: Chuong_Vo
cooked: <p>Hi, can anyone provide any idea or method of defining custom shape of kernel please give some help!<br>
Thank you!<img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>

username: ptrblck
cooked: <p>I’m not sure I understand the use case completely, but would zeroing out specific spatial positions of a square kernel work (also zeroing out the gradients of these positions)?</p>

username: Chuong_Vo
cooked: <p>Hi. Thank you for replying!<br>
Actually, I’m going to change this method since it seems to have too high convolution cost.<br>
My concern right now is to change the input’s pixel accordingly before performing the convolution. May I continue to ask you about my new concern here or should I open a new topic?</p>

username: ptrblck
cooked: <p>Please continue your explanations here. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Chuong_Vo
cooked: <p>Thank you! You are so kind!<br>
My problem is a little hard to understand so I will try my best to express it. Hope you could understand this.<br>
So the convolution I would like to perform is in conv2d case.<br>
Instead of changing the shape of the kernel so many times, I would like to keep the process of conv2d the same about kernel shape (3x3) and the way natural conv2d should perform in learning,etc.<br>
The thing I want to do can be said in this way:<br>
First, In the normal way of 2d convolution, the multiplication between the kernel and the input should be in  this way:<br>
<img src="https://raw.githubusercontent.com/vdumoulin/conv_arithmetic/master/gif/no_padding_no_strides.gif" alt="" role="presentation" width="244" height="259"><br>
But now, for every small 3x3 part of the input, I will only keep the center pixel. The surrounding pixel of the 3x3 part of the input will be replaced by the corresponding pixels in other positions on the input(which I already knew). The example of the surrounding pixels’ positions is just like on the picture of the first question:</p>
<aside class="quote no-group quote-modified" data-username="Chuong_Vo" data-post="1" data-topic="65334">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/b2d939/48.png" class="avatar"> Chuong_Vo:</div>
<blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1003b38d943f21833c298ee5817281efc858cf3c" title="Positions"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_690x172.png" alt="Positions" data-base62-sha1="2hFzjUkSEbSOewOFynYpwEHP828" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/1003b38d943f21833c298ee5817281efc858cf3c_2_1380x344.png 2x" data-dominant-color="FBFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Positions</span><span class="informations">1680×420 14.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Positions1680×420 14.7 KB</p>
</blockquote>
</aside>
<p>These new pixels and the center pixel will come together and form a new 3x3 matrix that will be used to multiply with the kernel in a normal way. So I technically only interfere the first step of 2D convolution.<br>
Is there anyway of implementing that? Thank you!</p>

username: ptrblck
cooked: <p>Thanks for the detailed description!<br>
I think the best shot at the problem would be to use the functional API and to recreate the patches as you need them.</p>
<p>Since you only want to keep the center pixel, you could probably just use indexing of the input tensor to get these values.<br>
How would you get the outer values? Do you have some kind of mapping or would also indexing / gather work?<br>
As you can see I’m quite unsure about the creation of these input patches, so feel free to give some (code) examples.<br>
Once you have the input patches, you could use <code>F.conv2d</code> and your <code>weight</code> parameter to perform the vanilla convolution.</p>

username: Chuong_Vo
cooked: <p>Thank you for your suggestion!<br>
Right now I have the idea of implementing the creation of matrix used for each multiplication with the kernel like in the code below:</p>
<pre><code class="lang-auto">import numpy as np

coor_dict ={
    
    (1,1): np.matrix([[0, 263, 536, 852, 1920, 2988, 3304, 3577], [2, 2, 1, 0, 0, 0, 1, 2]])
    #etc...
}
coordinate = np.matrix([1,1])
indices = np.asmatrix(coor_dict[(coordinate[0,0],coordinate[0,1])])

image = np.random.uniform(low=0, high=255, size=(2160,3840))

patch = np.zeros((3,3))
patch = image[0:3,0:3]

print('Old patch: \n', patch)

patch[2,1] = image[indices[1,0],indices[0,0]]
patch[2,2] = image[indices[1,1],indices[0,1]]
patch[1,2] = image[indices[1,2],indices[0,2]]
patch[0,2] = image[indices[1,3],indices[0,3]]
patch[0,1] = image[indices[1,4],indices[0,4]]
patch[0,0] = image[indices[1,5],indices[0,5]]
patch[1,0] = image[indices[1,6],indices[0,6]]
patch[2,0] = image[indices[1,7],indices[0,7]]

print('new patch: \n', patch)
</code></pre>
<p>In this case, the input image would be w=3840, h=2160. I have a dictionary containing all the indices of the surrounding pixels on the input image corresponding to each center pixel of the 3x3 patch (In this case: (1,1).<br>
After performing the multiplication between this patch and the kernel, we can continue to form a new patch in the next position. I would like the convolution to have padding = 1 and stride = 1 to have the same size output but increased in number of channels.<br>
I am just a beginner in both python and pytorch so if you can provide some detail instructions, it would mean a huge favor to me!</p>

username: Chuong_Vo
cooked: <p>What I would like to do now is to inspect the part of the source code that perform the task of getting each 3x3 component from the input for each multiplication step in convolution. Can you please show me the file that has this task?<br>
If I want to change this file and apply to my project what should I do?<br>
Thank you!</p>

username: ptrblck
cooked: <p>If you want to experiment with the kernels and be flexible, I would recommend to stay on the Python side for now and use <code>torch.unfold</code> to create the patches as described <a href="https://discuss.pytorch.org/t/creating-nonoverlapping-patches-from-3d-data-and-reshape-them-back-to-the-image/51210/6">here</a>. Once you have the patches, you can apply your custom conv operation, reshape it back etc.</p>
<p>If you really want to manipulate the backend implementation, convolutions are dispatched <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp">here</a>.</p>

username: Chuong_Vo
cooked: <p>Thank you for the suggestion!<br>
I think that using <code>torch.unfold</code> maybe the solution for my case. But I’m finding it hard to understand how this function work and how this can be used to define a custom convolution layer. Can you provide me with some instructions?</p>

username: ptrblck
cooked: <p>The linked post gives you an example of how to create patches using a kernel size and a stride.<br>
You can ignore the padding as well as the “reshape back” section for now.</p>
<p>Each patch will represent basically the sliding window approach of a convolution, so that you can apply any method on it.</p>

username: Chuong_Vo
cooked: <p>Now what I’m able to do is to take an image of size(2160x3840), divide it into patches of size (3x3) with stride = 1. Then, I have a set of patches that I can access and change them.<br>
The code that I figured out from the link you provided is as below.</p>
<pre><code class="lang-auto">import torch
from torch.nn import functional as F
x = torch.randn(1, 1, 2160, 3840)
kc, kh, kw = 1, 3, 3  # kernel size
dc, dh, dw = 1, 1, 1  # stride

patches = x.unfold(1, kc, dc).unfold(2, kh, dh).unfold(3, kw, dw)
unfold_shape = patches.size()
print(unfold_shape)
patches = patches.contiguous().view(-1, kc, kh, kw)
print(patches.shape)
</code></pre>
<p>After applying changes on this set of patches. It will still have the same shape. Now is the time to perform convolution.<br>
Since it has stride 1, the patches will intersect each other. Since after the processing, the values of the intersecting patches are not the same. Therefore, the way of combining these patches back into 1 new picture seems impossible. I’m thinking about defining a custom convolution layer for this case.<br>
Is there a way to implement this for CNN? Thank you!</p>

username: ptrblck
cooked: <p>Note that a convolution performs a reduction.<br>
Each patch will be multiplied with the kernel and the sum of the result will be calculated yielding a single scalar value for each patch.<br>
These scalar outputs will create the activation output as is also done internally in the convolution implementations.</p>

username: Chuong_Vo
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Right now I’m trying to divide the image into 3x3 patches, process them and then stitch them back together  into a new one.<br>
There is a bit of a problem with my processing step: it takes too much time.<br>
Here is my code for the process:</p>
<pre><code class="lang-auto">coor = {(0,0) : np.matrix([[0, 263, 536, 852, 1920, 2988, 3304, 3577], [2, 2, 1, 0, 0, 0, 1, 2]])
        ...
       }
patches = torch.randn(921600, 1, 3, 3)

def change_patches(patches, coor, i, j):
    coordinate = [i+1,j+1]
    indices = torch.tensor(coor[(coordinate[0],coordinate[1])])
    a = int(coordinate[0]*1280+coordinate[1]/3)
    patches[a,0,2,1] = image[0,0,indices[1,0],indices[0,0]]
    patches[a,0,2,2] = image[0,0,indices[1,1],indices[0,1]]
    patches[a,0,1,2] = image[0,0,indices[1,2],indices[0,2]]
    patches[a,0,0,2] = image[0,0,indices[1,3],indices[0,3]]
    patches[a,0,0,1] = image[0,0,indices[1,4],indices[0,4]]
    patches[a,0,0,0] = image[0,0,indices[1,5],indices[0,5]]
    patches[a,0,1,0] = image[0,0,indices[1,6],indices[0,6]]
    patches[a,0,2,0] = image[0,0,indices[1,7],indices[0,7]]
    return(patches)
</code></pre>
<p>I’m trying to find a way of implementing this with less time. Since this process is sequential, I want to perform this in parallel. I found out about <code>multiprocessing.pool</code> which has <code>starmap()</code> function that can only work on CPU. I noted that pytorch also has a similar one as <code>torch.multiprocessing</code>.<br>
How do I use this one? Can you provide me with an example on my case?<br>
Thank you very much!!!</p>

username: Chuong_Vo
cooked: <p>Can you help with my problem? Thank you!</p>