Setting custom kernel for CNN in pytorch

markroxor: <p>Is there a way to specify our own custom kernel values for a convolution neural network in pytorch? Something like <a href="https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/layers/conv2d_transpose" rel="nofollow noopener">kernel_initialiser</a> in tensorflow? Eg. I want a 3x3 kernel in <code>nn.Conv2d</code> with initialization so that it acts as a identity kernel -</p>
<p><code>0 0 0</code><br>
<code>0 1 0</code><br>
<code>0 0 0</code></p>
<p>(this will effectively return the same output as my input in the very first iteration)</p>
<p>My non-exhaustive research on the subject -</p>
<p>I could use <a href="https://pytorch.org/docs/stable/_modules/torch/nn/init.html" rel="nofollow noopener">nn.init</a> but it only has some pre-defined kernel initialisaition values.</p>
<p>I tried to follow the discussion on this <a href="https://discuss.pytorch.org/t/weight-initilzation/157/2">thread</a> but it doesn’t suit my needs.</p>
<p>I might have missed something in my research please feel free to point out.</p>
<p>I have asked the same on SO <a href="https://stackoverflow.com/questions/52790775/setting-custom-kernel-for-cnn-in-pytorch" rel="nofollow noopener">here</a> but couldn’t find any answer.</p>

ptrblck: <p>I think the easiest way would be to use the functional API.<br>
You would have to define the weights and use <code>F.conv2d</code> to apply the convolution.<br>
Here is a small example:</p>
<pre><code class="lang-python">nb_channels = 1
h, w = 5, 5
x = torch.randn(1, nb_channels, h, w)
weights = torch.tensor([[0., 0., 0.],
                        [0., 1., 0.],
                        [0., 0., 0.]])
weights = weights.view(1, 1, 3, 3).repeat(1, nb_channels, 1, 1)

output = F.conv2d(x, weights)
</code></pre>

markroxor: <p>Thank you for your response.</p>
<p>I already know about <code>F.conv2d</code> but I wanted to use kernels not just for convolution but for CNN (<code>nn.Conv2d</code>) where learning of weights take place.</p>
<p>I don’t think <code>F.conv2d</code> will help. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=6" title=":frowning:" class="emoji" alt=":frowning:"></p>

ptrblck: <p>You would need to set <code>requires_grad=True</code> for the weights and it would also work as <code>nn.Conv2d</code> internally just calls the functional API, see <a href="https://github.com/pytorch/pytorch/blob/1f94ce1f97d95e0d20540accea3671ee8ff2dec3/torch/nn/modules/conv.py#L311">here</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>
<p>However, if you prefer to use the module, you could try the following code:</p>
<pre><code class="lang-python">weights = ...
conv = nn.Conv2d(nb_channels, 1, 3, bias=False)
with torch.no_grad():
    conv.weight = nn.Parameter(weights)

output = conv(x)
output.mean().backward()
print(conv.weight.grad)
</code></pre>

markroxor: <p>That makes sense, thank you. Let me try it out. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vijaytida: <p>Hi,<br>
I want to cast the data once they completed dot product before addition in f.conv2d. Can you please give me any suggestion ?</p>

ptrblck: <p>As far as I understand you would like to split the convolution operation and add a custom op before the summation.<br>
If so, I think you would need to implement the conv op manually using e.g. <code>unfold</code>.</p>

vijaytida: <p>How can I see the algorithmic implementation of convolution using pytorch?</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp" rel="nofollow noopener">aten/src/ATen/native/Convolution.cpp</a> might be a good starting point to see which algorithms are being dispatched.</p>

Christopher_Aykroyd: <p>So, how does it work when I have multiple channels?<br>
For example, the below code will output tensors of shape (B, 2, W, H). Instead, I would like the two kernels to be each applied to each channel with output image (B, 2C, W, H)</p>
<pre><code class="lang-auto">def get_grad_kernel(channels):
  Iy = [[ 1, 2, 1],
        [ 0, 0, 0],
        [-1,-2,-1]]
  Ix = [[-1, 0, 1],
        [-2, 0, 2],
        [-1, 0, 1]]
  return torch.Tensor([ [Ix]*channels, [Iy]*channels ])/4

grad_kernel = get_grad_kernel(channels=3)
compute_gradient = lambda image: F.conv2d(input=image, weight=grad_kernel, padding=1)
</code></pre>
<p>EDIT: From ptrblck’s reply, making the kernel as the following</p>
<pre><code class="lang-auto">  return torch.Tensor([ [Ix], [Iy] ]*channels)/4
</code></pre>
<p>and setting groups=image.size(1) works nicely</p>

ptrblck: <p>Setting <code>groups=in_channels</code> might work for your use case.</p>

Sepehr.nem: <p>Thanks for your informative answers.<br>
I am struggling to add more than one weight (let’s say 2 kernels 3x3) in conv2d() to get more than one output at the same time (nn.Conv2d(nb_channels, 2, 3, bias=False).<br>
Could you please help me out with this</p>

ptrblck: <p>The number of kernels is defined in <code>dim0</code> in the weight matrix and you could use this modified code for it:</p>
<pre><code class="lang-python">nb_channels = 1
h, w = 5, 5
x = torch.randn(1, nb_channels, h, w)
weights = torch.tensor([[[0., 0., 0.],
                         [0., 1., 0.],
                         [0., 0., 0.]],
                        [[0., 0., 0.],
                         [0., 1., 0.],
                         [0., 0., 0.]]])
weights = weights.view(2, 1, 3, 3)

output = F.conv2d(x, weights)
</code></pre>
<p>Let me know, if this would work for you or if you get stuck somewhere. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Sepehr.nem: <p>Thanks a lot for your answer.<br>
This works out for me but I want to implement the discussed Conv2D weights into this simple network:</p>
<p>class CNN(nn.Module):</p>
<pre><code>def __init__(self):
    super(CNN, self).__init__()
    self.features =nn.Conv2d(1, 2, kernel_size=3, stride=1, padding=0)
                           nn.ReLU(),
                           nn.MaxPool2d(kernel_size=2, stride=1)
                                 )
    self.drop_out = torch.nn.Dropout(0.6)

def forward(self, x):
    x=self.features(x)
    x=self.drop_out(x)
    return x
</code></pre>
<p>cnn=CNN()</p>
<p>I would be appreciated if give me a hint on this as well<br>
Thanks</p>

ptrblck: <p>You can assign a new <code>weight</code> parameter to <code>self.features</code> via:</p>
<pre><code class="lang-python">self.features = nn.Conv2d(...)
with torch.no_grad():
    weights = torch.tensor(...)
    self.features.weight = nn.Parameter(weights)
</code></pre>

Sepehr.nem: <p>Thank you for your answer</p>

spacemeerkat: <p>Is there a way to adapt this approach so that rather than performing convolutions using different kernels per channel, it performs <em>convolutions on all channels for a tensor in a batch but the kernel for each tensor in the batch changes</em>? So each batch item undergoes a convolution with a unique prespecified kernel.</p>
<p>e.g. something like this:</p>
<pre><code class="lang-auto">import torch 

batch_size = 8
channels = 10
img_size = 30
kernel_size = 3

batch = torch.rand((batch_size,channels,img_size,img_size))

# Make a unique kernel for each batch member but the kernel is convolved 
# with every channel 
weights = torch.rand((batch_size,1,kernel_size,kernel_size)).repeat(1,channels,1,1)
print(weights.shape)

conv = torch.nn.Conv2d(channels,channels,kernel_size,padding=4,bias=False)

with torch.no_grad():
    conv.weight = torch.nn.Parameter(weights,requires_grad=False)

output = conv(batch)
print(output.shape)
</code></pre>
<hr>
<p>Edit: This has been solved using for loops or groups  <a href="https://discuss.pytorch.org/t/how-to-apply-different-kernels-to-each-example-in-a-batch-when-using-convolution/84848/4">here</a></p>

Lukman_Hakim: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> I try this code but I got error "Illegal instruction (core dumped)<br>
"</p>

ptrblck: <p>I’m not sure which code you were trying. Could you post more information about your use case, setup (installed PyTorch version, CUDA etc.) and could you also try to get a backtrace from <code>gdb</code>?</p>

fevr: <p>Hello, I test the two methods mentioned above</p>
<pre><code class="lang-auto">nb_channels = 1
h, w = 5, 5
x = torch.ones(1, nb_channels, h, w)
weights = torch.tensor([[0.,0., 0.],
                        [0, 1., 0],
                        [0., 0., 0.]])
weights = weights.view(1, 1, 3, 3).repeat(1, nb_channels, 1, 1)
output = F.conv2d(x, weights)
</code></pre>
<p>output:<br>
tensor([[[[1., 1., 1.],<br>
[1., 1., 1.],<br>
[1., 1., 1.]]]])</p>
<pre><code class="lang-auto">class train_model(nn.Module):
    def __init__(self,n_feature,n_out):
        super(train_model, self).__init__()
        self.conv4 = nn.Conv2d(n_feature, n_out, 3, padding=0)
        with torch.no_grad():
            weights = torch.tensor([[0.,0., 0.],
                        [0, 1., 0],
                        [0., 0., 0.]])
            weights = weights.view( 1,1, 3, 3).repeat(1, n_feature, 1, 1)
            self.conv4.weight = nn.Parameter(weights)
        
    def forward(self, x):
        x = self.conv4(x)
        return x

model = train_model(n_feature=1,n_out=1)
x = torch.ones(1, 1,5, 5)
output=model(x)
</code></pre>
<p>I thought the two outputs should be the same (?), but the second one is not even constant.<br>
I can’t figure out why</p>