Custom nn.Conv2d

Sami_Hassan: <p>Hello, This is my first week as a PyTorch user.</p>
<p>How can we define a custom Conv2d function which works similar to nn.Conv2d but the multiplication and addition used inside nn.Conv2d are replaced with mymult(num1,num2) and myadd(num1,num2). It’s for some hdl simulation purpose. Even if it’s a slow implementation it doesn’t matter. We would like to test our theory on inference by loading already trained weights.</p>

ptrblck: <p>I think the easiest way would be to use <code>unfold</code> to create patches and apply your custom functions on each patch.<br>
<a href="https://discuss.pytorch.org/t/convolution-that-only-take-channel-wise-summation/21240/4">This post</a> might be a good starter.</p>

Sami_Hassan: <p>Can i do it like this <a href="https://discuss.pytorch.org/t/custom-a-new-convolution-layer-in-cnn/43682">https://discuss.pytorch.org/t/custom-a-new-convolution-layer-in-cnn/43682</a> ?</p>

ptrblck: <p>If that fits your use case better, go for it.<br>
<code>unfold</code> is basically the im2col operation, which would create the patches for each operation with a kernel.</p>

zahra: <p>Dear Sami,</p>
<p>Mentioned link “<a href="https://discuss.pytorch.org/t/custom-a-new-convolution-layer-in-cnn/43682/2" class="inline-onebox">Custom a new convolution layer in cnn</a>” is useful but it is just about forward pass of convolution layer. Backward pass have its own problems. For example, you should manage the dimension of produced gradient in your evaluations.</p>
<p>I suggest you to use Pytorch functions. Since Convolution is equivalent with Unfold + Matrix Multiplication + Fold maybe what ptrblck recommended you would be more useful and simpler.</p>
<p>I mean that whatever you can use ready Pytorch functions and I think most of the time there is a way instead of customizing layers.</p>
<p>If you want to customize a layer, I suggest you to check the open source implementation of multi-layer perceptron and convolution layer and their theories. Also, these two tutorials could help you:</p>
<p><a href="https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/numpy_extensions_tutorial.html</a></p>
<p><a href="https://towardsdatascience.com/only-numpy-understanding-back-propagation-for-transpose-convolution-in-multi-layer-cnn-with-c0a07d191981" rel="nofollow noopener">https://towardsdatascience.com/only-numpy-understanding-back-propagation-for-transpose-convolution-in-multi-layer-cnn-with-c0a07d191981</a></p>
<p>Of course, what ptrblck sayes is more professional than mine.</p>
<p>Good luck</p>

ptrblck: <aside class="quote no-group" data-username="zahra" data-post="5" data-topic="62068">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/aeb1de/48.png" class="avatar"> zahra:</div>
<blockquote>
<p>Of course, what ptrblck sayes is more professional than mine.</p>
</blockquote>
</aside>
<p>Oh, I wouldn’t say that. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
I’m just posting some suggestions and different approaches always help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Sami_Hassan: <p>Thanks alot for your replies. I am a little bit confused , apologies in advance as i am just a beginner. Do i have to override some function in nn.Conv2d() and use that unfolding part in it ? Can you please point me to some example.</p>

ptrblck: <p>You don’t have to override the conv functionality, but could directly implement your custom convolution using <code>unfold</code>.<br>
The link in my first example gives you an example of a vanilla convolution or are you looking for something else?</p>

Sami_Hassan: <p>Let me try to elaborate what is my goal Consider an example of Lenet - 5</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        
        super(Net,self).__init__()
        self.conv1=nn.Conv2d(1,6,5)
        self.conv2=nn.Conv2d(6,16,5)
        self.fc1 = nn.Linear(400,120)
        self.fc2 = nn.Linear(120,84)
        self.classifier = nn.Linear(84,10)
        self.features = nn.Sequential(*list(self.children()))


    



    def forward(self,x):
        x = F.max_pool2d(F.relu(self.conv1(x)),2)
        x = F.max_pool2d(F.relu(self.conv2(x)),2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        #x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x


    def num_flat_features(self,x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features*=s
        return num_features

model = torch.load('lenet5.pth')
#print(model.values())

model_new = Net()    # before loading trained weights




model_new.load_state_dict(torch.load('lenet5.pth'))  # After loading trained w

with torch.no_grad():
	correct = 0
	total = 0
	for images,labels in test_loader:
		#images = images.reshape(-1,28*28)
		out = model_new(images)
		_,predicted = torch.max(out.data,1)
		total += labels.size(0)
		correct += (predicted==labels).sum().item()
		print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))



</code></pre>
<p>I am suppose to check accuracy of the network after substituting the nn.conv2d multiplication/addition using my two functions mymult(num1,num2) and myadd(num1,num2). The same has to be done with some other already trained neural networks. This is a short example, we can even choose them from inside pytorch too.</p>

ptrblck: <p>You could use my linked code snippet to implement your custom convolution and substitute <code>nn.Conv2d</code> with your class.<br>
The easiest way would be to use my code in your <code>forward</code> method and set the <code>weight</code> and <code>bias</code> in the <code>__init__</code> of your <code>MyConv2d</code> class.</p>

Sami_Hassan: <p>These is how it looks like from what i understood from all of your posts</p>
<pre><code class="lang-auto">class Myconv2D(torch.autograd.Function):

    # Note that both forward and backward are @staticmethods
    @staticmethod
    # bias is an optional argument
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
       
        # input dim [batch_size, channels, height, width]

        batch_size = len(input)
        channels = input[0][0]
        h, w = input[0][0][0], input [0][0][0]
        image =  torch.tensor(batch_size, channels, h, w) # input image
        #print (image)
        #print (image.size())
        #print (image[0])

        kh, kw = 3, 3# kernel size
        dh, dw = 2, 2 # stride

        filt = nn.parameter(channels, kh, kw) # filter (create this as nn.Parameter if you want to train it)
        #print (filt)
        patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
        #print(patches.shape)
        patches = patches.contiguous().view(batch_size, channels, -1, kh, kw)
        #print(patches.shape)
        patches = patches.permute(0, 2, 1, 3, 4)
        patches = patches.view(-1, channels, kh, kw)
        #print (patches.shape)
        #print ("Filter shape",filt.shape)
        #print(patches)
        #print (patches,"\n","\n","\n",filt)
        #print ("Builtin multiplication")
        
        dummy = patches

        patches = patches * filt # same is done below with 4 nested loops with custom operation
        #print(patches)
        #print ("Custom Multiplication starts here:")
        #print(patches[0][0][0][0])
        #print(filt)
        
        for b in range(batch_size* int(h/3)* int(h/3)):

            for c in range(channels):

                for height in range(kh):

                    for width in range(kw):

                        patches[b][c][height][width]   = \   
                        dummy[b][c][height][width] * filt[c][height][width]  replace this multiply(*) with my function (mymult(num1,num2))


        #print(patches)
        patches = patches.sum(1) # previous and this patch are same what is it doing
        #print (patches)
        output = patches # is it right ?
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    # This function has only a single output, so it gets only one gradient
    @staticmethod
    def backward(ctx, grad_output):
        # This is a pattern that is very convenient - at the top of backward
        # unpack saved_tensors and initialize all gradients w.r.t. inputs to
        # None. Thanks to the fact that additional trailing Nones are
        # ignored, the return statement is simple even when the function has
        # optional inputs.
        input, weight, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None

        # These needs_input_grad checks are optional and there only to
        # improve efficiency. If you want to make your code simpler, you can
        # skip them. Returning gradients for inputs that don't require it is
        # not an error.
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias


class Myconv2D(nn.Module):
    def __init__(self):
        super(Myconv2D, self).__init__()
        self.fn = Myconv2D.apply
        # weight tensor = out_channels× in_channels/groups ×kH×kW
        self.weight = nn.Parameter(torch.randn(1, 1, 2, 2)) # when groups=1

    def forward(self, x):
        x = self.fn(x, self.weight)
        return x

</code></pre>
<p>The multiplication answers are the same as<br>
patches = patches * filt     and the custom 4-Nested loop structure in forward method of class Myconv2D(torch.autograd.Function)<br>
After that there is addition  “patches = patches.sum(1)” i am not sure what is it doing , I would like to replace the addition as well.</p>
<p>Can you please have a look at it.</p>
<p>I will be using this in the previous code.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    
    def __init__(self):
        
        super(Net,self).__init__()
        self.conv1=nn.Myconv2D(1,6,5)
        self.conv2=nn.Myconv2D(6,16,5)
        self.fc1 = nn.Linear(400,120)
        self.fc2 = nn.Linear(120,84)
        self.classifier = nn.Linear(84,10)
        self.features = nn.Sequential(*list(self.children()))
</code></pre>
<p>Is this the right direction. I am really Thankful for your time and effort. I wouldn’t even be able to start with this without your comments.</p>

ptrblck: <p>There seem to be a few minor mistakes in the code:</p>
<ul>
<li><code>h</code> and <code>w</code> should probably de defined as <code>input.size(2)</code> and <code>input.size(3)</code>, respectively.<br>
Currently you are assigning the same input value to both. However, since you are passing <code>input</code> directly, they can also be removed completely.</li>
<li>As already mentioned, you are passing <code>input</code>, so I’m not sure what <code>image</code> should be used for.</li>
<li>You are also passing the <code>weight</code> to <code>forward</code>, which would make <code>filt</code> unnecessary. Also, you are currently re-initializing <code>filt</code> in each forward pass, which won’t store the already learned weights, so use <code>weight</code> instead for your operations.</li>
<li>Are you using the nested loop to compare the outputs of <code>patches * filt</code> or what is it used for?</li>
<li><code>patches.sum(1)</code> calculates the sum over dim1. I’m currently not sure, which shape <code>patches</code> would have at that point.</li>
</ul>
<aside class="quote no-group" data-username="Sami_Hassan" data-post="11" data-topic="62068">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sami_hassan/48/18106_2.png" class="avatar"> Sami_Hassan:</div>
<blockquote>
<p>After that there is addition “patches = patches.sum(1)” i am not sure what is it doing , I would like to replace the addition as well.</p>
</blockquote>
</aside>
<p>How would you like to replace the addition?<br>
Also, could you print the shape of <code>patches</code> before calculating <code>sum(1)</code> and name all dimensions, so that we can understand the use case better? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Sami_Hassan: <p>Thanks a ton <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It did the job</p>

AG1991: <p>Dear Sami, could you please provide me with your final code? I am working on a similar problem. Thanks in advance</p>

Sami_Hassan: <p>Kindly look at the following code. <a href="https://stackoverflow.com/questions/59149785/custom-conv2d-operation-pytorch" rel="nofollow noopener">A</a>. I am trying to vectorize it as it is dead slow.</p>