How to test my own Conv2d implementation (just as a proof-of-concept)

vdw: <p>I’m trying to implement my own Conv2d layer for self-study and educational purposes. I feel quite confident that I understand the basic idea and steps – images like the one below illustrate the steps pretty well:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/137810d24456cf60d9ff0cfb5555b7c94de75bb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/137810d24456cf60d9ff0cfb5555b7c94de75bb4" title="conv2d-illustration"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/137810d24456cf60d9ff0cfb5555b7c94de75bb4_2_557x500.png" alt="conv2d-illustration" data-base62-sha1="2MejjwU3J2a1G6d1EbiTiSzGyxe" width="557" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/137810d24456cf60d9ff0cfb5555b7c94de75bb4_2_557x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/137810d24456cf60d9ff0cfb5555b7c94de75bb4_2_835x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/137810d24456cf60d9ff0cfb5555b7c94de75bb4.png 2x" data-dominant-color="EEEAEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">conv2d-illustration</span><span class="informations">974×874 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My current implementations looks as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

class VanillaConv2d(nn.Module):
        
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, padding=0):
        super().__init__()
        self.pad, self.stride = padding, stride
        # Note: kernel_size is always a tuple (not an int like for nn.Conv2d)
        self.filter_weights = nn.Parameter(torch.rand(out_channels, in_channels, kernel_size[0], kernel_size[1]))
        self.filter_biases = nn.Parameter(torch.rand(out_channels, ))
        
        
    def forward(self, X, padding_value=0):
        
        N, C, H,  W  = X.shape
        F, C, HH, WW = self.filter_weights.shape
        
        # Pad intputs
        X_pad = nn.functional.pad(X, (self.pad, self.pad, self.pad, self.pad), mode='constant', value=padding_value)
        
        # Get the height and weight of the padded inputs
        _, _, H_pad, W_pad = X_pad.shape
        
        # We need to compute the height and width of the outputs based on 
        # * Size of filters
        # * Padding
        # * Stride
        H_out = 1 + (H + 2*self.pad - HH) // self.stride
        W_out = 1 + (W + 2*self.pad - WW) // self.stride
    
        # Define the tensor that will hold the output of the layer
        out = torch.zeros((N,F,H_out,W_out))

        ########################################################################################################
        # Perform convolution operations
        ########################################################################################################
        
        # Looper over each sample in batch
        for n in range(N):
            # For each sample, loop over each filter
            for f in range(F):
                # For each filter, perform convolution (nested loops do the sliding window)
                for h in range(H_out):
                    for w in range(W_out):
                        h_, w_ = h*self.stride, w*self.stride
                        # Get the current patches (i.e., the patches for all channels of current sample)
                        patches = X_pad[n, :, h_:h_+HH, w_:w_+WW] # patches.shape: (C,HH,WW) same as self.filters[f]
                        # Multiple all channels and respective filter element-wise and sum all up
                        out[n, f, h, w] = torch.sum(patches * self.filter_weights[f]) + self.filter_biases[f]

        ########################################################################################################
        
        return out
</code></pre>
<p>Please note that the code is for understanding. I know it’s horrible from a performance point of view, and it’s never intended to be used for actually training a model with it. It’s only important that it performs the convolution(s) as illustrated in the figure above.</p>
<p>I mainly adopted available solutions of Assignment 2 of the Stanford CS321n course; this solution I mainly aligned to. However, this assignment requires an implementation purely using NumPy incl. an implementation of the backward pass. I want to skip this part, and therefore want to implement the layer in Python to make use of <code>autograd</code>. I like to think my code is kind of correct, but I cannot fully convince myself.</p>
<p>Now, I’ve tried to test with a basic example using the MNIST dataset; adopted <a href="https://nextjournal.com/gkoehler/pytorch-mnist" rel="noopener nofollow ugc">this tutorial</a>. When change the code for the model accordingly as follows.</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #self.conv1 = nn.Conv2d(1, 10, kernel_size=(5,5))      # orginal code using nn.Conv2d
        #self.conv2 = nn.Conv2d(10, 20, kernel_size=(5,5))     # which works just fine
        self.conv1 = VanillaConv2d(1, 10, kernel_size=(5,5))
        self.conv2 = VanillaConv2d(10, 20, kernel_size=(5,5))
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)
</code></pre>
<p>the training works but, of course, painfully slow. In fact, so slow that I can’t really use it – I don’t even have a GPU at hand :(. I tried with very small toy data which seem to work. But I fear the the toy dataset is so simple that all learning is done by the fully connected layers, independent what the convolution layers are actually doing.</p>
<p>How can I best test this? Or more basically, is my approach reasonable at all? I’m not sure if <code>autograd</code> handles the sliding window for the convolution as I would expect.</p>
<p><strong>EDIT:</strong> I’ve fixed the bug in the <code>forward()</code> method to work with strides other then 1.</p>

KFrank: <p>Hi Chris!</p>
<aside class="quote no-group" data-username="vdw" data-post="1" data-topic="167005" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vdw/48/10074_2.png" class="avatar"> vdw:</div>
<blockquote>
<p>How can I best test this?</p>
</blockquote>
</aside>
<p>I would instantiate a pytorch <code>Conv2d</code> and then instantiate a <code>VanillaConv2d</code>.</p>
<p>Then set <code>filter_weights</code> and <code>filter_biases</code> of your <code>VanillaConv2d</code> to<br>
be the same as those of <code>Conv2d</code> (or vice versa):</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    vanillaConv.filter_weights.copy_ (conv.weight)
    vanillaConv.filter_biases.copy_ (conv.bias)
</code></pre>
<p>(It looks like you are storing your weights in the same way as <code>Conv2d</code>, but,<br>
if not, transform them appropriately before calling <code>.copy_()</code>.)</p>
<p>Then I would perform a single forward pass on the same single input image<br>
with both <code>vanillaConv</code> and <code>conv</code> and verify that you get the same output<br>
(up to some reasonable round-off error).  Compute some sort of dummy<br>
scalar loss, e.g., <code>loss = output.sum()</code>, perform the backward pass, and<br>
verify that <code>vanillaConv.filter_weights.grad</code> and <code>conv.weight.grad</code><br>
(and also the bias grads) agree (up to some reasonable round-off error).</p>
<p>That is, there is no need to run a full (slow) training loop to test your<br>
<code>VanillaConv2d</code> – as long as it reproduces <code>Conv2d</code> on a single forward and<br>
backward pass (when initialized with the same weights), that’s test enough.</p>
<p>Do try some test cases where  <code>in_channels != out_channels</code> and<br>
<code>kernel_size[0] != kernel-size[1]</code> just to make sure that your not<br>
mixing up some loop ranges or indices somewhere.</p>
<aside class="quote no-group">
<blockquote>
<p>Or more basically, is my approach reasonable at all? I’m not sure if <code>autograd</code> handles the sliding window for the convolution as I would exepect.</p>
</blockquote>
</aside>
<p>I haven’t looked at your implementation in any detail, but (although, of course,<br>
inefficient) you should be able to make such an approach work.  As long as<br>
you stick to pytorch tensor operations in your <code>forward()</code>, you should get<br>
<code>autograd</code> “for free” and backpropagation through your sliding window should<br>
work properly.</p>
<p>Best.</p>
<p>K. Frank</p>

vdw: <p><a class="mention" href="/u/kfrank">@KFrank</a> , thanks so much this work great! I used <code>torch.isclose()</code> with the default parameters to see if the respective outputs and gradients are the same or similar. I tried different values for <code>in_features</code>, <code>out_features</code>, <code>kernel_size</code>, etc. and the lowest “similarity” I ever got for the 2 outputs of the forward pass was 0.998; the gradient similarities were always 1.0.</p>
<p>Only for anything other than <code>stride=1</code> I get an out-of-bounds error for my implementation; I’ve never used any other value yet. I’m sure I can fix that. Anyway, I’m already happy with that outcome.</p>
<p>Again, thanks so much!</p>