Custom Convolution Dot Product

lucamocerino: <p>Hi,<br>
my question is quite trivial. Is it possible to change the conv2d Functional ?<br>
I need a very deep but small change in the dot product in convolution is it possible ?<br>
Thanks every one.</p>

jpeg729: <p>Well you could write a custom autograd.Function as explained <a href="http://pytorch.org/docs/0.3.1/notes/extending.html">here</a>. That way you could code your custom conv2d using numpy operations, but you would have to code the backward pass explicitly.</p>
<p>Or you could explain what you want in a little more detail and maybe someone can show you how to accomplish it with standard tensor operations and a little ingenuity.</p>

lucamocerino: <p>Thank for the suggestion.<br>
But I need to replace the inner matrix multiplication in conv (and also in fc) with a different non math operation. I need to replace the elementary multiplication (i.e. in XNOR net substitute a multiplication with a bitwise xnor ) (stupid example, i know).</p>

Shen: <p>Hi, did you solve this problem? I’m facing a similar problem right now…</p>

lucamocerino: <p>The only solution is, as previously suggested, to build a custom functional and then a module. The real issue is the speed. A fully python implementation of a custom convolution is really slower. In order to have a reasonable speed you have to modify the C backend used in Pytorch. This kind of hack is <strong>not</strong> trivial at all !!!<br>
Good luck</p>

SimonW: <p><a class="mention" href="/u/shen">@Shen</a> <a class="mention" href="/u/lucamocerino">@lucamocerino</a> If we are talking about images, another approach is to unfold the tensor into columns (with <code>nn.Unfold</code>), do whatever your operation is, then <code>view</code> it as the output shape.</p>
<p>Unfold (i.e., im2col) + gemm (i.e., batched matrix multiplication) + view is actually a common and pretty efficient implementation of convolution.</p>

hughperkins: <p>Example of how to use unfold here:</p>
<pre><code>"""
test using torch unfold to do a convolution

we'll do a convolution both using standard conv, and unfolding it and matrix mul,
and try to get the same answer
"""
import torch
from torch import nn, optim
import torch.nn.functional as F

def run():
    in_channels = 2
    out_channels = 5
    size = 4
    torch.manual_seed(123)
    X = torch.rand(1, in_channels, size, size)
    conv = nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=3, padding=1, bias=False)
    out = conv(X)
    print('out', out)
    print('out.size()', out.size())
    print('')

    Xunfold = F.unfold(X, kernel_size=3, padding=1)
    print('X.size()', X.size())
    print('Xunfold.size()', Xunfold.size())

    kernels_flat = conv.weight.data.view(out_channels, -1)
    print('kernels_flat.size()', kernels_flat.size())

    res = kernels_flat @ Xunfold
    res = res.view(1, out_channels, size, size)
    print('res', res)
    print('res.size()', res.size())


run()</code></pre>

hughperkins: <p>(also a blog post on how this works: <a href="https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/" rel="nofollow noopener">https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/</a> )</p>

lucamocerino: <p><a class="mention" href="/u/hughperkins">@hughperkins</a> Is it possible the same thing with Pytorch 0.3 ? (i cannot upgrade the version)<br>
If yes can u show me an example. Thank you.</p>

hughperkins: <p>I think unfold is new in 0.4.</p>
<p>The actual unfold function itself is present in torch for years. So, you could use FFI to call it. It’s not that hard to use FFI from pytorch AFAIK, but it’s not something I can describe in a few lines, partly because I’d have to go away and google, search for pytorch forums posts on it. I’ve done FFI from python before though, and it’s <em>fairly</em> painless, if you have a few days to kill.</p>
<p>On the whole, if upgrading to 0.4 will take less than ~few hours effort, I’d go for the 0.4 solution probably.</p>
<p>(note that ‘unfold’ used to be called ‘im2col’ as Simon alludes to)</p>

Shen: <p>Your example and link help me a lot!! Thanks! I’m quite new to this field, and still have one question about the <em><strong>bias</strong></em> part. From what I saw, <strong><em>bias</em></strong> term is a vector (i.e. 1D tensor) with the size of output channels. How to add this <strong><em>bias</em></strong> term into the unfold version of the convolution?</p>

hughperkins: <p>Create a 1d tensor, with <code>requires_grad=True</code>. Use broadcasting to add it to the result of the matrix multiplication.</p>

Shen: <p>I tried this way before, but I cannot get the broadcasting work.</p>
<p>Say my multiplication result has the size [32, 6, 28, 28], and the bias term is a 1D tensor with the size 6. Broadcasting these two together will result in an error saying<br>
<strong>"The expanded size of the tensor (28) must match the existing size (6) at non-singleton dimension 3"</strong>.<br>
Is there anything wrong with my implementation?</p>

hughperkins: <p>you need to unsqueeze the last dimension of your tensor twice. concretely, your bias tensor needs to have the dimensions <code>[6, 1, 1]</code></p>

Shen: <p>Got it! Thanks for your help!</p>

Tycho_van_der_Oudera: <p>I am interested in the other way around: unfolding the kernel weight matrix (instead of X) and then multiplying it with a flattened X to get the same result. But, I have quite some difficulties getting the dimensions right.</p>
<p>Would this be possible?</p>
<p>(I understand that the matrices will become a lot larger this way)</p>

hughperkins: <p><a class="mention" href="/u/tycho_van_der_oudera">@Tycho_van_der_Oudera</a> Unclear to me what you are asking. The GEMM way of doing convolution involves ‘flattening out’ both the spatial input tensor and the spatial kernels. However, for the kernels, there’s no need to do any ‘unrolling’ as such, a pytorch <code>.view</code> is sufficient. (You can check my code above for an example).</p>

himat: <p>Why do you have unfold the input, but can just <code>.view</code> the kernel?</p>

Shangyin_Gao: <p>if you want to use matrix multiplication to calculate convolution, the input matrix needs to be ‘duplicated’. But the kernel not.</p>
<p><code>unfold</code> do the duplication</p>

Sami_Hassan: <p>Thanks for your code,</p>
<p>what is the @ part res = kernels_flat @ Xunfold</p>
<p>Can i replace the multiplication and addition with my own mymult(num1,num2) and my add(num1,num2) with this operation ?</p>