ConvTranspose2d using unfold

username: Luesch
cooked: <p>Hello,<br>
Supose i have an matrix <code>img</code> and a kernel <code>kernel</code>.<br>
Is it possible to do a transposed convolution doing a matrix multiplication.<br>
I know that when you unroll the kernel you have to transpose this but when unrolling the input i cant<br>
figure it out.</p>
<pre><code class="lang-auto">import torch
# as an input im using a tensor with the size of a mnist digit
img = torch.randn(1 ,1 ,28 ,28)
# kernel with 1 input dim and 1 output dim
kernel = torch.randn(1 ,1 ,3 ,3)
# unfold with kernel size 3 ,stride 2 , padding 1
unfold = torch.nn.Unfold(kernel_size=(3,3) ,stride=2 ,padding=1)
# unfolded data
img_unfolded = unfold(img)

"""
so how to do the convtranspose using img_unfolded and kernel?
"""
</code></pre>
<p>thanks for looking at it and have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: wasabi
cooked: <p>If your goal is to do transpose convolution, why don’t you use nn.functional.conv_transpose2d? <a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.conv_transpose2d" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional — PyTorch 1.8.1 documentation</a></p>

username: Luesch
cooked: <p>beacuse i want to do it from scratch to really understand whats happening behind those function calls.</p>

username: santacml
cooked: <p>Hello,</p>
<p>I have also been needing to do this exact thing myself (for various reasons) for a little while and struggling to figure it out. Honestly, it’s way harder to find answers for this than it should be. I have read a lot of material on convolution, pytorch unfold, convtranspose2d, and cnn gradients. Finally, I just got it.</p>
<p>The answer is fortunately actually very simple, it’s just that it seems everyone has a different view of this operation (upsampling, cnn gradient, deconvolution, etc) that doesn’t quite explain everything. So this answer isn’t very “googleable”. Note this is the inefficient way of doing things - to do this using unfold, we have to add a bunch of padding to all the sides. There are more efficient implementations, but this is the best vectorized implementation I can come up with.</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
img = torch.randn(1 ,50 ,28 ,28)
kernel = torch.randn(30,50 ,3 ,3)
true_convt2d = F.conv_transpose2d(img, kernel.transpose(0,1))

pad0 = 3-1 # to explicitly show calculation of convtranspose2d padding
pad1 = 3-1
inp_unf = torch.nn.functional.unfold(img, (3,3), padding=(pad0,pad1))
w = torch.rot90(kernel, 2, [2,3])
# this is done the same way as forward convolution
out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
out = out_unf.view(true_convt2d.shape)
print((true_convt2d-out).abs().max())
print(true_convt2d.abs().max())
</code></pre>
<p>Running this code will give a small error value around 1e-5, but you can see the magnitude of the output is around 90 so it’s close enough. I think this is due to optimization in the backend.</p>
<p>I hope this answer becomes “googleable” for others looking for this information. I am an new user so I can apparently only put 2 links in a post. If you want more, please DM me. There is a formula for calculating padding on data science stack exchange (though it’s not too hard to figure out) if you search “how-to-calculate-the-output-shape-of-conv2d-transpose”.</p>
<p>Disclaimer: I am pretty sure this is correct, but it still could be wrong. Also, this doesn’t take into account padding, strides, dilation, or groups.</p>
<p>Sources:<br>
<a href="https://arxiv.org/pdf/1603.07285.pdf" rel="noopener nofollow ugc">Thorough descriptions of convolution</a><br>
<a href="https://medium.com/apache-mxnet/transposed-convolutions-explained-with-ms-excel-52d13030c7e8" rel="noopener nofollow ugc">Visualization that explains rotation</a></p>

username: Luesch
cooked: <p>more helpful sources provided by <a class="mention" href="/u/santacml">@santacml</a><br>
<a href="https://danieltakeshi.github.io/2019/03/09/conv-matmul/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://danieltakeshi.github.io/2019/03/09/conv-matmul/</a><br>
<a href="https://towardsdatascience.com/backpropagation-in-a-convolutional-layer-24c8d64d8509" class="onebox" target="_blank" rel="noopener nofollow ugc">https://towardsdatascience.com/backpropagation-in-a-convolutional-layer-24c8d64d8509</a></p><aside class="onebox githubfolder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/vdumoulin/conv_arithmetic" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/4aa1705fad1adb7d6acb4352579d10803323ba725ee3cbdb0d6a9206f83ffab5/vdumoulin/conv_arithmetic" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/vdumoulin/conv_arithmetic" target="_blank" rel="noopener nofollow ugc">vdumoulin/conv_arithmetic</a></h3>


  <p><span class="label1">A technical report on convolution arithmetic in the context of deep learning</span></p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Honnang
cooked: <p>You are my hero!<br>
I have spent the whole day just on this</p>

username: LucaAmbrogioni
cooked: <p>Hi all. This is a solution with upscaling. You can figure it out by “inverting” the order of the operations done in the standard conv implementation. The upscaling requires a  stride operator in the final fold.</p>
<h2><a name="transposed-convolution-1" class="anchor" href="#transposed-convolution-1"></a>Transposed convolution</h2>
<h1><a name="input-2" class="anchor" href="#input-2"></a>Input</h1>
<p>inp = torch.randn(1, 2, 10, 12)</p>
<p>w = torch.randn(2, 5, 3, 3)</p>
<p>print(“Input size (before unfold): {}”.format(inp.shape))</p>
<p>print(“Kernel size (before reshaping): {}”.format(w.shape))</p>
<p>print(“_________________________________”)</p>
<h1><a name="torch-implementation-3" class="anchor" href="#torch-implementation-3"></a>Torch implementation</h1>
<p>torch_out = nn.functional.conv_transpose2d(inp, w, stride=2, padding=1, output_padding=0, groups=1, dilation=1)</p>
<h1><a name="unfold-4" class="anchor" href="#unfold-4"></a>Unfold</h1>
<p>inp_unf = torch.nn.functional.unfold(inp, (1,1), dilation=1, stride=1).transpose(1, 2)</p>
<p>print(“Input size (after unfold): {}”.format(inp_unf.shape))</p>
<p>print(“Kernel size (after reshaping): {}”.format(w.view(w.size(0), -1).t().shape))</p>
<p>print(“_________________________________”)</p>
<h1><a name="matmul-5" class="anchor" href="#matmul-5"></a>Matmul</h1>
<p>out_unf = inp_unf.matmul(w.view(w.size(0), -1)).transpose(1, 2)</p>
<p>print(“Output size (after matmul): {}”.format(out_unf.shape))</p>
<h1><a name="fold-6" class="anchor" href="#fold-6"></a>Fold</h1>
<p>out_tr = out_unf</p>
<p>out = torch.nn.functional.fold(out_tr, (19, 23), (3, 3), dilation=1, stride=2, padding=1)</p>
<p>print(“Output size (after fold): {}, equal to the up-sampled input size but with a different number of channels”.format(out.shape))</p>
<p>print(“Error: {}”.format(torch.sum((out - torch_out))**2))</p>
<h1><a name="output-7" class="anchor" href="#output-7"></a>Output</h1>
<p>Input size (before unfold): torch.Size([1, 2, 10, 12])<br>
Kernel size (before reshaping): torch.Size([2, 5, 3, 3])</p>
<hr>
<p>Input size (after unfold): torch.Size([1, 120, 2])<br>
Kernel size (after reshaping): torch.Size([45, 2])</p>
<hr>
<p>Output size (after matmul): torch.Size([1, 45, 120])<br>
Output size (after fold): torch.Size([1, 5, 19, 23]), equal to the up-sampled input size but with a different number of channels<br>
Error: 3.0656033267462135e-14</p>