CUDA version of convolution

promach: <p>Could anyone explain how this <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L419-L554" rel="nofollow noopener">CUDA version of convolution</a> works ?</p>

albanD: <p>Hi,</p>
<p>It does the backward pass wrt the input. So the reverse of the forward.<br>
Given the names, it performs unpooling (I guess the forward has some pooling builtin.<br>
Then im2col that creates the Teoplitz matrix of the input (<a href="https://en.wikipedia.org/wiki/Toeplitz_matrix">https://en.wikipedia.org/wiki/Toeplitz_matrix</a>).<br>
Then a matrix matrix multiplication between this and the weights.<br>
This will give the result. (the backward of a conv is a conv. Here implenented using a Toeplitz matrix).</p>

promach: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="86168">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Then im2col that creates the Teoplitz matrix of the input (<a href="https://en.wikipedia.org/wiki/Toeplitz_matrix" rel="noopener nofollow ugc">https://en.wikipedia.org/wiki/Toeplitz_matrix </a>).<br>
Then a matrix matrix multiplication between this and the weights.</p>
</blockquote>
</aside>
<p>Wait, how does <em>im2col</em> create a toeplitz matrix of the input ?</p>

albanD: <p>You can see it here: <a href="https://stackoverflow.com/questions/16798888/2-d-convolution-as-a-matrix-matrix-multiplication">https://stackoverflow.com/questions/16798888/2-d-convolution-as-a-matrix-matrix-multiplication</a><br>
But it is moving data around in a new Tensor.</p>

promach: <aside class="quote no-group" data-username="albanD" data-post="2" data-topic="86168">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>It does the backward pass wrt the input.</p>
</blockquote>
</aside>
<p>What did you exactly mean by this sentence above ?<br>
Could you pinpoint the exact line of code ?</p>
<p>Besides, what is the purpose of <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L476-L477" rel="noopener nofollow ugc">following lines of code</a> ?</p>
<p><code>int uppool_Ho = (Hi + Kh - 1);</code></p>
<p><code>int uppool_Wo = (Wi + Kw - 1);</code></p>
<p>And how do I interpret <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L472-L473" rel="noopener nofollow ugc">int transpose_uppool_blocks = (N * Co * Ho * Wo + transpose_uppool_threads -1) / transpose_uppool_threads;</a> ?</p>

albanD: <blockquote>
<p>What did you exactly mean by this sentence above ?</p>
</blockquote>
<p>It computes the gradient of the loss wrt of the input of the convolution using the gradient of the loss wrt the output.</p>
<blockquote>
<p>Could you pinpoint the exact line of code ?</p>
</blockquote>
<p>It’s not just one line, it is the whole function.</p>
<blockquote>
<p>Besides, what is the purpose of <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L476-L477">following lines of code </a> ?</p>
</blockquote>
<p>Not sure, it computes the output sizes of the unpooling right?</p>

promach: <p>What did you exactly mean by “unpooling” ?</p>
<p>What do we have <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L460-L462" rel="nofollow noopener">size(1), size(2) and size(3)</a> ?</p>
<pre><code class="lang-auto">    int Co = grad_y.size(1);
    int Ho = grad_y.size(2);
    int Wo = grad_y.size(3);
</code></pre>

albanD: <p>I’m not sure what it means exactly in this context, but in general it’s about the reverse of a pooling op. For maxpooling for example, we have this: <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MaxUnpool1d">https://pytorch.org/docs/stable/nn.html#torch.nn.MaxUnpool1d</a></p>