Expand dense kernel into a dilated one with zero holes

jlitalien: <p>I’m trying to expand a dense tensor into a dilated one with zero weights to fill the new holes. Suppose I have a <code>[N,K,K]</code> tensor, where <code>N</code> is the number of features and <code>K</code> is the kernel size. A dilated kernel with dilation <code>D</code> will have effective kernel size <code>L=K*(K-1)*(D-1)</code>. I want to map <code>[N,K,K] -&gt; [N,L,L]</code>, which is essentially blowing up the kernel along the first dimension to create the dilated pattern, while setting the newly inserted entries to zero.</p>
<p>Is there an easy way to perform such an operation in PyTorch? Right now I’m manually figuring out all non-zero dilation indices by looping over the effective kernel window, which is quite inefficient. I’m thinking there’s a way to do this with a smart combination of expand/reshape, but I’m not seeing it.</p>

kfxw: <p>Hi,</p>
<p>I think you can construct a transformation matrix, and use a matrix multiplication to implement it. I can show you an example of how to insert 0 in rows and columns. Suppose your K=2, weights are ‘A, B, C, D’, the transformations are like (hopefully you can get my point)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d5cd293fb04770e58eb4e743c7589a68a7fbe20b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d5cd293fb04770e58eb4e743c7589a68a7fbe20b" title="c5f4a304da01aae02937c690a562465"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5cd293fb04770e58eb4e743c7589a68a7fbe20b_2_345x231.jpeg" alt="c5f4a304da01aae02937c690a562465" data-base62-sha1="uvnfVvlphT7Iexg9lEiZcDtMSlJ" width="345" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5cd293fb04770e58eb4e743c7589a68a7fbe20b_2_345x231.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5cd293fb04770e58eb4e743c7589a68a7fbe20b_2_517x346.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d5cd293fb04770e58eb4e743c7589a68a7fbe20b_2_690x462.jpeg 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">c5f4a304da01aae02937c690a562465</span><span class="informations">1608×1080 62.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And you can use the combination of them for your task.</p>
<p>Correct me if I’m wrong <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

jlitalien: <p>Awesome. I worked it out and it ended up being the matrix you get from keeping every other <code>D</code> rows of an identity matrix of size <code>L</code>. Here’s an implementation and visualization.</p>
<pre><code class="lang-auto">import torch
import matplotlib.pyplot as plt

kernel_size = 7
dilation = 3

x = torch.rand(kernel_size, kernel_size)
eff_kernel_size = kernel_size + (kernel_size-1)*(dilation-1)

h = torch.eye(eff_kernel_size)[::dilation]
d = torch.mm(torch.mm(x.t(), h).t(), h)

fig = plt.figure()
ax1 = fig.add_subplot(1,2,1)
ax1.matshow(x.numpy(), interpolation='nearest')
ax1.title.set_text('Dense')
ax2 = fig.add_subplot(1,2,2)
ax2.matshow(d.numpy(), interpolation='nearest')
ax2.title.set_text('Dilated')
plt.show()

</code></pre>
<p><strong>EDIT:</strong> Since PyTorch 1.9, there is <code>torch.kron()</code> which performs the Kronecker product. So you can get a pretty good speed-up by simply running:</p>
<pre><code class="lang-auto">h = torch.zeros(dilation, dilation)
h[0,0] = 1
d = torch.kron(x, h)[:eff_kernel_size, :eff_kernel_size]
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8ea49623d6a2f674eac63be45870eb283c0e992.png" data-download-href="https://discuss.pytorch.org/uploads/default/f8ea49623d6a2f674eac63be45870eb283c0e992" title="dilation"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/8/f8ea49623d6a2f674eac63be45870eb283c0e992.png" alt="dilation" data-base62-sha1="zw0mUpQ0C5nAM1h3r8nPKWGcABc" width="690" height="407" data-dominant-color="A8A6B1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dilation</span><span class="informations">910×538 10.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>