Dilated causal convolutions

username: Se_di
cooked: <p>Hi,</p>
<p>I want to replicate this dilated causal convolution:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a4257ab3f238f0576b3be9ed59ae6b6acdd8e4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8a4257ab3f238f0576b3be9ed59ae6b6acdd8e4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a4257ab3f238f0576b3be9ed59ae6b6acdd8e4.png" alt="image" data-base62-sha1="o3RTCo1kOBV63sEfMkmbh2N3BR2" width="690" height="36" data-dominant-color="E3E3E3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">765×40 8.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>m being some different categories, k being time steps and 4 the channels.</p>
<p>I defined the convolutional layer like this:<br>
<code>nn.Conv1d(in_channels=4, out_channels=8, kernel_size=(1,3), stride=1, padding=2, dilation=1)</code></p>
<p>given m=3 and k=5 it gives me the following error:</p>
<p>RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 1, 3], but got 3-dimensional input of size [3, 4, 5] instead</p>
<p>If I change the Kernel size to 1 I get an output with the wrong shape:<br>
torch.Size([3, 8, 9])</p>
<p>and if I set the kernel to 3 the output as the shape:<br>
torch.Size([3, 8, 7])</p>
<p>But it has never the output shape [3, 5, 8.]<br>
Am I missing something here?</p>

username: googlebot
cooked: <p>IIRC causal convolutions use one-sided paddings. And kernel size should be 1d (3) for conv1d.</p>

username: Se_di
cooked: <p><a class="mention" href="/u/googlebot">@googlebot</a> thanks a lot for your answer! I got the right shapes now.</p>
<p>Can you help me with this as well?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e16ba964e1c3365e01a166958055f8969134b295.png" data-download-href="https://discuss.pytorch.org/uploads/default/e16ba964e1c3365e01a166958055f8969134b295" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e16ba964e1c3365e01a166958055f8969134b295.png" alt="image" data-base62-sha1="waa5T40SJLd7vWMHZLKo8T4haaF" width="690" height="15" data-dominant-color="DFDFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">753×17 5.34 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>for padding same i would do<br>
<code> left = (kernel_size - 1) // 2</code><br>
<code>  right = (kernel_size - 1) - left</code><br>
<code>F.pad(x, (left,right)</code><br>
however im unsure about the kernel size [m x 1] again . Would it for my example just be 3?</p>

username: googlebot
cooked: <p>Yes, 3x1 is to do the same using conv2d</p>
<p>As for padding, I think it should work like:</p>
<pre><code class="lang-auto">abcd
..abcd,,     //padding
 ABCDEF      //conv output
ABCD         //causal shift
</code></pre>
<p>So you either cut EF or don’t use <code>",,"</code> padding tail.</p>

username: Se_di
cooked: <p>Let me try to repeat this.<br>
So you say for the correlational layer with padding “same”:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e63858dea55ac6e18ec1d682e30dcaa366777bb7.png" data-download-href="https://discuss.pytorch.org/uploads/default/e63858dea55ac6e18ec1d682e30dcaa366777bb7" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e63858dea55ac6e18ec1d682e30dcaa366777bb7.png" alt="image" data-base62-sha1="wQCxzb4qI9JJO0SkDqv9kj74kF9" width="690" height="17" data-dominant-color="DCDCDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">769×20 5.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>correl = nn.Conv2d(in_channels=8, out_channels=8, kernel_size=(3,1), stride=1)</p>
<p>But I really don’t understand why they write [3x1] in the paper. But then I’m supposed to use a kernel_size of 3 which is a 3x3 matrix?</p>

username: googlebot
cooked: <p>in pytorch conv1d dispatches to conv2d, adding a fake dimension, I guess in their framework something similar happens, or they have other reasons to unsqueeze input to 4d.</p>
<aside class="quote no-group" data-username="Se_di" data-post="5" data-topic="100511">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/se_di/48/14840_2.png" class="avatar"> Se_di:</div>
<blockquote>
<p>But then I’m supposed to use a kernel_size of 3 which is a 3x3 matrix?</p>
</blockquote>
</aside>
<p>I didn’t say to use 3 with conv2d</p>

username: Se_di
cooked: <blockquote>
<p>in pytorch conv1d dispatches to conv2d, adding a fake dimension, I guess in their framework something similar happens, or they have other reasons to unsqueeze input to 4d.</p>
</blockquote>
<p>Oh I understand.</p>
<blockquote>
<p>I didn’t say to use 3 with conv2d</p>
</blockquote>
<p>Sry if I did understand you wrong. But what did you mean?</p>
<p>As I understand it the [1x3] or kernel size 3 for conv1d with dilation learns the causality of each channel. However, [3x1] learns the correlation between these channels.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b42faebbbf9100b26f9cdea7a5f291563a580ae4.png" alt="image" data-base62-sha1="pI06s4Hy49NPD4LlDBH0DyeNjA8" width="343" height="211"></p>
<p>But I’m unsure how to express that [3x1] filter with conv1d.</p>

username: googlebot
cooked: <p>Kernel size is not about channels, it is a “window” size in spatial/time dimension, 1d window for 3d input, 2d for 4d input. Tensor format is (batch, channels, time), with kernel_size 3 you’re aggregating 3 timesteps, while in_channels -&gt; out_channels map is dense.</p>

username: Se_di
cooked: <p>yes thats what I tried to visualize with the [1x3] red rectangle. a window of three time steps in the time dimension.<br>
But what im trying to understand is the [3x1] window and how this is expressed with pytorch conv1d.</p>

username: googlebot
cooked: <p>For the first picture, you’d normally get 8 channels x 3 timesteps windows, unless you use channel groups (groups=8 parameter would divide channels as 8/8).</p>
<p>There is no built-in support for uneven channel groups (e.g. 3/8 channels). You can implement that with masks (F.conv1d(input, weight*mask, …)), but it is a bit cumbersome.</p>

username: Se_di
cooked: <p><a class="mention" href="/u/googlebot">@googlebot</a> thanks for you help I think I understood it now…</p>
<p>so for the conv1d with kernel size 3 it is like this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9eb5169be6250196f65abcc4df2d5af00783021b.png" alt="image" data-base62-sha1="mDZq3tzgbYC0RRtaF0x8XINBbEn" width="645" height="212"></p>
<p>the filter with size 5 moving over the time steps. and depending on how “high” the next matrix should be the more filters we use. so for 8 output channels 8 filters etc.</p>
<p>now I’m thinking of the correlation convolution. just imagine there would be 3 acel matrices of three different objects…  like in the image below. with a [mx1] kernel I want to go over each input channel of the three elements. I would pad one matrix m0 and one m4 so that I would be able to get 3 output matrices.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39d6d700a3e6ccceb1af08483564a074472ed1ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/39d6d700a3e6ccceb1af08483564a074472ed1ed" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39d6d700a3e6ccceb1af08483564a074472ed1ed_2_690x379.png" alt="image" data-base62-sha1="8fFvv3qMa3TbhuUkEsJzBZ8WrrD" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39d6d700a3e6ccceb1af08483564a074472ed1ed_2_690x379.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/9/39d6d700a3e6ccceb1af08483564a074472ed1ed_2_1035x568.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/9/39d6d700a3e6ccceb1af08483564a074472ed1ed.png 2x" data-dominant-color="FCFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1304×718 14 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>so as in the picture. input would be (3,3,5) and a possible output (3,3,5). by that having 3 filters. I’m just unsure if this would be possible.</p>
<p>I was also considering to just “.transpose(0,2)” the matrix. so that the batch_size becomes the time steps and the time steps the batch size. and then use a regular conv1d with kernel size m the previously batch size and now time steps.</p>
<p>Hope I could make it somehow understandable. Do you think this would be possible without transposing?</p>

username: googlebot
cooked: <p>Is your object set intrinsically ordered or should output be permutation invariant? Is it a variable size set?</p>

username: Se_di
cooked: <p>variable size is set, yes.</p>
<p>yes, there is an order in terms of different channel features and somehow they are depending on each other.</p>

username: googlebot
cooked: <p>Then perhaps 4d input is appropriate:<br>
(batch_size=1, channels=3 [xyz], objects=m, timesteps=k)</p>
<p>note that batch_size=m is initially incorrect if objects are not independent (unless later layers add autoregressive dependencies)</p>
<p>With above 4d input, you can perhaps do two separate causal convolutions 1x3 and 3x1. That’s not something I’ve ever done, so not sure.</p>