Conv1D kernel size explained

syomantak: <p>In the doc for <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv1d.html" rel="noopener nofollow ugc">Conv1D</a>, kernel size is described as</p>
<blockquote>
<p><strong>kernel_size</strong> ([ <em>int</em> ] <em>or</em> [ <em>tuple</em> ])</p>
</blockquote>
<p>Can someone explain how kernel size being tuple makes sense? It made sense in Conv2D as the kernel is 2 dimensional (height and width).</p>

Nikronic: <p>Hi,</p>
<p>One difference I can mention is that you cannot pas 3D tensor to Conv2d. For instance for sound signals in shape of <code>[batch, channels, timestap]</code>, conv2d does not work and the only choice is conv1d. But you use 2d kernel size (a tuple) for conv1d, it will act in the same way conv2d does. For instance, when you use a tuple for kernel size in conv1d, it forces you to use a 4D tensor as the input. Here is an example that produces same values:</p>
<pre><code class="lang-auto">torch.manual_seed(0)
x = torch.ones((1,1, 2, 2))
c = nn.Conv1d(1, 1, (1, 1))
c.weight = nn.Parameter(torch.tensor([[[[0.5]]]]))
c.bias = nn.Parameter(torch.tensor([0.]))
c(x) 
 
###  output
tensor([[[[0.5000, 0.5000],
          [0.5000, 0.5000]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
###

---
cc = nn.Conv2d(1, 1, (1, 1))
cc.weight = nn.Parameter(torch.tensor([[[[0.5]]]]))
cc.bias = nn.Parameter(torch.tensor([0.]))
cc(x)

###
tensor([[[[0.5000, 0.5000],
          [0.5000, 0.5000]]]], grad_fn=&lt;MkldnnConvolutionBackward&gt;)
###
</code></pre>
<p>Actually, I could not find any information that why is that (so far!), but I think based on this definition of <a href="https://github.com/pytorch/pytorch/blob/e35199a691c4f42bb31217c59284d0702d1a5efd/torch/nn/modules/conv.py#L15" rel="nofollow noopener">_ConvNd</a>, torch will treat different input tensors regarding their input dimensons. So, if you pass 3D input, it will call Conv1d, if pass 4D, will call Conv2d and so on.<br>
In the line below, it expands weight parameter depending on input size and weight size is determined by kernel. I think this s the reason.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/e35199a691c4f42bb31217c59284d0702d1a5efd/torch/nn/modules/conv.py#L54" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/e35199a691c4f42bb31217c59284d0702d1a5efd/torch/nn/modules/conv.py#L54" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/e35199a691c4f42bb31217c59284d0702d1a5efd/torch/nn/modules/conv.py#L54</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="44" style="counter-reset: li-counter 43 ;">
<li>    # `_reversed_padding_repeated_twice` is the padding to be passed to</li>
<li>    # `F.pad` if needed (e.g., for non-zero padding types that are</li>
<li>    # implemented as two ops: padding + conv). `F.pad` accepts paddings in</li>
<li>    # reverse order than the dimension.</li>
<li>    self._reversed_padding_repeated_twice = _reverse_repeat_tuple(self.padding, 2)</li>
<li>    if transposed:</li>
<li>        self.weight = Parameter(torch.Tensor(</li>
<li>            in_channels, out_channels // groups, *kernel_size))</li>
<li>    else:</li>
<li>        self.weight = Parameter(torch.Tensor(</li>
<li class="selected">            out_channels, in_channels // groups, *kernel_size))</li>
<li>    if bias:</li>
<li>        self.bias = Parameter(torch.Tensor(out_channels))</li>
<li>    else:</li>
<li>        self.register_parameter('bias', None)</li>
<li>    self.reset_parameters()</li>
<li>
</li><li>def reset_parameters(self):</li>
<li>    init.kaiming_uniform_(self.weight, a=math.sqrt(5))</li>
<li>    if self.bias is not None:</li>
<li>        fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Bests</p>

syomantak: <p>I am still a little confused. So Conv1D with 2D kernel is essentially Conv2D?</p>

Nikronic: <p>I think so, I ran few experiments (as I could not interpret more from source code) and it seems I was correct about <code>_ConvNd</code> idea. Here is some images of my experiments, although there might be wrong or not adequate but based on source code in previous post, I think it is true.</p>
<p>Note that because of power limitations, I have used same config for both Conv1d and Conv2d and only changed input size to have bigger dimensions which you can as the title of each graph. Although, in case of cuda, I increased batch size to get more reliable values.</p>
<p>Results on CPU:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d06f0f0a3edcaac32b49ee99990f40e1f998efe.png" data-download-href="https://discuss.pytorch.org/uploads/default/3d06f0f0a3edcaac32b49ee99990f40e1f998efe" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d06f0f0a3edcaac32b49ee99990f40e1f998efe_2_690x455.png" alt="1" data-base62-sha1="8HRZOOmm1IoVP5UCKE0XOTonbwG" width="690" height="455" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/d/3d06f0f0a3edcaac32b49ee99990f40e1f998efe_2_690x455.png, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d06f0f0a3edcaac32b49ee99990f40e1f998efe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d06f0f0a3edcaac32b49ee99990f40e1f998efe.png 2x" data-dominant-color="F2EEEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">893×590 135 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h3><a name="h-1" class="anchor" href="#h-1"></a></h3>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0524ec69035c62976a918b6ffb03debf1c4af6e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a0524ec69035c62976a918b6ffb03debf1c4af6e" title="2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0524ec69035c62976a918b6ffb03debf1c4af6e_2_690x458.png" alt="2" data-base62-sha1="mSgJNJ85K1z7GuB3E4BdS4eCv38" width="690" height="458" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a0524ec69035c62976a918b6ffb03debf1c4af6e_2_690x458.png, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0524ec69035c62976a918b6ffb03debf1c4af6e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/0/a0524ec69035c62976a918b6ffb03debf1c4af6e.png 2x" data-dominant-color="F7F5F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2</span><span class="informations">887×590 98.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>###############<br>
Results on GPU:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9ee6c2e65d20bac18f899473c17fd2f81286753.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9ee6c2e65d20bac18f899473c17fd2f81286753" title="1cuda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9ee6c2e65d20bac18f899473c17fd2f81286753_2_690x465.png" alt="1cuda" data-base62-sha1="v5UqBwgfm2mjCV1xhYZvqJz14vV" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9ee6c2e65d20bac18f899473c17fd2f81286753_2_690x465.png, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9ee6c2e65d20bac18f899473c17fd2f81286753.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9ee6c2e65d20bac18f899473c17fd2f81286753.png 2x" data-dominant-color="FCFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1cuda</span><span class="informations">874×590 26.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h3><a name="h-2" class="anchor" href="#h-2"></a></h3>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c3a726bbe365a3305a88911673ef2901108c4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/46c3a726bbe365a3305a88911673ef2901108c4e" title="2cuda"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46c3a726bbe365a3305a88911673ef2901108c4e_2_690x465.png" alt="2cuda" data-base62-sha1="a60BET2ga7wJLy3ud0seHTAtD66" width="690" height="465" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/46c3a726bbe365a3305a88911673ef2901108c4e_2_690x465.png, https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c3a726bbe365a3305a88911673ef2901108c4e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c3a726bbe365a3305a88911673ef2901108c4e.png 2x" data-dominant-color="FCFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2cuda</span><span class="informations">874×590 22.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What I found is that in small tensors, Conv2d works faster but in bigger tensors they both do the same but Conv2d is always a little bit faster.</p>
<p>I hope it helps</p>

Alexander_Skusnov: <p>Your code doesn’t run:</p>
<pre><code class="lang-auto">torch.manual_seed(0)
x = torch.ones((1,1, 2, 2))
print(x)
c = nn.Conv1d(1, 1, (1, 1))
c.weight = nn.Parameter(torch.tensor([[[[0.5]]]]))
c.bias = nn.Parameter(torch. Tensor([0.]))
c(x)
###
tensor([[[[1., 1.],
          [1., 1.]]]])
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_5860\1401148504.py in &lt;module&gt;
      5 c.weight = nn.Parameter(torch.tensor([[[[0.5]]]]))
      6 c.bias = nn.Parameter(torch.tensor([0.]))
----&gt; 7 c(x)

d:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1194             return forward_call(*input, **kwargs)
   1195         # Do not call functions when jit is used
   1196         full_backward_hooks, non_full_backward_hooks = [], []

d:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self, input)
    311 
    312     def forward(self, input: Tensor) -&gt; Tensor:
--&gt; 313         return self._conv_forward(input, self.weight, self.bias)
    314 
    315 

d:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in _conv_forward(self, input, weight, bias)
    307                             weight, bias, self.stride,
    308                             _single(0), self.dilation, self.groups)
--&gt; 309         return F.conv1d(input, weight, bias, self.stride,
    310                         self.padding, self.dilation, self.groups)
    311 

RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 1, 2, 2]
</code></pre>

lizishuai: <p>pytorch  version 1.7.1 is ok，2.0version make mistakes.</p>