Understanding input shape to PyTorch conv1D?

username: someAdjectiveNoun
cooked: <p>This seems to be one of the common questions on here (<a href="https://stackoverflow.com/questions/62162576/calculating-shape-of-conv1d-layer-in-pytorch" rel="nofollow noopener">1</a>, <a href="https://discuss.pytorch.org/t/convolutional-nn-for-text-input/3442">2</a>, <a href="https://stackoverflow.com/questions/56845144/dataset-shape-mismatch-conv1d-input-layer" rel="nofollow noopener">3</a>), but I am still struggling to define the right shape for input to <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv1d.html" rel="nofollow noopener">PyTorch conv1D</a>.</p>
<p>I have text sequences of length 512 (number of tokens per sequence) with each token being represented by a vector of length 768 (embedding). The batch size I am using is 6.</p>
<p>So my input tensor to conv1D is [6, 512, 768].</p>
<pre><code>input = torch.randn(6, 512, 768) 
</code></pre>
<p>Now, I want to convolve over the length of my sequence (512) with a kernel size of 2 using the conv1D layer from PyTorch.</p>
<p>I assumed that “in_channels” are the embedding dimension of the conv1D layer. If so, then a conv1D layer will be defined in this way where</p>
<p>in_channels = embedding dimension (768) <br><br>
out_channels = 100 (arbitrary number) <br><br>
kernel = 2 <br></p>
<pre><code>convolution_layer = nn.conv1D(768, 100, 2)
feature_map = convolution_layer(input)
</code></pre>
<p>But with this assumption, I get the following error:</p>
<pre><code>RuntimeError: Given groups=1, weight of size 100 768 2, expected input `[4, 512, 768]` to have 768 channels, but got 512 channels instead
</code></pre>
<p>Then I assumed that “in_channels” is the sequence length of the input sequence. If so, then a conv1D layer will be defined in this way where</p>
<p>in_channels = sequence length (512) <br><br>
out_channels = 100 (arbitrary number) <br><br>
kernel = 2 <br></p>
<pre><code>convolution_layer = nn.conv1D(512, 100, 2) 
feature_map = convolution_layer(input)
</code></pre>
<p>This works fine and I get an output feature map of dimension <code>[batch_size, 100, 767]</code>. However, I am confused. Shouldn’t the convolutional layer convolve over the sequence length of 512 and output a feature map of dimension <code>[batch_size, 100, 511]</code>?</p>
<p>I will be really grateful for your help.</p>

username: thomasjo
cooked: <p>I think the confusion here stems from the fact that PyTorch by default uses a <em>NCHW</em> memory format, with tensor dimensions structured accordingly. In other words, tensor dimensions are <code>(batch, channel, height, width)</code>.</p>
<p>Therefore, in your example, if you change the <code>input</code> tensor to the assumed structure, everything should work as expected;</p>
<pre><code class="lang-python">input = torch.randn(6, 768, 512) 
</code></pre>
<p>Note that some work has been done, which shipped in v1.5, that allows you to change the memory format to one that is (arguably) more intuitive, namely the <em>NHWC</em> format. An additional benefit of the <em>NHWC</em> format is that it can lead to drastic performance improvements in some specific scenarios due to CuDNN optimizations.</p>
<p>Some references regarding the memory format:</p>
<ul>
<li><a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html</a></li>
<li><a href="https://github.com/pytorch/pytorch/issues/28619" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/28619</a></li>
</ul>

username: someAdjectiveNoun
cooked: <p>Here is more context to it. <a href="https://stackoverflow.com/questions/62372938/understanding-input-shape-to-pytorch-conv1d" rel="nofollow noopener">https://stackoverflow.com/questions/62372938/understanding-input-shape-to-pytorch-conv1d</a></p>