Why aren't inputs to conv1d channels last?

Yash_Jakhotiya: <p>Just to replicate a historic approach (<a href="https://aclanthology.org/D14-1181.pdf" rel="noopener nofollow ugc">Kim, 2014</a>), I am experimenting with conv1d for a sentence classification task. Embedding size for each word = 300. Sequence length (i.e. number of words in a sentence) =  512. So my input sentence is [512, 300]. (Let us omit batch size for this discussion). This way I can refer to each word in my input sentence as input[word_index].  A conv layer needs to convolve over sequence length (say, to get bi-grams or tri-grams). This means your embedding needs to be seen as ‘in_channels’.</p>
<p>Now, after digging around SO answers, it turns out, conv1d expects my input to have channels first, i.e. a sentence needs to be [300, 512].</p>
<p>This breaks the row-major convention which are historic defaults by C, NumPy, etc.</p>
<p>For output, as a diagram shows in Kim’s paper, one might expect sequence length to be the first index and channels (corresponding to your multiple filters) to be the second one. nn.conv1d gives output in [out_channels, sequence_length_with_padding_stride_calculation] format.</p>
<p>Why does PyTorch favor channels first?<br>
If it’s a matter of convention, shouldn’t there be an easy way to choose your convention?<br>
If it’s because CuDNN prefers NCHW, can PyTorch not internally change input?</p>
<p>The to and contiguous approach outlined <a href="https://pytorch.org/tutorials/intermediate/memory_format_tutorial.html" rel="noopener nofollow ugc">here</a> is convoluted.  Why would you want devs to explicitly convert their input and their models? Can we not have a parameter in model declaration that decides channels_first or channels_last and the input is processed accordingly? If you want to maintain backwards compatibility, the parameter might have channels_first as default.</p>
<p>P.S. the conv1d documentation could greatly benefit if there is an explanation of this below the example, and possibly, if your input is channels_last, how to convert it and have contiguous memory allocation. If this needs to be added only in conv1d, I can raise a PR. If this needs explanation across multiple pages, a longer discussion might be helpful.</p>

ptrblck: <aside class="quote no-group" data-username="Yash_Jakhotiya" data-post="1" data-topic="132429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash_jakhotiya/48/42355_2.png" class="avatar"> Yash_Jakhotiya:</div>
<blockquote>
<p>Why does PyTorch favor channels first?</p>
</blockquote>
</aside>
<p>For historic performance reasons, if I’m not mistaken.</p>
<aside class="quote no-group" data-username="Yash_Jakhotiya" data-post="1" data-topic="132429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash_jakhotiya/48/42355_2.png" class="avatar"> Yash_Jakhotiya:</div>
<blockquote>
<p>If it’s a matter of convention, shouldn’t there be an easy way to choose your convention?<br>
If it’s because CuDNN prefers NCHW, can PyTorch not internally change input?</p>
</blockquote>
</aside>
<p>PyTorch can change the memory layout via <code>to(memory_format=torch.channels_last)</code>, which is the preferred format for mixed-precision convolutions using TensorCores.</p>
<aside class="quote no-group" data-username="Yash_Jakhotiya" data-post="1" data-topic="132429">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yash_jakhotiya/48/42355_2.png" class="avatar"> Yash_Jakhotiya:</div>
<blockquote>
<p>Why would you want devs to explicitly convert their input and their models? Can we not have a parameter in model declaration that decides channels_first or channels_last and the input is processed accordingly?</p>
</blockquote>
</aside>
<p>Wouldn’t this already be the case or are you concerned about having to add the <code>to()</code> operation to the input as well? Under the hood, the parameter layout might be respected and the inputs thus transformed for you, but explicit transformations are preferred over implicit ones.</p>

Yash_Jakhotiya: <p>I am inclined towards a design where something like,</p>
<pre><code class="lang-auto">input = torch.randn(N, L, C)

conv_layer = torch.nn.Conv1d(in_channels=C, out_channels=C, kernel_size=3, memory_format='channels_last')

output = conv_layer(input)
</code></pre>
<p>works out of the box.</p>
<p>Is this possible in the current setup? And if not, what would be the minimal changes required here to make it work?</p>

peterukk: <p>Is channels-last still not supported in conv1D in Pytorch 2? I’m implementing a custom architecture doing convolutions inside an RNN and the model is quite slow. I’m suspecting the memory use could be one problem</p>