Stride in conv_transpose1d

toma: <p>My expectation of output from <code>conv_transpose1d</code> does not match the actual output from pytorch. In fact, the dimension of the output does not match my expectation, which makes me wonder: what exactly does <code>stride</code> do?</p>
<p>Here’s an example:</p>
<pre><code class="lang-py">&gt;&gt;&gt; from torch import Tensor, nn
&gt;&gt;&gt; nn.functional.conv_transpose1d(Tensor([[[1,2,3,4]]]), Tensor([[[1,0,0,0]]]), padding=3, stride=2, dilation=1)
tensor([[[0., 3., 0., 4.]]])
</code></pre>
<p>With <code>padding=3</code>, I expect the input to be unchanged (i.e. no padding), and the output to be <code>4*1=4</code>. But the output is <code>[0, 3, 0, 4]</code>. Why?</p>

J_Johnson: <p>Your input is a sequence length of 4. Padding of 3 added to both sides and that length becomes 10. Your kernel size is 4. Stride is 2. So the kernel is applied to indices as follows:</p>
<ol>
<li>0, 1, 2, 3</li>
<li>2, 3, 4, 5</li>
<li>4, 5, 6, 7</li>
<li>6, 7, 8, 9</li>
</ol>
<p>Thus, 4 outputs.</p>

toma: <p>The documentation for <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose1d.html" rel="noopener nofollow ugc">convtranspose1d</a> says that the <code>padding</code> parameter is not the actual padding applied to the input. The actual padding applied to the input is <code>dilation*(kernel_size-1)-padding</code> which in this example would be <code>1*(4-1)-3=0</code>, i.e. no padding. So my expectation is that the input should be unchanged and to remain the same dimension as the kernel.</p>

J_Johnson: <p>Please see here:</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose1d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose1d.html</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/999cf5a4c4851b791f059eaa09099a210978cb13.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/999cf5a4c4851b791f059eaa09099a210978cb13" title="IMG_20230930_130105"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/999cf5a4c4851b791f059eaa09099a210978cb13_2_690x127.jpeg" alt="IMG_20230930_130105" data-base62-sha1="lUVkUp1AQ5BW651QVV6yInBxyld" width="690" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/999cf5a4c4851b791f059eaa09099a210978cb13_2_690x127.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/999cf5a4c4851b791f059eaa09099a210978cb13_2_1035x190.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/999cf5a4c4851b791f059eaa09099a210978cb13_2_1380x254.jpeg 2x" data-dominant-color="F8F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20230930_130105</span><span class="informations">1943×360 95 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

toma: <p>Yes the output dimension is consistent with the output dimension shown in the <code>Output</code> documentation, but not with the rest of the documentation. The actual padding, as I noted in the previous comment, should be zero and hence the input should not be modified and have the same dimension as the kernel (hence my expectation of a scalar output).</p>
<p>Perhaps I can rephrase my question. The documentation provides no actual equation. What exact equation is being used to compute <code>ConvTranspose1d</code> with the given parameters in my contrived example?</p>

J_Johnson: <p>I think you are confusing <code>padding</code> and <code>output_padding</code>. The part of the equation you describe is for <code>output_padding</code> (i.e. applied after the convolution). But your code set <code>padding</code> (i.e. applied before the convolution).</p>
<p>Please see my last post for the output size equation with your numbers inserted.</p>

toma: <p>I think we may talking about two different "padding"s:</p>
<ol>
<li><code>padding</code> function parameter,</li>
<li>Applied “padding” of the input that you normally have within say a convolution or dft operation</li>
</ol>
<p>The exact excerpt from the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.conv_transpose1d.html" rel="noopener nofollow ugc">documentation</a> of the referenced function regarding <code>padding</code> (the  <em>parameter</em>, and not <code>output_padding</code>) reads:</p>
<p><strong>padding</strong> – <code>dilation * (kernel_size - 1) - padding</code> zero-padding will be added to both sides of each dimension in the input. Can be a single number or a tuple <code>(padW,)</code>. Default: 0</p>
<p>To add to the confusion, <code>padding</code> (parameter) in <code>convtranspose</code> and <code>conolution</code> (pytorch) functions behaves differently. In <code>convolution</code> functions, <code>padding</code> is actual expected padding. Based on the documentation of conv<em>transpose</em> function, <code>padding</code> (the parameter) is awkwardly not <em>actually</em> padding (as you referenced in your first response). The <em>actual</em> padding is given by the equation. In my example, <code>dilation=0</code>, <code>kernel_size=4</code> and <code>padding=3</code>. So the <em>actual</em> zero “padding” size of the <em>input</em> should be (based on the documentation) <code>InputPadding = 1*(4-1)-  3  = 0</code>, i.e. no padding before convolution. <code>out_padding</code> is implicitly 0 hence no padding after convolution.</p>
<p>This means that the operated on arrays should be <code>[1,2,3,4]</code> and <code>[1,0,0,0]</code> (unchanged), hence my expectation that the output be a scalar <code>4*1 = 4</code>.</p>
<p>It seems that the documentation is completely wrong and the implementation is doing something very different. Unfortunately looking up the actual implementation of the function seems to require building aten and I haven’t gotten around to doing it yet.</p>

J_Johnson: <p>In the <code>nn.ConvTranspose1d</code> documentation, it states:</p>
<blockquote>
<p>The <code>padding</code> argument effectively adds <code>dilation * (kernel_size - 1) - padding</code> amount of zero padding to both sizes of the input. This is set so that when a <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d" rel="noopener nofollow ugc"><code>Conv1d</code></a> and a <a href="https://pytorch.org/docs/stable/generated/torch.nn.ConvTranspose1d.html#torch.nn.ConvTranspose1d" rel="noopener nofollow ugc"><code>ConvTranspose1d</code></a> are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. <strong>However, when <code>stride &gt; 1</code> , <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d" rel="noopener nofollow ugc"><code>Conv1d</code></a> maps multiple input shapes to the same output shape. <code>output_padding</code> is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side. Note that <code>output_padding</code> is only used to find output shape, but does not actually add zero-padding to output.</strong></p>
</blockquote>
<p>(emphasis mine)</p>
<p>If you set the stride to 1 while maintaining padding = 3, you get an output size of 1.</p>
<p>Please disregard my initial reply. I was confusing it with Conv1d operations. In this case, you have the reverse operation. Working in reverse, if you started with an <strong>output size of 4</strong>, and added padding of 3 on each side, you have a size of 10. Then the operation would be performed on that sequence of size 10 as:</p>
<ol>
<li>0, 1, 2, 3</li>
<li>2, 3, 4, 5</li>
<li>4, 5, 6, 7</li>
<li>6, 7, 8, 9</li>
</ol>
<p>And so you get an input size of 4, as mentioned earlier.</p>
<p>But if you set stride to 1, and padding to 3, and start with an <strong>output size of 1</strong>, add 3 padding to both sides which makes a size of 7. Then you have:</p>
<ol>
<li>0, 1, 2, 3</li>
<li>1, 2, 3, 4</li>
<li>2, 3, 4, 5</li>
<li>3, 4, 5, 6</li>
</ol>
<p>And so you get an input size of 4.</p>
<p>In this way, you can think of “padding” as the reverse operation, thus cutting off that much of the end result.</p>

toma: <p>Thanks for the response. That gave me a better understanding of how to view the “intention” of the function. I was still having issues replicating the outputs. So I dug into it more to try to “reverse engineer” the outputs with different parameters. It seems that the equation that represents the function behavior, for <code>s &gt; 1</code>, can be described as (excluding other parameters):</p>
<p>$$<br>
\tilde{X}_i := \begin{cases}<br>
X[ \lfloor i, s \rfloor ] &amp;&amp; i \mod s = 0 \<br>
0 &amp;&amp; \text{otherwise}<br>
\end{cases}<br>
$$</p>
<p>where <code>i</code> is in <code>[0, s*(N-1)+1)</code>. Then the function output becomes the non-circular discrete convolution one may expect, something along the lines of:</p>
<p>$<br>
C_{i} = \sum_j \tilde{X}_{i+j} * K^{T}_j<br>
$</p>
<p>So with with <code>s&gt;0</code>, the “padding” is not the padding one may expect in a convolution operation, but “commingling” of the zeros between elements.</p>
<p>I wish they had actually specified the equation in the documentation instead of the “verbal” description, which is insufficient, vague, and misleading.</p>

J_Johnson: <p>Here are some visuals as to what is occuring in the Pytorch implementation, including when stride &gt; 1:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md">
  <header class="source">

      <a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" target="_blank" rel="noopener nofollow ugc">vdumoulin/conv_arithmetic/blob/master/README.md</a></h4>


      <pre><code class="lang-md"># Convolution arithmetic

A technical report on convolution arithmetic in the context of deep learning.

The code and the images of this tutorial are free to use as regulated by the 
licence and subject to proper attribution:

* \[1\] Vincent Dumoulin, Francesco Visin - [A guide to convolution arithmetic
  for deep learning](https://arxiv.org/abs/1603.07285)
  ([BibTeX](https://gist.github.com/fvisin/165ca9935392fa9600a6c94664a01214))

## Convolution animations

_N.B.: Blue maps are inputs, and cyan maps are outputs._

&lt;table style="width:100%; table-layout:fixed;"&gt;
  &lt;tr&gt;
    &lt;td&gt;&lt;img width="150px" src="gif/no_padding_no_strides.gif"&gt;&lt;/td&gt;
    &lt;td&gt;&lt;img width="150px" src="gif/arbitrary_padding_no_strides.gif"&gt;&lt;/td&gt;
    &lt;td&gt;&lt;img width="150px" src="gif/same_padding_no_strides.gif"&gt;&lt;/td&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/vdumoulin/conv_arithmetic/blob/master/README.md" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>