What is the meaning of trailing dimensions?

username: pinocchio
cooked: <p>I was reading the <a href="https://pytorch.org/docs/stable/nn.html#pack-padded-sequence" rel="noopener nofollow ugc">documentation</a> and I came across the following sentence:</p>
<blockquote>
<ul>
<li>is any number of trailing dimensions, including none.</li>
</ul>
</blockquote>
<p>what is the meaning of “trailing dimensions”? May I also get an example to clarify the point?</p>
<p>for context:</p>
<blockquote>
<p>Packs a Tensor containing padded sequences of variable length.</p>
<p><code>input</code>  can be of size  <code>T x B x *</code>  where T is the length of the longest sequence (equal to  <code>lengths[0]</code> ),  <code>B</code>  is the batch size, and  <code>*</code>  is any number of dimensions (including 0). If  <code>batch_first</code>  is  <code>True</code> ,  <code>B x T x *</code>   <code>input</code>  is expected.</p>
<p>For unsorted sequences, use enforce_sorted = False. If  <code>enforce_sorted</code>  is  <code>True</code> , the sequences should be sorted by length in a decreasing order, i.e.  <code>input[:,0]</code>  should be the longest sequence, and  <code>input[:,B-1]</code>  the shortest one. enforce_sorted = True is only necessary for ONNX export.</p>
</blockquote>
<hr>
<p>I actually meant to ask about <code>pad_sequence</code> <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence" class="inline-onebox" rel="noopener nofollow ugc">torch.nn — PyTorch 2.1 documentation</a></p>

username: tom
cooked: <p>If you have an input of shape <code>(T, B)</code>, you will get a packed sequence with shape <code>(sum of lengths,)</code> as data.<br>
If you have an input of shape <code>(T, B, F_1)</code>, you will get a packed sequence with shape <code>(sum of lengths, F_1)</code> as data.<br>
If you have an input of shape <code>(T, B, F_1, F_2)</code>, you will get a packed sequence with shape <code>(sum of lengths, F_1, F_2)</code> as data.<br>
…<br>
So <code>*</code> is a wildcard (similar to its use in file glob masks) representing any possible value of <code>input.shape[2:]</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: pinocchio
cooked: <p>what is sum of lengths? is it <code>T*B</code>? Who’s lengths?</p>

username: pinocchio
cooked: <p>also my question is in the context of padding not packing…so idk why packing is relevant…</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#pad-sequence" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.html#pad-sequence</a></p>
<blockquote>
<p>pad_sequence</p>
<p>torch.nn.utils.rnn.pad_sequence(sequences, batch_first=False, padding_value=0)[SOURCE]<br>
Pad a list of variable length Tensors with padding_value</p>
<p>pad_sequence stacks a list of Tensors along a new dimension, and pads them to equal length. For example, if the input is list of sequences with size L x * and if batch_first is False, and T x B x * otherwise.</p>
<p>B is batch size. It is equal to the number of elements in sequences. T is length of the longest sequence. L is length of the sequence. * is any number of trailing dimensions, including none.</p>
</blockquote>

username: tom
cooked: <p>Yeah, well, you linked <code>pack_padded_sequence</code> before.<br>
There it is the same, except that the lengths (which is an input to <code>pack_padded_sequence</code>) are not given as a parameter but are implicit in <code>[len(s) for s in sequences]</code> the sum of lengths is then the sum of the <code>lengths</code> parameter (<code>sum(lengths)</code>) or <code>sum([len(s) for s in sequences])</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Paulo_Mann
cooked: <p>AFAIK this means that the resulting tensor can have <em>an arbitrary number of dimensions with any shape</em> after T and B dimensions.</p>
<p>For example, it could be a tensor of sizes:</p>
<ol>
<li><code>torch.size([T, B])</code></li>
<li><code>torch.size([T, B, 10]</code></li>
<li><code>torch.size([T, B, 25, 500])</code></li>
<li>and so on…</li>
</ol>

username: pinocchio
cooked: <p>a real runnable example would be useful…</p>

username: tom
cooked: <p>From the documentation:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch.nn.utils.rnn import pad_sequence
&gt;&gt;&gt; a = torch.ones(25, 300)
&gt;&gt;&gt; b = torch.ones(22, 300)
&gt;&gt;&gt; c = torch.ones(15, 300)
&gt;&gt;&gt; pad_sequence([a, b, c]).size()
torch.Size([25, 3, 300])
</code></pre>
<p>Can be mutated to</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch.nn.utils.rnn import pad_sequence
&gt;&gt;&gt; a = torch.ones(25, 300, 10)
&gt;&gt;&gt; b = torch.ones(22, 300, 10)
&gt;&gt;&gt; c = torch.ones(15, 300, 10)
&gt;&gt;&gt; pad_sequence([a, b, c]).size()
torch.Size([25, 3, 300, 10])
</code></pre>
<p>to</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch.nn.utils.rnn import pad_sequence
&gt;&gt;&gt; a = torch.ones(25, 300, 10, 5)
&gt;&gt;&gt; b = torch.ones(22, 300, 10, 5)
&gt;&gt;&gt; c = torch.ones(15, 300, 10, 5)
&gt;&gt;&gt; pad_sequence([a, b, c]).size()
torch.Size([25, 3, 300, 10, 5])
</code></pre>
<p>to</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch.nn.utils.rnn import pad_sequence
&gt;&gt;&gt; a = torch.ones(25, 300, 10, 5, 7)
&gt;&gt;&gt; b = torch.ones(22, 300, 10, 5, 7)
&gt;&gt;&gt; c = torch.ones(15, 300, 10, 5, 7)
&gt;&gt;&gt; pad_sequence([a, b, c]).size()
torch.Size([25, 3, 300, 10, 5, 7])
</code></pre>
<p>I think you get the rough idea? <code>*</code> replaces <code>(300,)</code> or <code>(300, 10)</code> or <code>(300, 10, 5)</code> or <code>(300, 10, 5, 7)</code>.</p>
<p>Best regards</p>
<p>Thomas</p>