How to turn list of varying length tensor into a tensor

Shihan_su: <p>Hi, I am currently trying to do batch training on RNN.<br>
The first step is to pad the batch of sequence using pack_padded_sequence().<br>
But the function seems to take Variable as input, which means it need to be a tensor.</p>
<p>Currently, my input format is a list of tensors with varying length.<br>
When I try to turn that list into a tensor, it throws me the error: ‘FloatTensor’ object does not support indexing</p>
<p>It seems that I cannot create tensor with varying length on any dimension.</p>
<p>My goal is to created a batch of padded sequence from list of tensors with varying length.<br>
What may be the correct way of implementation ?</p>

smth: <p>The code here will show you how to pad your tensors of different lengths into a single tensor used by <code>pack_padded_sequence</code>:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L1336-L1344" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L1336-L1344" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/test/test_nn.py#L1336-L1344</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="1336" style="counter-reset: li-counter 1335 ;">
<li>    self._test_Conv2d_naive_groups(torch.cuda.FloatTensor)</li>
<li>
</li>
<li>def test_batchnorm_eval(self):</li>
<li>    self._test_batchnorm_eval()</li>
<li>
</li>
<li>@unittest.skipIf(not TEST_CUDA, "CUDA unavailable")</li>
<li>def test_batchnorm_eval_cuda(self):</li>
<li>    self._test_batchnorm_eval(torch.cuda.FloatTensor)</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Shihan_su: <p>If I understand correctly, that means I need to go through some procedure to pad the sequence myself.<br>
In this case, what are the purpose of pack_padded_seqence() and pad_packed_sequence() functions ?<br>
I thought they were used somehow to pad sequence with varying length automatically.</p>

ZeweiChu: <p>These two functions help you pack (already) padded sequences and pad (already) packed sequences. I think the purpose is not to help you pad sequences, but to save some spaces. Check out the following example:</p>
<blockquote>
<p>import itertools<br>
import torch<br>
import torch.nn as nn<br>
import torch.nn.utils.rnn as rnn_utils<br>
from torch.autograd import Variable, gradcheck<br>
def test_pack_padded_sequence():<br>
def pad(tensor, length):<br>
return torch.cat([tensor, tensor.new(length - tensor.size(0), *tensor.size()[1:]).zero_()])<br>
lengths = [2,1]#[10, 8, 4, 2, 2, 2, 1]<br>
max_length = lengths[0]<br>
batch_sizes = [sum(map(bool, filter(lambda x: x &gt;= i, lengths))) for i in range(1, max_length + 1)]<br>
offset = 0<br>
padded = torch.cat([pad(i * 100 + torch.range(1, 5 * l).view(l, 1, 5), max_length) for i, l in enumerate(lengths, 1)], 1)<br>
padded = Variable(padded, requires_grad=True)<br>
expected_data = [[torch.range(1, 5) + i * 100 for i in range(batch_size)] for batch_size in batch_sizes]<br>
expected_data = list(itertools.chain.from_iterable(expected_data))<br>
expected_data = torch.cat(expected_data)<br>
for batch_first in (True, False):<br>
src = padded<br>
if batch_first:<br>
src = src.transpose(0, 1)<br>
print(“SRCSRCSRC”)<br>
print(src)<br>
print(“LENGTHS”)<br>
print(lengths)<br>
packed = rnn_utils.pack_padded_sequence(src, lengths, batch_first=batch_first)<br>
print(“PACKED”)<br>
print(packed)<br>
unpacked, unpacked_len = rnn_utils.pad_packed_sequence(packed, batch_first=batch_first)<br>
print(“UNPACKED”)<br>
print(unpacked)<br>
print(“UNPACKED_LEN”)<br>
print(unpacked_len)<br>
test_pack_padded_sequence()</p>
</blockquote>

apaszke: <p>And apart from saving space, it allows to use <code>nn.LSTM</code> with variable length sequences, which can speed up your model quite a lot if you’re using bidirectional RNNs, or using the last hidden states for some purposes. We don’t have a function that takes in a list of tensors and pad then, but it sounds useful. Can you open a feature request?</p>

ecolss: <p>Any detailed examples on these 2 functions, <code>pack_padded</code> <code>pad_packed</code>? Seems confusing to me.</p>
<p>I thought, <code>pack_padded</code> would turn an already padded tensor (i.e. with each row having the same length, suppose 2d tensors here),  into another one with each row having different length (just like sparse matrix?)</p>

smth: <p><a class="mention" href="/u/ecolss">@ecolss</a> see the examples in the release notes here: <a href="https://github.com/pytorch/pytorch/releases/tag/v0.1.10">https://github.com/pytorch/pytorch/releases/tag/v0.1.10</a></p>

ecolss: <p>As illustrated in the release notes, PackedSequence holds data and a list of sequence sizes of a packed sequence batch.</p>
<p>Then why would we have to first pad the seq to equal length manually and then pass it to pack_padded func to generate this PackedSequence object? Wouldn’t it be easier to just construct the object directly from the original seq without manual padding?</p>

lgelderloos: <p><a class="mention" href="/u/ecolss">@ecolss</a> ,  <a class="mention" href="/u/apaszke">@apaszke</a> suggested making a feature request for this as it seems useful. was wondering if you followed up on it?</p>

ecolss: <p>I’m not following up on this, but I agree with you on this.</p>

nilbot: <p>just to give a consistent link for those lines, you can press ‘y’ in github and get the link.<br>
here is the updated link:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/892940b45a20d94450661a3108f4723a0f4815b1/test/test_nn.py#L1914-L1923" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/892940b45a20d94450661a3108f4723a0f4815b1/test/test_nn.py#L1914-L1923" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/892940b45a20d94450661a3108f4723a0f4815b1/test/test_nn.py#L1914-L1923</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="1914" style="counter-reset: li-counter 1913 ;">
<li>def test_pack_padded_sequence(self):</li>
<li>    def pad(tensor, length):</li>
<li>        return torch.cat([tensor, tensor.new(length - tensor.size(0), *tensor.size()[1:]).zero_()])</li>
<li>    lengths = [10, 8, 4, 2, 2, 2, 1]</li>
<li>    max_length = lengths[0]</li>
<li>    batch_sizes = [sum(map(bool, filter(lambda x: x &gt;= i, lengths))) for i in range(1, max_length + 1)]</li>
<li>    offset = 0</li>
<li>    padded = torch.cat([pad(i * 100 + torch.arange(1, 5 * l + 1).view(l, 1, 5), max_length)</li>
<li>                        for i, l in enumerate(lengths, 1)], 1)</li>
<li>    padded = Variable(padded, requires_grad=True)</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

RylanSchaeffer: <p>For anyone wondering if PyTorch has created a function that takes a list of tensors and pads them to be the same length, the following function does the trick:</p>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence</a></p>
<p>Example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; l = [torch.Tensor([1., 2.]), torch.Tensor([3.])]
&gt;&gt;&gt; torch.nn.utils.rnn.pad_sequence(l, batch_first=True, padding_value=0)
tensor([[1., 2.],
        [3., 0.]])
</code></pre>