How to do padding based on lengths?

PTA: <p>I have a list of sequences and I padded it to the same length (<code>emb_len</code>). I have a separate tensor that I want to concat it to every data point in the sequences.</p>
<p>Intuitively, it is something like this</p>
<p>a b c d e f g 0 0 0<br>
u u u u u u u u u u</p>
<p>h i j k l 0 0 0 0 0<br>
u u u u u u u u u u</p>
<p>but the correct one (I suppose) would be</p>
<p>a b c d e f g 0 0 0<br>
u u u u u u u 0 0 0</p>
<p>h i j k l 0 0 0 0 0<br>
u u u u u 0 0 0 0 0</p>
<p>I did something like</p>
<pre><code class="lang-auto">torch.cat([seq_embed, 
             torch.cat([second_embed.unsqueeze(1).expand(batch_size, emb_len, second_emb_len), 
             torch.zeros([batch_size, second_embed.size(1) - emb_len, second_emb_len], dytpe=torch.long)], 1)]
            ,2)
</code></pre>
<p>However, this is not going to be working because <code>emb_len</code> is a tensor with variable numbers, which is something like <code>torch.LongTensor([1,2,3,4,5])</code> and there will be errors like <code>a Tensor with # elements cannot be converted to Scalar</code>. Thus, is there any way to solve this problem</p>

tom: <p>I think you are looking for <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.utils.rnn.pad_sequence" rel="nofollow noopener"><code>torch.nn.utils.rnn.pad_sequence</code></a>.</p>
<p>If you want to do this manually:</p>
<ul>
<li>One greatly underappreciated (to my mind) feature of PyTorch is that you can allocate a tensor of zeros (of the right type) and then copy to slices without breaking the autograd link. This is what <code>pad_sequence</code> does (the source code is linked from the “headline” in the docs). The crucial bit is:</li>
</ul>
<pre><code class="lang-auto">    out_tensor = sequences[0].data.new(*out_dims).fill_(padding_value)
    for i, tensor in enumerate(sequences):
        length = tensor.size(0)
        # use index notation to prevent duplicate references to the tensor
        if batch_first:
            out_tensor[i, :length, ...] = tensor
        else:
            out_tensor[:length, i, ...] = tensor
</code></pre>
<p>If the tensors require grad, so will out_tensor and the gradients will flow back to the tensors in the list.</p>
<ul>
<li>Another way to do this, that seems closer to your description, is to use a <code>cat</code> (or pad) in a list comprehension and push that to another <code>cat</code>.</li>
</ul>
<pre><code class="lang-auto"># setup
import torch
l = [torch.tensor([1,2,3]), torch.tensor([4,5]),torch.tensor([6,7,8,9])]
emb_len=4
# this is what you want:
lp = torch.stack([torch.cat([i, i.new_zeros(emb_len - i.size(0))], 0) for i in l],1)
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

fangyh: <p>I wrote a simple one recently after reading <code>nn.utils.rnn.pad_sequence.</code> Hope this helps.</p>
<pre><code class="lang-python">def padding_tensor(sequences):
    """
    :param sequences: list of tensors
    :return:
    """
    num = len(sequences)
    max_len = max([s.size(0) for s in sequences])
    out_dims = (num, max_len)
    out_tensor = sequences[0].data.new(*out_dims).fill_(0)
    mask = sequences[0].data.new(*out_dims).fill_(0)
    for i, tensor in enumerate(sequences):
        length = tensor.size(0)
        out_tensor[i, :length] = tensor
        mask[i, :length] = 1
    return out_tensor, mask
</code></pre>

PTA: <p>Hi Tom,</p>
<p>I do not think there is a way to use pad_packed_seq if you want to concat two tensor “vertically”. In my example, the upper line (abcdefg000) could not be combined with the lower line (u for any point, 0 for any padding). So what I have to do is to manually pad the upper sequence (abcdefg =&gt; abcdefg000) and find someway to do stack with u.</p>
<p>The second solution seems to be working. I will give it a shot today and let you know.</p>
<p>Thanks!</p>

tom: <p>I think you can pack 2d (seq len, 0/1) tensors using pad_sequence, but you would need to concatenate first. You could do your own indexing variant (by writing into 2<em>i and 2</em>i+1, I would expect that to be more efficient than many <code>cat</code>s).<br>
Another option might be to first pad the data and then get the mask (<code>padded_data &gt; 0</code>) from the joint padded tensor or so.</p>
<p>Best regards</p>
<p>Thomas</p>

PTA: <p>Just realized that the stack one is also not solving the problem. Let me rephrase the task a little bit.</p>
<p>What I have here is a batch of,</p>
<p>padding_data_1: e.g., abc000, efgh00. Everyone has the same length, padded with 0, where a, b, c, …, stand for embedding vectors.</p>
<p>data_1_len: e.g., [3,4]. The original length of data_1 before padding</p>
<p>data_2: a single embedding for each data point, e.g., [i, j], where i, j stand for embedding vectors.</p>
<p>The target is then</p>
<p>[abc000     ]          [  efgh00]<br>
[iii000         ]  and [   jjjj00]</p>
<p>(Concatenated vertically with padding)</p>

tom: <p>You can adapt the solution with copying to the large tensor from above, (I must admit I still don’t fully understand whether you want one or two tensors as a result): I think you can do</p>
<pre><code class="lang-auto">out_tensor[i, :length, ...] = single_embedding_per_point[i, None]
</code></pre>
<p>and similar to use broadcasting (with the new singleton dimension generated by the <code>None</code> index for the <code>:length</code> index on the left hand side). Use <code>single_embedding_per_point[i][None]</code> if it’s a list of tensors rather than one large tensor.</p>
<p>Best regards</p>
<p>Thomas</p>

PTA: <p>This is a working solution without taking care of some corner cases. Will this be clear on what I am doing? Do you have any suggestions on how to improve the efficiency of the code? Thanks!</p>
<pre><code class="lang-auto">torch.cat([first_embed, 
               torch.stack([
                    torch.cat([
                          _emb.expand(_s_len, 10), 
                          torch.zeros(PADDING_LEN - _s_len, 10, dtype=torch.float)]
                    , 0) 
                    for _s_len, _emb in zip(original_len, second_embed)]
               , 0)
, 2)
</code></pre>
<p>where <code>first_embed</code> is <code>padding_data_1</code>, <code>data_1_len</code> is <code>original_len</code> and <code>second_embed</code> is <code>data_2</code> in previous replies.</p>

tom: <pre><code class="lang-auto">FE_DIM = 100
first_embed = torch.randn(5,20,FE_DIM)
second_embed = torch.randn(5, 10)
original_len = [10, 8, 9, 7, 3]
PADDING_LEN = 20
def first():
    res = torch.cat([first_embed, 
               torch.stack([
                    torch.cat([
                          _emb.expand(_s_len, 10), 
                          torch.zeros(PADDING_LEN - _s_len, 10, dtype=torch.float)]
                    , 0) 
                    for _s_len, _emb in zip(original_len, second_embed)]
               , 0)
          ], 2)
    return res
def second():
    res2 = torch.zeros(5, 20, FE_DIM+10)
    res2[:,:,:FE_DIM] = first_embed
    for i, e in enumerate(second_embed):
        res2[i, :original_len[i], FE_DIM:] = e
    return res2
</code></pre>
<p>Which looks better? Probably a question of style, my style is the second.<br>
A quick</p>
<pre><code class="lang-auto">%timeit first()
%timeit second()
</code></pre>
<p>says</p>
<pre><code class="lang-auto">10000 loops, best of 3: 111 µs per loop
10000 loops, best of 3: 76 µs per loop
</code></pre>
<p>so the second is slightly faster (but it might be different for your parameters).</p>
<p>Best regards</p>
<p>Thomas</p>

pinocchio: <p>I used <code>torch.nn.utils.rnn.pad_sequence</code> for my dataloader class:</p>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (batch != 0).to(device)
    return batch, lengths, mask
</code></pre>
<hr>
<p>Many Many related posts:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773" class="inline-onebox">How to create batches of a list of varying dimension tensors?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a></li>
<li><a href="https://discuss.pytorch.org/t/using-variable-sized-input-is-padding-required/18131" class="inline-onebox">Using variable sized input - Is padding required?</a></li>
<li><a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418" class="inline-onebox">DataLoader for various length of data</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442" class="inline-onebox">How to do padding based on lengths?</a></li>
</ul>
<p>bucketing:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></li>
</ul>
<hr>
<p>Even in Stack overflows there is a question about this:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    <img alt="Trung Le" src="https://lh4.googleusercontent.com/-zBEaoloCnKo/AAAAAAAAAAI/AAAAAAAAAA4/DuvhLWns_Ak/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">How does Pytorch Dataloader handle variable size data?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, tensor, variable-length</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    Trung Le
  </a>
  on <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">10:08AM - 07 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>crossposted: <a href="https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data" rel="nofollow noopener">https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data</a></p>

kshre: <p>Here is a generalised version for a list of jagged 2-dim tensors which have all but the first dim varying in length:</p>
<pre><code class="lang-auto">def padding_tensor(sequences):
    """
    :param sequences: list of tensors
    :return:
    """
    num = len(sequences)
    max_len = max([s.shape[0] for s in sequences])
    out_dims = (num, max_len, *sequences[0].shape[1:])
    out_tensor = sequences[0].data.new(*out_dims).fill_(0)
    mask = sequences[0].data.new(*out_dims).fill_(0)
    for i, tensor in enumerate(sequences):
        length = tensor.size(0)
        out_tensor[i, :length] = tensor
        mask[i, :length] = 1
    return out_tensor, mask
</code></pre>