How to create batches of a list of varying dimension tensors?

fbrah: <p>I am trying to create batches for my training. my inputs are tensors with varying dimension.</p>
<p>Let’s say I have a list of tensors for source (input) and target (output).<br>
tensors in the source are of varying dimension. For example: torch.Size([1, 100]), torch.Size([2, 100]), torch.Size([3, 100]), torch.Size([4, 100])<br>
but tensors in target are all torch.Size([1, 100])</p>
<p>I want to have batches with the same dimensions. like all the [1,100] be batched together. all [2,100] batched together. and consequently I want to have the corresponding target tensors paired with source tensors.</p>
<p>Can anyone help me or point me to the right direction? Many thanks</p>
<p>Note: The thing is dim(0) which is  varying here can be considered similar to sequence length when we do normal language modeling or seq2seq.</p>

Prerna_Dhareshwar: <p>Hi,<br>
Usually with different sequence length you can pad all inputs to become the same length. After padding a sequence, if you are using an torch.nn RNN block such as LSTM() or GRU(), you can use <code>pack_padded_sequence</code> to feed in a padded input.</p>
<p>Otherwise you could create batches according to the length of the 0th dimension, like you said, but that might be inefficient. And using <code>torch.utils.data.Dataset</code> and <code>torch.utils.data.DataLoader</code> won’t be possible.</p>

fbrah: <p>Thanks <a class="mention" href="/u/prerna_dhareshwar">@Prerna_Dhareshwar</a>.<br>
But my only concern here is that this tensors are actually vector representations ( 100 dim vectors) . They are not real sequences.<br>
In that case, for example if I want to pad the first input which is  of dimension torch.Size([1, 100]) , the padded version would be torch.Size([5, 100]) in which all element [1:,:] would be the ‘padding’ index?</p>
<p>Doesn’t it make problem during training?</p>

Prerna_Dhareshwar: <p>Why do you think it would be a problem during training?</p>

fbrah: <p>Not exactly sure but maybe because most of the time dim(1) will be padded to make them equal size.<br>
However here, dim(1) of my tensors are actually model dimension.<br>
But I think you are right, I am going to try that and see how that works.<br>
Just to double check:<br>
when using padding then all the batches in my case would be of size [5,100] ? right?<br>
and if we want to make it efficient by batching similar size tensors together, we have to write our customize Iterator and then <code>torch.utils.data.Dataset</code>  and  <code>torch.utils.data.DataLoader</code> are not usable?</p>

Prerna_Dhareshwar: <p>Hi,</p>
<p>It shouldn’t be an issue even if you’re padding sequences of size 1.<br>
Yes, after padding, all your sequences will have same length. Make sure you read the documentation for <code>pack_padded_sequence</code> to understand how exactly it works.</p>
<p>Yes you are correct, since DataLoader() will create the batches and it will be hard to control the training examples that go in a batch.</p>

fbrah: <p>I think I know what’s the problem:<br>
Since this tensors are actually real float values, if I want to pad them with ‘0’, ‘0’ might have a meaning in vector space.<br>
It’s not similar to case where everything is token indices and we have a padding index and the embedding will transfer it into vector space.<br>
These tensors are actually in vector space and I am not sure what padding value should I use.</p>

Prerna_Dhareshwar: <p>So it’s okay to just pad with 0’s because when you create a <code>pack_padded_sequence</code>, apart from the padded input, another input you provide is the actual length of the sequence. So whatever you pad with after this is not considered in the output of the RNN. For eg: If your input has size <code>x.shape = (1,1,100)</code> which is (sequence, batch, input dimension), and you pad it such that <code>x_padded.shape = (5,1,100)</code>, then you create a <code>pack_padded_sequence</code> as follows-</p>
<pre><code class="lang-auto">x_packed = pack_padded_sequence(x_padded, [1])
</code></pre>
<p>The second argument <code>[1]</code> indicates that the length of the sequence x is only 1. So it doesn’t consider the values after the first value. Does this make sense?</p>

fbrah: <p>Oh I see, that totally make sense now, Thanks a lot.</p>
<p>I am sorry, I just got confused with the steps,<br>
I should pad the sequences, then use DataLoader()  to create batches and then pack_padded_sequence to feed in inputs?</p>
<p>and can I use pack_padded_sequence for any deep learning networks? like Transformer?</p>

Prerna_Dhareshwar: <p>Yes, you’re right, you pad and use pack_padded_sequence to feed in inputs.</p>
<p>You can use it wherever it specifies in the docs that it accepts inputs of that format. I’m not sure what all that includes.</p>

fbrah: <p>It seems like PackedSequence Object can only be use by <code>nn.RNN</code> ,  <code>nn.LSTM</code> ,  <code>nn.GRU</code>.<br>
How we can creates batches to use them in our own defined function/classes which in my case is mostly Attentions (dot product)</p>
<p>I would appreciate any help.</p>

Prerna_Dhareshwar: <p>I guess that is because only in RNNs does the fact that the data is a sequence matter - i.e., memory is retained from earlier parts of the sequence even at the end of the sequence. Do you need to retain the data in this ‘sequence’ format for your non-RNN application? What exactly are you trying to do?</p>

fbrah: <p>I want to use these as input to a Transformer network. Essentially I want to use these tensors in the attention layers.<br>
Hypothetically imagine that I have the word embeddings so I don’t need the embedding layer of transformer,<br>
but I want to use the position embedding since, the position of these tensors are important for me. That is for the input X of size [2,100] , the notion of sequence is important.</p>

Paulo_Mann: <p>I think it could be possible for you to implement something like <code>PackedSequence</code> implements, but for your case. In other words, you can manually pad your “sequences” (they are really not, as you mentioned) and pass their “lengths” along with them through the Dataset <code>collate_fn</code> function, so you know how to get the essential values back (i.e., you have a size of (5,100), but in fact you only need [0:2, :] from this vector). In practice, you could do something like it:</p>
<pre><code class="lang-auto">def collate_fn(data):
    """
       data: is a list of tuples with (example, label, length)
             where 'example' is a tensor of arbitrary shape
             and label/length are scalars
    """
    _, labels, lengths = zip(*data)
    max_len = max(lengths)
    n_ftrs = data[0][0].size(1)
    features = torch.zeros((len(data), max_len, n_ftrs))
    labels = torch.tensor(labels)
    lengths = torch.tensor(lengths)

    for i in range(len(data)):
        j, k = data[i][0].size(0), data[i][0].size(1)
        features[i] = torch.cat([data[i][0], torch.zeros((max_len - j, k))])

    return features.float(), labels.long(), lengths.long()
</code></pre>
<p>The function above is fed to the <code>collate_fn</code> param in the DataLoader, as this example:</p>
<p><code>DataLoader(toy_dataset, collate_fn=collate_fn, batch_size=5)</code></p>
<p>With this <code>collate_fn</code> function, you always gonna have a tensor where all your examples have the same size. So, when you feed your <code>forward()</code> function with this data, you need to use the length to get the original data back, to not use those meaningless zeros in your computation.</p>

pinocchio: <p>what is the use of returning lengths?</p>

pinocchio: <p>also, how would you use the built in <code>torch.nn.utils.rnn.pad_sequence</code> in your example?</p>
<p>I did some examples and this is what I got:</p>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (batch != 0).to(device)
    return batch, lengths, mask
</code></pre>
<p>I think it should work for arbitrary sequences of tensors with weird dimensions.</p>

Paulo_Mann: <blockquote>
<p>What is the use of returning lengths?</p>
</blockquote>
<p>I think it can be better understood by an example. Suppose we have a problem where I need to classify whether a video is a teaching class or a person practicing a sport. Videos are composed of a set of frames, where each one is an image. So, one way to go is to pass the set of frames through an LSTM model. Now suppose every frame of our video have <code>torch.size([C, H, W])</code>, where C is the RGB channels, H is the height and W is the width of the image. We also have a set of videos, in any case, every video might have a different length; therefore, a different number of total frames. For example, the <strong>video<sub>1</sub></strong> have 5350 frames while <strong>video<sub>2</sub></strong> have 3323 frames. You can model <strong>video<sub>1</sub></strong> and <strong>video<sub>2</sub></strong> with the following tensors: <code>torch.size([5350, C, H, W])</code> and <code>torch.size([3323, C, H, W])</code> respectively. As you can see, both tensors have different sizes in the first dimension, which prevents us from <em>stacking</em> both tensors in only one tensor. To make this happens, we can save a tensor called <code>lengths = [5350, 3323]</code> and then pad all videos tensors with zeros to make them have <strong>equal length</strong>, <em>i.e.</em>, both have the size of the biggest length, which is 5350, resulting in two tensors with the following shape: <code>torch.size([5350, C, H, W])</code>. Then, after that, we can <em>stack</em> both tensor to obtain only 1 tensor with the following shape: <code>torch.size([2, 5350, C, H, W])</code>, which means that 2 is the <code>batch_size</code> (you can stack them <a href="https://pytorch.org/docs/stable/torch.html#torch.stack" rel="noopener nofollow ugc">with this function</a>). But, as you can see, we have lost the information on the sequence when stacking both tensors, which means that for the tensor of <strong>video<sub>2</sub></strong>, all examples of <code>video2_tensor[3324:, ...]</code> will have 0 as values. To remedy this, we need to use the <code>lengths</code> vector to get the original sequence back, and not a bunch of zeros.</p>
<blockquote>
<p>also, how would you use the built in  <code>torch.nn.utils.rnn.pad_sequence</code>  in your example</p>
</blockquote>
<p>Yes! You could use it, and your code seems fine to me. But why the <code>mask = (batch != 0).to(device)</code> line?</p>

pinocchio: <p>seems there are many related posts. I collected a list to have them all accessible in one place:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773" class="inline-onebox">How to create batches of a list of varying dimension tensors?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a></li>
<li><a href="https://discuss.pytorch.org/t/using-variable-sized-input-is-padding-required/18131" class="inline-onebox">Using variable sized input - Is padding required?</a></li>
<li><a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418" class="inline-onebox">DataLoader for various length of data</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442" class="inline-onebox">How to do padding based on lengths?</a></li>
</ul>
<p>bucketing:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></li>
</ul>
<hr>
<p>Stack-overflow question:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    <img alt="Trung Le" src="https://lh4.googleusercontent.com/-zBEaoloCnKo/AAAAAAAAAAI/AAAAAAAAAA4/DuvhLWns_Ak/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">How does Pytorch Dataloader handle variable size data?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, tensor, variable-length</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    Trung Le
  </a>
  on <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">10:08AM - 07 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>crossposted: <a href="https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data" rel="nofollow noopener">https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data</a></p>

marctorsoc: <p>Just to add something, it is a lot more efficient to pad batch by batch (via collate_fn) rather than padding the whole dataset, so I strongly encourage to do that. I have an example for sequence tagging in <a href="https://github.com/marctorrellas/conl2002" rel="nofollow noopener">https://github.com/marctorrellas/conl2002</a>. Feel free to ask if any doubts</p>

aarish: <p>Thanks for this! I just had a follow up question - once I pass the batch to the forward function, the only solution I can think of is to iterate through each individual tensor in order to remove padding (since each tensor has its own length). Is there a more elegant way of doing this? Iterating through each tensor in the batch would be very inefficient and time consuming</p>