DataLoader for various length of data

username: cdjhz
cooked: <p>I’ve been working on implementing a seq2seq model and tried to use <code>torch.utils.data.DataLoader</code> to batch data following the <strong>Data Loading and Processing Tutorial</strong>. It seems DataLoader cannot handle various length of data. Or are there other ways to batch different length of data?</p>

username: dhpollack
cooked: <p>you could create a transformation that trims / pads each sample to a specific length and then use the <a href="http://pytorch.org/docs/master/nn.html?#torch.nn.utils.rnn.pack_padded_sequence" rel="nofollow noopener">pack padded sequence</a> function</p>

username: chihyaoma
cooked: <p>You need to customize your own dataloader.</p>
<p>What you need is basically pad your variable-length of input and torch.stack() them together into a single tensor. This tensor will then be used as an input to your model.</p>
<p>I think it’s worth to mention that using <code>pack_padded_sequence</code> isn’t absolutely necessary. <code>pack_padded_sequence</code> is kind of designed to work with the LSTM/GPU/RNN from cuDNN. They are optimized to run very fast.</p>
<p>But, if you have your own proposed method that prevents you from using standard LSTM/GPU/RNN, as mentioned <a href="https://discuss.pytorch.org/t/batch-processing-with-variable-length-sequences/3150/7">here</a>:</p>
<blockquote>
<p>The easiest way to make a custom RNN compatible with variable-length sequences is to do what this repo does (masking) <a href="https://github.com/jihunchoi/recurrent-batch-normalization-pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - jihunchoi/recurrent-batch-normalization-pytorch: PyTorch implementation of recurrent batch normalization</a></p>
</blockquote>

username: cdjhz
cooked: <p>Thx sir. Do you mean getting a batch of data and padding them manually? That’s exactly what I’m doing. I’m just wondering if there’s a ‘pytorch’ proper way to do this.</p>

username: dhpollack
cooked: <p>I meant to create your own Dataset class and then do a transform to pad to a given length.  An example of a custom dataset class below.  The idea would be to add a transform to that which pads to tensors so that upon every call of <strong>getitem</strong>() the tensors are padded and thus the batch is all padded tensors.  You could also have the <strong>getitem</strong>() function return a third value, which is the original length of the tensor so you can do masking.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/datasets/mnist.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
import torch.utils.data as data
from PIL import Image
import os
import os.path
import errno
import numpy as np
import torch
import codecs


class MNIST(data.Dataset):
    """`MNIST &lt;http://yann.lecun.com/exdb/mnist/&gt;`_ Dataset.

    Args:
        root (string): Root directory of dataset where ``processed/training.pt``
            and  ``processed/test.pt`` exist.
        train (bool, optional): If True, creates dataset from ``training.pt``,
            otherwise from ``test.pt``.
        download (bool, optional): If true, downloads the dataset from the internet and
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Felix_Kreuk
cooked: <p>I was wondering if there is a more efficient way of padding sequences. The easiest option is to just pad all sequences to the max length possible, currently I implemented my own Dataset object and use a Transform that pads all sequences to the same length. But is there a way to do that per batch and not globally for the whole dataset (pad the batch when DataLoader samples the batch)? Sounds like I need to create a DataLoader?</p>
<p>Edit:<br>
I found a possible solution at: <a href="http://pytorch.org/docs/master/_modules/torch/utils/data/sampler.html" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/utils/data/sampler.html</a>.<br>
Specifically, you can implement your own version of <code>BatchSampler</code> to padd the according to the longest sequence in the batch. I will post my implementatino when done.</p>

username: dhpollack
cooked: <p>I think you want to use the collate_fn function in the DataLoader class.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/dhpollack/programming_notebooks/blob/master/pytorch_attention_audio.py#L245" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/dhpollack/programming_notebooks/blob/master/pytorch_attention_audio.py#L245" target="_blank" rel="nofollow noopener">dhpollack/programming_notebooks/blob/master/pytorch_attention_audio.py#L245</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="235" style="counter-reset: li-counter 234 ;">
<li>    return Feats</li>
<li>
</li>
<li>class Labeler(object):</li>
<li>"""Labels from text to int + 1</li>
<li>
</li>
<li>"""</li>
<li>
</li>
<li>def __call__(self, labels):</li>
<li>    return torch.LongTensor([int(l)+1 for l in labels])</li>
<li>
</li>
<li class="selected">def pad_packed_collate(batch):</li>
<li>"""Puts data, and lengths into a packed_padded_sequence then returns</li>
<li>   the packed_padded_sequence and the labels. Set use_lengths to True</li>
<li>   to use this collate function.</li>
<li>
</li>
<li>   Args:</li>
<li>     batch: (list of tuples) [(audio, target)].</li>
<li>         audio is a FloatTensor</li>
<li>         target is a LongTensor with a length of 8</li>
<li>   Output:</li>
<li>     packed_batch: (PackedSequence), see torch.nn.utils.rnn.pack_padded_sequence</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I did one there with packed sequences.  I don’t know if this is the fastest way, but it would accomplish what you want to do.  Also you could use any of the pre-built samplers that you wanted.</p>

username: Felix_Kreuk
cooked: <p>Thanks David, <code>collate_fn</code> was a good direction <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:">.  I wrote a simple code that maybe someone here can re-use. I wanted to make something that pads a generic dim, and I don’t use an RNN of any type so PackedSequence was a bit of overkill for me. It’s simple, but it works for me.</p>
<pre><code class="lang-auto">def pad_tensor(vec, pad, dim):
    """
    args:
        vec - tensor to pad
        pad - the size to pad to
        dim - dimension to pad

    return:
        a new tensor padded to 'pad' in dimension 'dim'
    """
    pad_size = list(vec.shape)
    pad_size[dim] = pad - vec.size(dim)
    return torch.cat([vec, torch.zeros(*pad_size)], dim=dim)


class PadCollate:
    """
    a variant of callate_fn that pads according to the longest sequence in
    a batch of sequences
    """

    def __init__(self, dim=0):
        """
        args:
            dim - the dimension to be padded (dimension of time in sequences)
        """
        self.dim = dim

    def pad_collate(self, batch):
        """
        args:
            batch - list of (tensor, label)

        reutrn:
            xs - a tensor of all examples in 'batch' after padding
            ys - a LongTensor of all labels in batch
        """
        # find longest sequence
        max_len = max(map(lambda x: x[0].shape[self.dim], batch))
        # pad according to max_len
        batch = map(lambda (x, y):
                    (pad_tensor(x, pad=max_len, dim=self.dim), y), batch)
        # stack all
        xs = torch.stack(map(lambda x: x[0], batch), dim=0)
        ys = torch.LongTensor(map(lambda x: x[1], batch))
        return xs, ys

    def __call__(self, batch):
        return self.pad_collate(batch)


</code></pre>
<p>to be used with the data loader:<br>
<code>train_loader = DataLoader(ds, ..., collate_fn=PadCollate(dim=0))</code></p>

username: dhpollack
cooked: <p>Felix, I think your code only pads correctly if dim=0.  This is because in the pad vector in the pad_tensor function has *vec.size()[1:] hardcoded into it.  I think you need to create a vector that is pad - vec.size(dim) in the dim dimension and not always in the zeroth dimension.  However, I could be wrong.  I adapted the code to work with python3 and added the ability to pad with different values, so I may have screwed something up in the process.</p>

username: Felix_Kreuk
cooked: <p>David, you are correct, I updated the <code>pad</code> function to work with any <code>dim</code>, thanks.</p>

username: oegedijk
cooked: <p>If you are going to pack your padded sequences later, you can also immediately sort the batches from longest sequence to shortest:</p>
<pre><code class="lang-auto">def sort_batch(batch, targets, lengths):
    """
    Sort a minibatch by the length of the sequences with the longest sequences first
    return the sorted batch targes and sequence lengths.
    This way the output can be used by pack_padded_sequences(...)
    """
    seq_lengths, perm_idx = lengths.sort(0, descending=True)
    seq_tensor = batch[perm_idx]
    target_tensor = targets[perm_idx]
    return seq_tensor, target_tensor, seq_lengths

def pad_and_sort_batch(DataLoaderBatch):
    """
    DataLoaderBatch should be a list of (sequence, target, length) tuples...
    Returns a padded tensor of sequences sorted from longest to shortest, 
    """
    batch_size = len(DataLoaderBatch)
    batch_split = list(zip(*DataLoaderBatch))

    seqs, targs, lengths = batch_split[0], batch_split[1], batch_split[2]
    max_length = max(lengths)

    padded_seqs = np.zeros((batch_size, max_length))
    for i, l in enumerate(lengths):
        padded_seqs[i, 0:l] = seqs[i][0:l]

    return sort_batch(torch.tensor(padded_seqs), torch.tensor(targs).view(-1,1), torch.tensor(lengths))
</code></pre>
<p>This assumes that your Dataset spits out something like</p>
<pre><code class="lang-auto">def __getitem__(self, idx):
        return self.sequences[idx], torch.tensor(self.targets[idx]), self.sequence_lengths[idx]
</code></pre>
<p>And the  you pass the pad_and_sort collator to the DataLoader as:</p>
<p><code>train_gen = Data.DataLoader(train_data, batch_size=128, shuffle=True, collate_fn=pad_and_sort_batch)</code></p>

username: TrentBrick
cooked: <p>For future readers, if like me you were looking for the above (which is great) but also to batch your sequences by their length to minimize padding necessary then I wrote a Batch Sampler for this: <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></p>

username: GalAvineri
cooked: <p>To answer the original question, you can pass a (simple and short) custom collate function to the data loader that uses <code>pack_sequence</code>.</p>
<p><code>pack_sequence</code> does not require the sequences to be padded or sorted by length, so it is simpler to use.</p>
<p>Here is the code that does this (based on this answer to a similar question: <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3" class="inline-onebox">How to create a dataloader with variable-size input</a> )</p>
<pre><code class="lang-auto">from torch.nn.utils.rnn import pack_sequence
from torch.utils.data import DataLoader

def my_collate(batch):
    # batch contains a list of tuples of structure (sequence, target)
    data = [item[0] for item in batch]
    data = pack_sequence(data, enforce_sorted=False)
    targets = [item[1] for item in batch]
    return [data, targets]

# ...
# later in you code, when you define you DataLoader - use the custom collate function
loader = DataLoader(dataset,
                      batch_size,
                      shuffle,
                      collate_fn=my_collate, # use custom collate function here
                      pin_memory=True)
</code></pre>

username: pinocchio
cooked: <p>This is how I solved it:</p>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (batch != 0).to(device)
    return batch, lengths, mask
</code></pre>
<hr>
<p>Related posts:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773" class="inline-onebox">How to create batches of a list of varying dimension tensors?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a></li>
<li><a href="https://discuss.pytorch.org/t/using-variable-sized-input-is-padding-required/18131" class="inline-onebox">Using variable sized input - Is padding required?</a></li>
<li><a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418" class="inline-onebox">DataLoader for various length of data</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442" class="inline-onebox">How to do padding based on lengths?</a></li>
</ul>
<p>bucketing:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></li>
</ul>
<hr>
<p>Stack overflows version:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    <img alt="Trung Le" src="https://lh4.googleusercontent.com/-zBEaoloCnKo/AAAAAAAAAAI/AAAAAAAAAA4/DuvhLWns_Ak/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">How does Pytorch Dataloader handle variable size data?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, tensor, variable-length</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    Trung Le
  </a>
  on <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">10:08AM - 07 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>crossposted: <a href="https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data" rel="nofollow noopener">https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data</a></p>

username: Amin_Jun
cooked: <p>Here’s a collator I use, it works for tensors of any dimension:</p>
<pre><code class="lang-auto">class ZeroPadCollator:

    @staticmethod
    def collate_tensors(batch: List[torch.Tensor]) -&gt; torch.Tensor:
        dims = batch[0].dim()
        max_size = [max([b.size(i) for b in batch]) for i in range(dims)]
        size = (len(batch),) + tuple(max_size)
        canvas = batch[0].new_zeros(size=size)
        for i, b in enumerate(batch):
            sub_tensor = canvas[i]
            for d in range(dims):
                sub_tensor = sub_tensor.narrow(d, 0, b.size(d))
            sub_tensor.add_(b)
        return canvas

    def collate(self, batch, ) -&gt; List[torch.Tensor]:
        dims = len(batch[0])
        return [self.collate_tensors([b[i] for b in batch]) for i in range(dims)]
</code></pre>
<p>Then I simply use:</p>
<pre><code class="lang-auto">    zero_pad = ZeroPadCollator()
    loader = DataLoader(train, args.batch_size, collate_fn=zero_pad.collate)```</code></pre>

username: ISMAX
cooked: <p>For the others who might have the same issue with RNN and multiple lengths sequences, here is my solution if your dataset __getitem__ method returns a pair (seq, target) :</p>
<pre><code class="lang-python">from torch.nn.utils.rnn import pad_sequence

def collate_fn_pad(list_pairs_seq_target):
    seqs = [seq for seq, target in list_pairs_seq_target]
    targets = [target for seq, target in list_pairs_seq_target]
    seqs_padded_batched = pad_sequence(seqs)   # will pad at beginning of sequences
    targets_batched = torch.stack(targets)
    assert seqs_padded_batched.shape[1] == len(targets_batched)
    return seqs_padded_batched, targets_batched
    
dataloader = DataLoader(dataset, batch_size=32, shuffle=True, collate_fn=collate_fn_pad)

for seq, labels in dataloader:
        y_pred = rnn(seq)
</code></pre>

username: gabrieldernbach
cooked: <p>Out of curiosity. You this at test time only, right? During training you may want truly stochastic batches.</p>