How to use variable-sized input to Dataloader

Sine: <p>My input is variable-sized, such as torch.Size([4, 2, 100]), torch.Size([4, 2, 100]), torch.Size([5, 2, 100]), torch.Size([6, 2, 100]), torch.Size([6, 2, 100])</p>
<p>I want the same size in each batch, for example torch.Size([4, 2, 100]), torch.Size([4, 2, 100]) are in the same batch.</p>
<p>My original method is as follow:</p>
<ol>
<li>Sorting the input by the first dimension size.</li>
<li>Dividing into a list of tuple with batch size, like [ (Size([4, 2, 100]),Size([4, 2, 100])), (Size([5, 2, 100]), Size([5, 2, 100]))]. Batch size is 2.</li>
<li>Shffuling the index of the list.</li>
</ol>
<p>Although my method can work, I want to speed up the program by using Dataloader.</p>
<p>Does the input with the smaller original size (such as Size([1, 2, 100])) have any effect during training when I pad all the inputs into the same size (such as Size([10, 2, 100]))?</p>
<p>Thank you.</p>

InnovArul: <aside class="quote no-group" data-username="Sine" data-post="1" data-topic="106544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sine/48/32491_2.png" class="avatar"> Sine:</div>
<blockquote>
<p>My input is variable-sized, such as torch.Size([4, 2, 100]), torch.Size([4, 2, 100]), torch.Size([5, 2, 100]), torch.Size([6, 2, 100]), torch.Size([6, 2, 100])</p>
</blockquote>
</aside>
<p>Can you elaborate a bit more on the usecase?</p>
<aside class="quote no-group" data-username="Sine" data-post="1" data-topic="106544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sine/48/32491_2.png" class="avatar"> Sine:</div>
<blockquote>
<p>when I pad all the inputs into the same size (such as Size([10, 2, 100]))?</p>
</blockquote>
</aside>
<p>Yes, you can always pad the input to a standard size to get it working with dataloader.<br>
Even otherwise, you can write a custom collate function to handle different-sized data instances.<br>
I am not sure of your usecase though</p>

Sine: <aside class="quote no-group" data-username="InnovArul" data-post="2" data-topic="106544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>Can you elaborate a bit more on the usecase?</p>
</blockquote>
</aside>
<p>My input is the multi-turned dialogue, and its size is [context length, utterance length, embedding size].<br>
However, the context length is variable-sized.</p>

InnovArul: <p>A simple way would be to use a custom collate function and collect the variable-sized data to make a batch.<br>
There are other advanced solutions with <code>padded_sequence</code>s, as well.</p>
<aside class="quote quote-modified" data-post="15" data-topic="6418">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pinocchio/48/22093_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418/15">DataLoader for various length of data</a> 
  </div>
  <blockquote>
    This is how I solved it: 
def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (ba…
  </blockquote>
</aside>


Sine: <aside class="quote no-group" data-username="InnovArul" data-post="4" data-topic="106544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/48/5282_2.png" class="avatar"> InnovArul:</div>
<blockquote>
<p>A simple way would be to use a custom collate function and collect the variable-sized data to make a batch.<br>
There are other advanced solutions with <code>padded_sequence</code>s, as well.</p>
</blockquote>
</aside>
<p>Thank you.</p>
<p>Does padded sequence interfere with training?<br>
I use Transformer model.<br>
For example,<br>
[[[1,2,3,4,0,0,0],[2,3,5,0,0,0,0],[0,0,0,0,0,0,0]],[[3,2,3,4,0,0,0],[0,0,0,0,0,0,0],[0,0,0,0,0,0,0]]]</p>

InnovArul: <aside class="quote no-group" data-username="Sine" data-post="5" data-topic="106544">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sine/48/32491_2.png" class="avatar"> Sine:</div>
<blockquote>
<p>[[[1,2,3,4,0,0,0],[2,3,5,0,0,0,0],[0,0,0,0,0,0,0]],[[3,2,3,4,0,0,0],[0,0,0,0,0,0,0],[0,0,0,0,0,0,0]]]</p>
</blockquote>
</aside>
<p>It would increase the computation. Apart from that, you may have to assign the logits of the padded entries to a higher negative value, so that they don’t contribute to attention probability.</p>