Using variable sized input - Is padding required?

gcamilo: <p>My input has variable size. I haven’t found a way to use a DatasetLoader without padding the inputs with the maximum size in the batch. Is there any way around it? Is it possible without using the DatasetLoader class?</p>

justusschock: <p>A batch must always consist of elements of the same size. However if your input is large enough and you can handle the corresponding output sizes you can feed batches off different sizes to the model.</p>

gcamilo: <p>How can I vary the batch size?</p>

justusschock: <p>This does not work with the default dataloader but in general you could handle the loading by yourself and simply add a batch dimension to your data and use <a href="https://pytorch.org/docs/stable/torch.html?highlight=cat#torch.cat" rel="nofollow noopener">torch.cat</a> to stack them to a batch:</p>
<pre><code class="lang-auto">batch_elements = []

for i in range(curr_batch_size):
    # generate some sample data
    tmp = torch.rand((1, 50, 50))
    # add batch dimension
    tmp = tmp.unsqueeze(0)
    batch_elements.append(tmp)
batch = torch.cat(batch_elements, 0)
</code></pre>
<p>Replace <code>tmp = torch.rand((1, 50, 50))</code> by your own data samples. In This case I used 50x50 pixel images with one channel as sample data. To show the workflow with general data I did not integrate the batch dimension into the shape of the random tensor but added it afterwards.</p>
<p>EDIT: Alternatively you could use something like <a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418?u=justusschock">this</a>. But note that this will pad your input and (depending on the maximal difference of your input sizes) you could end up with padding an enormous amount of zeros (or constants or whatever).</p>

Brando_Miranda: <aside class="quote no-group" data-username="justusschock" data-post="2" data-topic="18131">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/justusschock/48/4637_2.png" class="avatar"> justusschock:</div>
<blockquote>
<p>However if your input is large enough and you can handle the corresponding output sizes you can feed batches off different sizes to the model.</p>
</blockquote>
</aside>
<p>How exactly are you doing this? Your answer seems to assume everything is of same size already.</p>
<p>The original question is if padding is required for variable size input. Let’s be direct and answer that directly. Is that always necessary or not? When is padding necessary and when is it not?</p>
<p>To my understanding it’s always required cuz batches have to be of the same size (unless I’m not understanding something or don’t know something).</p>

justusschock: <p>Items in the same batch have to be the same size, yes, but having a fully convolutional network you can pass batches of different sizes, so no, padding is not always required. In the extreme case you could even use batchsize of 1 and your input size could be completely random (assuming, that you adjusted strides, kernelsize, dilation etc in a proper way).</p>
<p>This is why it is hard to answer this question in general.</p>

pinocchio: <p>My take on how to solve this issue:</p>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (batch != 0).to(device)
    return batch, lengths, mask
</code></pre>
<hr>
<p>There seems to be a large collection of posts all over pytorch that makes it difficult to solve this issue. I have collected a list of all of them hopefully making things easier for all of us. Here:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773" class="inline-onebox">How to create batches of a list of varying dimension tensors?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a></li>
<li><a href="https://discuss.pytorch.org/t/using-variable-sized-input-is-padding-required/18131" class="inline-onebox">Using variable sized input - Is padding required?</a></li>
<li><a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418" class="inline-onebox">DataLoader for various length of data</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442" class="inline-onebox">How to do padding based on lengths?</a></li>
</ul>
<p>bucketing:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></li>
</ul>
<hr>
<p>Also, Stack-overflow has a version of this question too:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    <img alt="Trung Le" src="https://lh4.googleusercontent.com/-zBEaoloCnKo/AAAAAAAAAAI/AAAAAAAAAA4/DuvhLWns_Ak/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">How does Pytorch Dataloader handle variable size data?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, tensor, variable-length</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    Trung Le
  </a>
  on <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">10:08AM - 07 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>crossposted: <a href="https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data" rel="nofollow noopener">https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data</a></p>

thawro: <p>try this: <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/20">link</a></p>