How to create a dataloader with variable-size input

zchrissirhcz: <p>Hi, I’d like to create a dataloader with different size input images, but don’t know how to do that.</p>
<p>I’ve read the official tutorial on loading custum data (<a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Writing Custom Datasets, DataLoaders and Transforms — PyTorch Tutorials 2.1.1+cu121 documentation</a>), however in the tutorial, all the input images are rescaled to 256x256 and randomly cropped to 224*224.</p>
<p>I’d like to do another image classification task, and in my task I want to: keep image their original size, and the batch size should be greater than 1.</p>
<p>I noticed that there exists related questions/issues, such as: <a href="https://github.com/pytorch/pytorch/issues/1512" class="inline-onebox" rel="noopener nofollow ugc">[feature request] Support tensors of different sizes as batch elements in DataLoader · Issue #1512 · pytorch/pytorch · GitHub</a></p>
<aside class="quote" data-post="1" data-topic="1906">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kirk86/48/151_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-create-convnet-for-variable-size-input-dimension-images/1906">How to create convnet for variable size input dimension images</a> 
  </div>
  <blockquote>
    Hey guys, I was wondering if it’s possible to create a convnet with variable size input images as training data. If so can someone provide a simple example on how to achieve sth like that. For instance in tensorflow I would go and simply define the input shape on the first conv layer as (None, None, 3). Can we do sth like that in pytorch? 
Thanks!
  </blockquote>
</aside>

<p>However, none of above gives exactly implementation details on how to create a variable-input size dataloader.</p>
<p>At the same time, I try to do modification on that dataloader tutorial’s IPython Notebook file. I defined my own dataset class and own ToTensor() function, and only do ToTensor() operation on transform parameter. In the <strong>Iterating through the dataset</strong> part, it can only show several batch of data, then crash like:</p>
<pre><code>---------------------------------------------------------------------------
</code></pre>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
25         plt.title(‘Batch from dataloader’)<br>
26<br>
—&gt; 27 for i_batch, sample_batched in enumerate(dataloader):<br>
28     print(i_batch, sample_batched[‘image’].size(),<br>
29           sample_batched[‘label’].size())</p>
<p>/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in <strong>next</strong>(self)<br>
199                 self.reorder_dict[idx] = batch<br>
200                 continue<br>
 → 201             return self._process_next_batch(batch)<br>
202<br>
203     next = <strong>next</strong>  # Python 2 compatibility</p>
<p>/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in _process_next_batch(self, batch)<br>
219         self._put_indices()<br>
220         if isinstance(batch, ExceptionWrapper):<br>
 → 221             raise batch.exc_type(batch.exc_msg)<br>
222         return batch<br>
223</p>
<p>RuntimeError: Traceback (most recent call last):<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 40, in _worker_loop<br>
samples = collate_fn([dataset[i] for i in batch_indices])<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 106, in default_collate<br>
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 106, in <br>
return {key: default_collate([d[key] for d in batch]) for key in batch[0]}<br>
File “/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py”, line 91, in default_collate<br>
return torch.stack(batch, 0, out=out)<br>
File “/usr/local/lib/python2.7/dist-packages/torch/functional.py”, line 66, in stack<br>
return torch.cat(inputs, dim, out=out)<br>
RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2709</p>
<p>Any idea? Thanks!</p>

smth: <p>you need to write a custom collate_fn and pass it to your data loader. <a href="https://discuss.pytorch.org/t/questions-about-dataloader-and-dataset/806/2?u=smth">Questions about Dataloader and Dataset</a></p>

jdhao: <p>By default, torch stacks the input image to from a tensor of size <code>N*C*H*W</code>, so every image in the batch must have the same height and width. In order to load a batch with variable size input image, we have to use our own <code>collate_fn</code> which is used to pack a batch of images.</p>
<p>For image classification, the input to <code>collate_fn</code> is a list of with size <code>batch_size</code>. Each element is a tuple where the first element is the input image(a <code>torch.FloatTensor</code>) and the second element is the image label which is simply an <code>int</code>. Because the samples in a batch have different size, we can store these samples in a list ans store the corresponding labels in <code>torch.LongTensor</code>. Then we put the image list and the label tensor into a list and return the result.</p>
<p>here is a very simple snippet to demonstrate how to write a custom <code>collate_fn</code>:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader
from torchvision import transforms
import torchvision.datasets as datasets
import matplotlib.pyplot as plt

# a simple custom collate function, just to show the idea
def my_collate(batch):
    data = [item[0] for item in batch]
    target = [item[1] for item in batch]
    target = torch.LongTensor(target)
    return [data, target]


def show_image_batch(img_list, title=None):
    num = len(img_list)
    fig = plt.figure()
    for i in range(num):
        ax = fig.add_subplot(1, num, i+1)
        ax.imshow(img_list[i].numpy().transpose([1,2,0]))
        ax.set_title(title[i])

    plt.show()

#  do not do randomCrop to show that the custom collate_fn can handle images of different size
train_transforms = transforms.Compose([transforms.Scale(size = 224),
                                       transforms.ToTensor(),
                                       ])

# change root to valid dir in your system, see ImageFolder documentation for more info
train_dataset = datasets.ImageFolder(root="/hd1/jdhao/toyset",
                                     transform=train_transforms)

trainset = DataLoader(dataset=train_dataset,
                      batch_size=4,
                      shuffle=True,
                      collate_fn=my_collate, # use custom collate function here
                      pin_memory=True)

trainiter = iter(trainset)
imgs, labels = trainiter.next()

# print(type(imgs), type(labels))
show_image_batch(imgs, title=[train_dataset.classes[x] for x in labels])
</code></pre>

rbrigden: <p>Currently I am using AdaptiveAvgPool2d to downsample variable sized images. Your strategy of using a custom collate function to build a dataloader for variable sized input is very useful, but only gets me half-way there. I want to improve GPU performance by feeding the data into AdaptiveAvgPool2d by batch, so that the data is sent to the data by batch, instead of by sample. Any suggestions?</p>

jdhao: <p>I do not think that is possible, for a batch of images which have variable size.The output size will only be same after the <code>AdaptiveAvgPool2d</code> layer. You can not stack image before that layer.</p>
<p>Also, if you want to stack image after <code>AdaptiveAvgPool2d</code>, you may have to separate your network into two part, which also makes the network more complex.</p>
<p>I suggest that you calculate the loss for each image and accumulate the loss, after you do <code>N</code> forward, do one backward to update the network parameter.  You can see some discussions <a href="https://discuss.pytorch.org/t/pytorch-gradients/884/10">here</a> and <a href="https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20">here</a>.</p>
<p>Another way is to just pad your images to have the same size and keep the image aspect ratio. I think it is also a viable solution if you do not want to change the image aspect ratio.</p>

deeplearning09: <p>Hey!<br>
How did you structure your neural network to have varying input sizes?</p>

snakers41: <p>Hi guys!</p>
<p>Please consider this <a href="https://github.com/pytorch/pytorch/issues/1512#issuecomment-405015099" rel="nofollow noopener">idea</a> I just came up with.</p>
<p>The ideas is:</p>
<ul>
<li>have multiple heads in your CNN and some if-logic in forward statement</li>
<li>have a cluster_index for each different type of image</li>
<li>feed only consistent batches, but randomly (batch 1 - size 1, batch 2 - size 2, batch 3 - size 1, etc)</li>
</ul>

PRAVEEN_KUMAR: <p>I have the <strong>data</strong> folder in this 4 subfolder <strong>(cat1,cat2,cat3,cat4)</strong> I want to read data folder according to the cat (subfolder )  wise not according to data(root) folder wise. or read data according to data(root) folder but have separate out data folder according to the cat(subfolder) wise.<br>
How design this type customeDataloader.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/ba6c3b0ab7574a8aef512e82bffad7402210b4ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba6c3b0ab7574a8aef512e82bffad7402210b4ff" title="folderstructure"><img src="https://discuss.pytorch.org/uploads/default/original/2X/b/ba6c3b0ab7574a8aef512e82bffad7402210b4ff.png" alt="folderstructure" data-base62-sha1="qBaGWdken9meckMnLuNV5m8SPdZ" width="395" height="500" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">folderstructure</span><span class="informations">468×592 3.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

HotpotPandaXG: <p>hi did you solve this problem eventually? thanks</p>

Harkirat_Behl: <p>Hi Guys,</p>
<p>For the case where you want to work with batches, and the image/label dimension in different batches might be different, you want to modify the collate_fn as pointed out by smth. The correct way is to construct a new collate fn.<br>
Lets say that label for image 1 has dimension 10, whereas label for image 2 has dimension 15, and you are trying to create a batch of these images.<br>
The main problem with unequal sizes is that the current collate_fn implementation uses itertools.zip. This creates an iteratable with the size of the smaller list, thus it will only create an iteratable with 10 elements in this case.<br>
The solution is to use itertools.zip_longest. This will create an iteratable of dimension of the largest in the batch, i.e 15 in our cases. This function uses padding, so the values will be None for label 1 in the rest of the dimensions, which you will have to take care during run time in your code after you have got the batch.<br>
Hope this helps! Let me know if anything else is needed</p>

gollum: <p>Can I use torch.nn.DataParallel wrapped around my model after making a collate function to tackle variable sized inputs??<br>
If yes how? Because when using a custom collate function like <a href="https://github.com/txie-93/cgcnn/blob/b98c1cd109da971f599dd492ac7a5dad2c29e886/cgcnn/data.py#L96" rel="nofollow noopener">this</a> I am unable to run my model on multiple GPUs</p>

pinocchio: <p>There seems to be a large collection of posts all over pytorch that makes it difficult to solve this issue. I have collected a list of all of them hopefully making things easier for all of us. Here:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773" class="inline-onebox">How to create batches of a list of varying dimension tensors?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278" class="inline-onebox">How to create a dataloader with variable-size input</a></li>
<li><a href="https://discuss.pytorch.org/t/using-variable-sized-input-is-padding-required/18131" class="inline-onebox">Using variable sized input - Is padding required?</a></li>
<li><a href="https://discuss.pytorch.org/t/dataloader-for-various-length-of-data/6418" class="inline-onebox">DataLoader for various length of data</a></li>
<li><a href="https://discuss.pytorch.org/t/how-to-do-padding-based-on-lengths/24442" class="inline-onebox">How to do padding based on lengths?</a></li>
</ul>
<p>bucketing:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284" class="inline-onebox">Tensorflow-esque bucket by sequence length</a></li>
</ul>
<hr>
<p>Also, Stack-overflow has a version of this question too:</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    <img alt="Trung Le" src="https://lh4.googleusercontent.com/-zBEaoloCnKo/AAAAAAAAAAI/AAAAAAAAAA4/DuvhLWns_Ak/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">How does Pytorch Dataloader handle variable size data?</a>
</h4>

<div class="tags">
  <strong>python, pytorch, tensor, variable-length</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/10945214/trung-le" target="_blank" rel="nofollow noopener">
    Trung Le
  </a>
  on <a href="https://stackoverflow.com/questions/55041080/how-does-pytorch-dataloader-handle-variable-size-data" target="_blank" rel="nofollow noopener">10:08AM - 07 Mar 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<hr>
<p>crossposted: <a href="https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data" rel="nofollow noopener">https://www.quora.com/unanswered/How-does-Pytorch-Dataloader-handle-variable-size-data</a></p>

pinocchio: <p>My version of how to solve the problem is as follows:</p>
<pre><code class="lang-auto">def collate_fn_padd(batch):
    '''
    Padds batch of variable length

    note: it converts things ToTensor manually here since the ToTensor transform
    assume it takes in images rather than arbitrary tensors.
    '''
    ## get sequence lengths
    lengths = torch.tensor([ t.shape[0] for t in batch ]).to(device)
    ## padd
    batch = [ torch.Tensor(t).to(device) for t in batch ]
    batch = torch.nn.utils.rnn.pad_sequence(batch)
    ## compute mask
    mask = (batch != 0).to(device)
    return batch, lengths, mask
</code></pre>

Arta_A: <aside class="quote no-group" data-username="jdhao" data-post="3" data-topic="8278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jdhao/48/2558_2.png" class="avatar"> jdhao:</div>
<blockquote>
<p>collate_fn=my_collate</p>
</blockquote>
</aside>
<p>How can I use this method when my original and target are images, like pix2pix or semantic segmentation tasks? I asked <a href="https://discuss.pytorch.org/t/invalid-argument-tensors-for-dataloader-pytoch/55449">this</a> question someone referred to this post for a detailed explanation.</p>

CuriousCat-7: <p>In your case, I think you can modify collate function to make that return list of tensor rather than stack of tensor. The mmdetection is a complicated but good example.</p>

chocolocked: <p>Just wondering, since batch will results in a list instead, where to put the ‘.to(device)’ when running on the GPU. apparently now we cannot<br>
for x, y in batch:<br>
x.to(device), y.to(device)</p>

yongen9696: <p>You have to convert them into tensor type.</p>

hitbuyi: <p>It’s helpful, what’s parameter batch from in your function?</p>

thawro: <p>You can just implement a custom batch sampler which ensures that only batches with same input size are created. Look <a href="https://stackoverflow.com/a/77481630/10816142" rel="noopener nofollow ugc">here</a></p>