Torchvision and dataloader different images shapes

jmaronas: <p>There are some datasets, specially those for object detection, where the images have different shapes. Is there any way to use <code>torch.utils.data.DataLoader</code> in order to return a batch of images with different shapes, for instance a list of torch.tensors.</p>
<p>Thanks.</p>

SimonW: <p>supply a custom <code>collate_fn</code> to the dataloder</p>

sbelharbi: <p>following <a class="mention" href="/u/simonw">@SimonW</a> 's answer.<br>
Here is an example of <code>collate_fn</code>:<br>
In this example <code>mask</code> are the mask of the images (that have different sizes). While, data are some cropped windows from the images (the crops have the same size). The idea is to use a list to wrap the data with different sizes.</p>
<p>In your case, the images have different sizes, so you can use the same thing done here for the masks. Need to convert what is needed for the forward to tensors.</p>
<pre><code class="lang-python">def default_collate(batch):
    """
    Override `default_collate` https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader

    Reference:
    def default_collate(batch) at https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader
    https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/3
    https://github.com/pytorch/pytorch/issues/1512

    We need our own collate function that wraps things up (imge, mask, label).

    In this setup,  batch is a list of tuples (the result of calling: img, mask, label = Dataset[i].
    The output of this function is four elements:
        . data: a pytorch tensor of size (batch_size, c, h, w) of float32 . Each sample is a tensor of shape (c, h_,
        w_) that represents a cropped patch from an image (or the entire image) where: c is the depth of the patches (
        since they are RGB, so c=3),  h is the height of the patch, and w_ is the its width.
        . mask: a list of pytorch tensors of size (batch_size, 1, h, w) full of 1 and 0. The mask of the ENTIRE image (no
        cropping is performed). Images does not have the same size, and the same thing goes for the masks. Therefore,
        we can't put the masks in one tensor.
        . target: a vector (pytorch tensor) of length batch_size of type torch.LongTensor containing the image-level
        labels.
    :param batch: list of tuples (img, mask, label)
    :return: 3 elements: tensor data, list of tensors of masks, tensor of labels.
    """
    data = torch.stack([item[0] for item in batch])
    mask = [item[1] for item in batch]  # each element is of size (1, h*, w*). where (h*, w*) changes from mask to another.
    target = torch.LongTensor([item[2] for item in batch])  # image labels.

    return data, mask, target
</code></pre>
<p>Instance: <code>dataloader = DataLoader( .... collate_fn=default_collate, ...)</code></p>
<p>Loop over the data loader:</p>
<pre><code class="lang-python">for img, mask, label in train_loader:
   # do stuff.
</code></pre>
<p>In the <code>forward</code> function of your model, you need to treat your input as a list.</p>
<pre><code class="lang-python">class myModel(nn.Module):
    def forward(self, input):
        """ 
         Classify a list of samples.
        :apram input: is a list of n tensors with different height and width ...
        :return scores: tensor of scores of shape (n, #nbr_classes)
         
        """
        scores = []
        for i, x in enumerate(input):
            # x is an image.
            score = # forward x
            if i == 0:
                score = score
            else:
                scores = torch.cat((scores, score), dim=0)
      return scores
</code></pre>

jmaronas: <p>thank you, nice example.</p>

lurenyi233: <p>Hi, sbelharbi, can you tell me more about how to modify the forward function here to " to treat your input as a list"?<br>
As I found the input of CNN can only be Tensor:</p>
<blockquote>
<p>TypeError: conv1d(): argument ‘input’ (position 1) must be Tensor, not list</p>
</blockquote>

sbelharbi: <p>hi,<br>
yes, cnn, and pytorch modules, generally operate on tensors not lists.<br>
the example <a href="https://discuss.pytorch.org/t/torchvision-and-dataloader-different-images-shapes/41026/3">above</a> of the model is meant for the evaluation primarily. although, it could work for training but there may be a gradient issue.</p>
<p>so, in order to process a list of tensors of different shapes without changing your model, you can do the following:</p>
<ol>
<li>the list needs to contain tensors. do not feed the list to the model. the collate function above allows to build such list.</li>
<li>you need to loop over the list OUTSIDE the model. for each loop, you will get a tensor from the list. feed that tensor to your model.</li>
<li>you need to accumulate the gradient (for training). please see <a href="https://discuss.pytorch.org/t/multiple-forward-before-backward-call/20893/3">here</a> two ways for gradient accumulation. each tensor in the list could be one single sample or a tiny mini-batch. make sure to divide the loss by the true total number of samples after forwarding all the tensors.<br>
thanks</li>
</ol>

onyekaokonji: <p>Hi <a class="mention" href="/u/sbelharbi">@sbelharbi</a> thank you for your points and suggestions, this is over a year old, i hope you see and respond to this. When I implement the 3-steps you highlighted, I notice my training process becomes extremely slow. I cant seem to understand why. Can you check my implementation.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc3097248e454119037aa0352f896a72f798087e.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc3097248e454119037aa0352f896a72f798087e" title="Screenshot 2022-10-04 at 12.52.55"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3097248e454119037aa0352f896a72f798087e_2_674x500.png" alt="Screenshot 2022-10-04 at 12.52.55" data-base62-sha1="qQNRS36ASlI9VL16vcEcJbJaZ0G" width="674" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3097248e454119037aa0352f896a72f798087e_2_674x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3097248e454119037aa0352f896a72f798087e_2_1011x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc3097248e454119037aa0352f896a72f798087e_2_1348x1000.png 2x" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-04 at 12.52.55</span><span class="informations">1730×1282 218 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
`</p>

sbelharbi: <p>hi <a class="mention" href="/u/onyekaokonji">@onyekaokonji</a>,</p>
<ol>
<li>
<p>code: – unrelated to slowness.<br>
i cant tell from your code. also, your dataset class is not clear. but i assume <code>images</code> is a list of tensors.<br>
one main issue: you are zeroing out the gradient every call over a single sample.<br>
you shouldnt do that because you are losing the gradient you just captured. i assume you are trying to accumulate gradient over all samples of a minibatch.<br>
you should zero it out right after the first loop before entering inner loop; so to accumulate it gradient when looping over samples (inner loop).<br>
–<br>
also, the inner loop, you can skip the if test; unless there is a reason to leave it.<br>
you can call step right after you leave inner loop. you update weights after processing all samples of a minibatch.<br>
–<br>
<code>num_accumulation_steps</code> should be computed as the number of samples per minibatch. it could change towsrd last samples. (= <code>len(imgs)</code>).</p>
</li>
<li>
<p>why it is slow?<br>
it is expected to be slower than standard minibatch because now you are processing each sample alone instead of processing all the minibatch at once. here, processing time could be close to <code>x32</code> times processing a single sample. you are losing most of gpu benefit (ability to process several samples at once).<br>
–<br>
my suggestion is to resize images to same size to be able to hold them in single tensor and process them all at once. unless, it is absolutely necessary to maintain image sizes.<br>
–<br>
avoid using .item() unless necessary. it requires gpu-to-cpu transfer (time consuming). you can do all operations over gpu tensors, unless you need cpu values.</p>
</li>
</ol>
<p>thanks</p>

onyekaokonji: <p>Thanks for your response <a class="mention" href="/u/sbelharbi">@sbelharbi</a> if i get your point about the zero_grad() correctly, it should be placed in the outer loop before the inner one right? Pls see screenshot attached.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f4dbe33318d1132df5470ae29b61847511409cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f4dbe33318d1132df5470ae29b61847511409cd" title="Screenshot 2022-10-06 at 02.48.16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4dbe33318d1132df5470ae29b61847511409cd_2_666x500.png" alt="Screenshot 2022-10-06 at 02.48.16" data-base62-sha1="krIQbWpvhq53GAE6cah0WwoRNff" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4dbe33318d1132df5470ae29b61847511409cd_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4dbe33318d1132df5470ae29b61847511409cd_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f4dbe33318d1132df5470ae29b61847511409cd_2_1332x1000.png 2x" data-dominant-color="232322"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2022-10-06 at 02.48.16</span><span class="informations">1750×1312 237 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For num_accumulation_steps, I actually want it to be 32, which is why I fixed as so.</p>
<p>And yeah! unfortunately, the images I’m working with need the aspect ratio maintained and unfortunately, their aspect ratio isn’t 1:1 and they aren’t all of equal aspect ratios either, so I’m forced to feed the images singly.</p>

sbelharbi: <p>yeah, now it seems fine.<br>
for <code>num_accumulation_steps</code>, it could be different than 32 for the last samples if your dataset size is not divisible by 32 or any batch size. to be always sure that you are averaging by the right scalar, you can divide by the number of samples in current minibatch.</p>
<p>to speedup, you can use multi-gpus to process a sample per-gpu in parallel (<a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.12.1+cu102 documentation</a>), and deal with gradient accumulation. but you may need several gpus to see significant speedup.</p>
<p>you can crop random patches with same size for training. you may not need to train on full images.<br>
you can keep same size for evaluation, and evaluate sample per sample.</p>
<p>thanks</p>

onyekaokonji: <p>Thanks <a class="mention" href="/u/sbelharbi">@sbelharbi</a> I will look into your suggestions.</p>