Issue with example stacking in batch in DataLoader

username: huripari96
cooked: <p>Hi all, I am having an issue with the division of my training set in batches using torch.utils.data.DataLoader.<br>
I reported the error message at the end of the thread and attached it as an image.<br>
I am working with a torchvision.datasets.DatasetFolder of numpy arrays of dimension (128,128,3) to train a Generative Adversarial Network with progressive growing training starting from 4x4 resolution. These are the functions I am using to create the dataset and the loader</p>
<p>def npy_loader(path):<br>
sample = np.load(path, allow_pickle=True)<br>
return sample</p>
<p>def get_loader(image_size):<br>
transform = transforms.Compose(<br>
[<br>
CustomNormalization(), # normalization of the inputs<br>
NumpyToTensor3D(), # output tensor is in the shape (C x H x W) so it works with transforms.Resize()<br>
transforms.Resize((image_size, image_size)),<br>
]<br>
)<br>
batch_size = 64<br>
dataset = datasets.DatasetFolder(root=config.DATASET, loader=npy_loader, extensions=‘.npy’, transform=transform)<br>
loader = DataLoader(<br>
dataset,<br>
batch_size=batch_size,<br>
num_workers=4,<br>
pin_memory=True,<br>
)<br>
return loader, dataset</p>
<p>The error happens in the first epoch when arriving at the batch where there is the stack problem because of a different shape in an example. The loop through the batches is defined in the following way:</p>
<p>loop = tqdm(loader, leave=True)<br>
for batch_idx, (real, _) in enumerate(loop):</p>
<p>As you can see there are 2 custom-made transformation classes, I tried running the code without them and I got the same error, so apparently, they are not the problem. I also tried running the code without the transforms.Resize() class and I still got the error.</p>
<p>I also tried to change the batch size to 32 and 128 but still got the error and always at the same example of the training set (n. 403756). I have checked the training set several times and the original shape of each file is correct.<br>
If I set the batch size to 1 the error does not happen anymore, so I am thinking that probably there is some mistake in the definition of the DataLoader so that the stacking of the examples in batches fails because of dimension.<br>
Also, in this case, I am running the training with approximately 1.600.000 examples in the training set but if I reduce the training set to a smaller sample (around 70.000 examples) the training is performed well and I don’t get the error. Could the dataset dimension be a problem? It does not seem possible to me.</p>
<p>Has anyone else experienced a similar problem? Do you have any suggestions?<br>
Thanks in advance for your help</p>
<p>Traceback (most recent call last):<br>
File “/home/b/b382153/PrOGAN_pytorch/train.py”, line 197, in <br>
main ()<br>
File " /home/b/b382153/ProGAN_pytorch/train.py", line 174, in main<br>
tensorboard_step, alpha = train_fn(<br>
File “/home/b/b382153/PrOGAN_pytorch/train.py”, line 70, in train_fn<br>
for batch_idx, (real, -) in enumerate (loop):<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/tqdm/std.py”,line1195,in <em>iter.</em> for obj in iterable:<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/dataloader.py”,line628,in<br>
-_next<br>
data = self._next_data()<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/dataloader.py”,line1333,in_next_data return self._process_data(data)<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/dataloader.py”,line1359,in_process_data data.reraise ()<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/_utils.py”,line543,inreraise raise exception<br>
<strong>RuntimeError: Caught RuntimeError in DataLoader worker process 0.</strong><br>
Original Traceback (most recent call last):<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/1ib/python3.10/site-packages/torch/utils/data/_utils/worker.py”,line302,in_worker_100p<br>
data = fetcher. fetch (index)<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py”,line61,infetch return self.collate_fn (data)<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”,line265,indefault_collate return collate (batch, collate_fn_map=default_collate_fn_map)<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”,line143,incollate return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/collate•py”,line143,in<br>
return [collate(samples, collate_fn_map=collate_fn_map) for samples in transposed] # Backwards compatibility.<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-×86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”,line120,incollate return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)<br>
File “/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-wuuo72/lib/python3.10/site-packages/torch/utils/data/_utils/collate.py”,line163,incollate_tensor_fn return torch. stack(batch, 0, out=out)<br>
<strong>RuntimeError: stack expects each tensor to be equal size, but got [3, 4, 4] at entry 0 and [481, 128, 4, 4] at entry 44</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c1cc5e0f033e26e233ba15f2d71f029f3b01215.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c1cc5e0f033e26e233ba15f2d71f029f3b01215" title="Screenshot 2023-03-21 at 12.39.34"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c1cc5e0f033e26e233ba15f2d71f029f3b01215.png" alt="Screenshot 2023-03-21 at 12.39.34" data-base62-sha1="d8RzwSXhtSdqOCEzBq6kXW3zj4V" width="690" height="300" data-dominant-color="506ECA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-03-21 at 12.39.34</span><span class="informations">1154×503 20.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Do you recognize the shapes reported in the error message and if so which one would be expected?</p>
<aside class="quote no-group" data-username="huripari96" data-post="1" data-topic="175780">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/huripari96/48/58755_2.png" class="avatar"> huripari96:</div>
<blockquote>
<p>RuntimeError: stack expects each tensor to be equal size, but got [3, 4, 4] at entry 0 and [481, 128, 4, 4] at entry 44</p>
</blockquote>
</aside>
<p>Assuming your <code>Dataset.__getitem__</code> returns an image tensor I would guess the first one would be valid even though the spatial size of <code>4x4</code> seems to be quite small.<br>
What is each sample supposed to return, in particular number of channels and spatial size?</p>

username: huripari96
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for replying.<br>
Yes, the first shape [3, 4, 4] is the correct one. The 4x4 is indeed small but it is because of the way the progressive growing during training is done. It starts from a low resolution of the inputs and progressively shifts to higher resolutions, so there’s a 4x4 phase of training, then an 8x8, then 16x16 and so on until 128x128. This method gives different advantages during the training phase.<br>
Each sample is supposed to be of the shape [3, 4, 4] so for a batch size of 32 the shape of the batches will be [32, 3, 4, 4].</p>

username: ptrblck
cooked: <p>Thanks for the description. Could you add debug transformations showing the shape of the input to your custom transforms to narrow down if any of your transforms is changing the shape?<br>
Something like this might work:</p>
<pre><code class="lang-python">class PrintTransform:
    def __call__(self, x):
        if isinstance(x, PIL.Image.Image):
            print(x.size)
        elif isinstance(x, torch.Tensor) or isinstance(x, np.ndarray):
            print(x.shape)
        return x


transform = transforms.Compose([
    PrintTransform(),
    transforms.ToTensor(),
    transforms.Resize((224, 224)),
    PrintTransform(),
    transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5)),
    PrintTransform()
])

img = transforms.ToPILImage()(torch.randint(0, 256, (3, 256, 256)).byte())
out = transform(img)
</code></pre>
<p>but you might need to add more checks in case you are using other input classes.</p>

username: huripari96
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> I ran the code removing all the transformations and the error keeps appearing so it seems like they do not cause it. Could it be something with the definition of the dataset with datasets.DatasetFolder, I debugged the code more and the unwanted change of shape happens there.</p>

username: ptrblck
cooked: <p>Yes, this would be my next suggestion and you could check the shape of each sample returned by your <code>Dataset</code>. If needed, you could also create a custom <code>Dataset</code> deriving from <code>ImageFolder</code> which would allow you to return each sample path with the corresponding data or you could also try to add debugging <code>print</code> statements directly into the <code>ImageFolder</code> class by manipulating the Python script. You can check the location of this file via <code>print(torch.__path__)</code> which should point to the root folder where your current PyTorch binary is installed.</p>

username: huripari96
cooked: <p>Okay, thank you very much for the help. So, despite working with numpy arrays you still suggested using <code>ImageFolder</code> rather than <code>DatasetFolder</code>?</p>

username: ptrblck
cooked: <p>I don’t think it would make a difference since <code>ImageFolder</code> would additionally create the <code>self.imgs</code> attribute as seen <a href="https://github.com/pytorch/vision/blob/0387b8821d67ca62d57e3b228ade45371c0af79d/torchvision/datasets/folder.py#L271-L317">here</a>, so you can also use the <code>DatasetFolder</code> class directly.</p>