Weird behaviour by dataloader

Prateek_Singh: <p>Dataloader has this weird issue where it loads only up to the batch just below index 400. The check to find errors in the data turned out to be negative and I also tried taking a subset (350 - end, including the index where it gave the error) and it worked just fine this time. I do not understand the issue and would appreciate any help.</p>
<p>But before that, let me explain the scenario a bit better.</p>
<p>The data has around 700 images is saved in the form of an array of dictionaries with the metadata in the following format.</p>
<pre><code class="lang-auto">&gt;&gt; ct_dataset[50]

 {'Folder': '373::CT Thin Plain',
 'Label': 'Normal',
 'Error': 0,
 'Resample': array([[[-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         ...,
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024]]
 
        ...,
 
        [[-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         ...,
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024],
         [-1024, -1024, -1024, ..., -1024, -1024, -1024]]], dtype=int16)}
</code></pre>
<p>The reshape entry has the dimensions of (100, 250, 250) which is uniform across the entire dataset.</p>
<p>But when I run the following command to check the dataset:</p>
<pre><code class="lang-auto">for i, d in enumerate(dataloaders['train']):
    print(i, d)
</code></pre>
<p>I get the following error,</p>
<pre><code class="lang-auto">RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 100 and 0 in dimension 1 at /opt/conda/conda-bld/pytorch_1549635019666/work/aten/src/TH/generic/THTensorMoreMath.cpp:1307
</code></pre>
<p>But the dimension 0 of all images is 100. When I set <code>shuffle=False</code> on the dataloader, I find that it stops just before index 400 of the dataset. But when I subset the dataset from 350-end to replicate the data error, it loads all data properly. This confuses me a lot since if this was a data error, it should have thrown an error on index 50 of the subset.</p>
<p>Hope the issue makes sense, any advice would be well appreciated.</p>

Prateek_Singh: <aside class="quote no-group" data-username="Prateek_Singh" data-post="1" data-topic="48602">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prateek_singh/48/13321_2.png" class="avatar"> Prateek_Singh:</div>
<blockquote>
<p>But the dimension 0 of all images is 100</p>
</blockquote>
</aside>
<p>This should be dimension 1. Sorry for the typo</p>

ptrblck: <p>Could you add a print statement in <code>__getitem__</code> of your <code>Dataset</code> and check the current index?<br>
This could make it easier to find the faulty sample, which is causing this issue.</p>

Prateek_Singh: <p>Got the issue. There was one sample which had (0,250,250) dimensions. Don’t know why it wasn’t captured when I tried printing unique dims for all data.</p>
<p>Anyways, thank you for the guidance. I really appreciate it.</p>
<p>PS: I had to turn num_workers=0 in DataLoader to get better understanding since it made the processing serial.</p>
<p>Thanks again.</p>