Dataloader speed variation

bma: <p>My dataloader does not seem consistent in speed performance. My data is saved in .h5 formats such as as dataset_sub1.h5, dataset_sub2.h5 … I am using tqdm to show progress. When loading dataset_sub1.h5, it takes 1.88s/it, (1 it corresponds to 200 samples). But for the next file dataset_sub2.h5, it is much slower with 5.38s/it. (The batch size is the same)<br>
I am using single GPU Tesla P40.<br>
I have tried several times, results show that the speed is not consistent, please see the screenshot below.</p>
<h2><a name="my-questions-are-1" class="anchor" href="#my-questions-are-1"></a>My questions are:</h2>
<p>What is causing the variation, data loading or GPU computation?<br>
How to fix the issue?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/6/6dbb2f3f3823dbdd006c0a67c4d3d346cee61753.png" alt="screenshot" data-base62-sha1="fEJ5i4hhLWSaje7z11JN7zUXtCz" width="658" height="198"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b7bba6750fb6b01191a04cb236ba4b9d375629bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7bba6750fb6b01191a04cb236ba4b9d375629bf" title="screenshot"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b7bba6750fb6b01191a04cb236ba4b9d375629bf_2_331x500.png" alt="screenshot" data-base62-sha1="qdnpMuRAMXwPf3vBrP77VtmjrH1" width="331" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b7bba6750fb6b01191a04cb236ba4b9d375629bf_2_331x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b7bba6750fb6b01191a04cb236ba4b9d375629bf_2_496x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b7bba6750fb6b01191a04cb236ba4b9d375629bf_2_662x1000.png 2x" data-dominant-color="404040"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">screenshot</span><span class="informations">670×1010 122 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h2><a name="class-definition-2" class="anchor" href="#class-definition-2"></a>Class definition</h2>
<p>Class data_hdf5(Dataset):</p>
<pre><code>def __init__(self, hdf5_path, transform=None):
    hdf5_file = h5py.File(hdf5_path, mode='r')
    self.hdf5_file = hdf5_file
    self.transform = transform

def __len__(self):
    return self.hdf5_file['filenames'].size

def __getitem__(self, idx):
    gray  = self.hdf5_file['images'][idx]
    image = color.gray2rgb(gray)
    filename = self.hdf5_file['filenames'][idx]

    if self.transform:
        image = self.transform(image)
        image = image.type(torch.FloatTensor)
    else:
        image = torch.from_numpy( gray ).type(torch.FloatTensor)

    label = torch.from_numpy( self.hdf5_file['labels'][idx] ).type(torch.FloatTensor)
    sample = {'image': image, 'label':label}

    return sample
</code></pre>
<h2><a name="main-script-using-dataloader-3" class="anchor" href="#main-script-using-dataloader-3"></a>main script using dataloader</h2>
<p>set_dict = {‘train’: list(range(1,4)))}</p>
<p>hdf5_path = [ ‘data/dataset_sub’+str(i)+‘.h5’ for i in range(1,4) ]<br>
datasets = [ data_hdf5(tmp, transform=data_transforms<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>) for tmp in hdf5_path ]</p>
<p>dataloaders = [ torch.utils.data.DataLoader(tmp, batch_size=batch_size<span class="chcklst-box checked fa fa-check-square-o fa-fw"></span>, shuffle=True, num_workers=1, pin_memory=False) for tmp in datasets ] # a list of dataloaders, each loads a dataset_sub*.h5 file.</p>

zhanghaoinf: <p>Hi, bma,<br>
Have you find solution to this problem ? I also encountered the problem of unstable dataloader speed.</p>

Lizhen_Ji: <p><a class="mention" href="/u/bma">@bma</a> Hi, similar problem here, I have large image classification dataset and make it on h5py, but the training time is getting slower and slower(shuffle=True), have you solved the problem? thanks</p>