How to speed up the data loader

Hou_Qiqi: <p>Hi</p>
<p>I want to know how to speed up the dataloader. I am using torch.utils.data.DataLoader(8 workers) to train resnet18 on my own dataset. My environment is Ubuntu 16.04, 3 * Titan Xp, SSD 1T.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Epoch: [1079][0/232]</th>
<th>Time 5.149 (5.149)</th>
<th>Data 5.056 (5.056)</th>
<th>Loss 0.0648 (0.0648)</th>
<th>Prec@1 98.047 (98.047)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Epoch: [1079][10/232]</td>
<td>Time 0.208 (0.778)</td>
<td>Data 0.001 (0.610)</td>
<td>Loss 0.0904 (0.0833)</td>
<td>Prec@1 95.312 (96.165)</td>
</tr>
<tr>
<td>Epoch: [1079][20/232]</td>
<td>Time 0.206 (0.599)</td>
<td>Data 0.001 (0.421)</td>
<td>Loss 0.0771 (0.0877)</td>
<td>Prec@1 96.484 (95.964)</td>
</tr>
<tr>
<td>Epoch: [1079][30/232]</td>
<td>Time 0.208 (0.529)</td>
<td>Data 0.074 (0.353)</td>
<td>Loss 0.0681 (0.0886)</td>
<td>Prec@1 98.047 (96.006)</td>
</tr>
<tr>
<td>Epoch: [1079][40/232]</td>
<td>Time 1.799 (0.538)</td>
<td>Data 1.697 (0.376)</td>
<td>Loss 0.0647 (0.0860)</td>
<td>Prec@1 96.484 (96.103)</td>
</tr>
<tr>
<td>Epoch: [1079][50/232]</td>
<td>Time 0.201 (0.516)</td>
<td>Data 0.001 (0.360)</td>
<td>Loss 0.0578 (0.0836)</td>
<td>Prec@1 96.484 (96.186)</td>
</tr>
<tr>
<td>Epoch: [1079][60/232]</td>
<td>Time 0.443 (0.494)</td>
<td>Data 0.328 (0.340)</td>
<td>Loss 0.0602 (0.0835)</td>
<td>Prec@1 97.656 (96.203)</td>
</tr>
<tr>
<td>Epoch: [1079][70/232]</td>
<td>Time 0.201 (0.483)</td>
<td>Data 0.001 (0.334)</td>
<td>Loss 0.1140 (0.0858)</td>
<td>Prec@1 94.922 (96.121)</td>
</tr>
<tr>
<td>Epoch: [1079][80/232]</td>
<td>Time 0.616 (0.489)</td>
<td>Data 0.521 (0.343)</td>
<td>Loss 0.0932 (0.0857)</td>
<td>Prec@1 95.703 (96.161)</td>
</tr>
<tr>
<td>Epoch: [1079][90/232]</td>
<td>Time 0.200 (0.485)</td>
<td>Data 0.001 (0.341)</td>
<td>Loss 0.0596 (0.0847)</td>
<td>Prec@1 98.047 (96.175)</td>
</tr>
<tr>
<td>Epoch: [1079][100/232]</td>
<td>Time 0.362 (0.477)</td>
<td>Data 0.269 (0.334)</td>
<td>Loss 0.0877 (0.0854)</td>
<td>Prec@1 94.531 (96.117)</td>
</tr>
</tbody>
</table>
</div><p>The log shows that the dataloader takes at least 50% time of the training process.  So I want to speed up the training process by reducing the time for dataloader.</p>
<p>I analyses the time for the datalayer <em>get_item</em>()</p>
<p>total time: 0.02<br>
load img time: 0.0140, 78.17%<br>
random crop and resize time: 0.0001, 0.68%<br>
random flip time: 0.0001, 0.40%<br>
other time:  22.36%</p>
<p>It shows that the reading image takes the most of time (I read 3 images in this section). I uses the codes from torch.utils.data.Dataset</p>
<p>def pil_loader(path):<br>
# open path as file to avoid ResourceWarning (<a href="https://github.com/python-pillow/Pillow/issues/835" rel="nofollow noopener">https://github.com/python-pillow/Pillow/issues/835</a>)<br>
with open(path, ‘rb’) as f:<br>
with Image.open(f) as img:<br>
return img.convert(‘RGB’)</p>
<p>Therefore, I tried <a href="https://discuss.pytorch.org/t/rookie-ask-how-to-speed-up-the-loading-speed-in-pytorch/4714/4">Rookie ask: how to speed up the loading speed in pytorch</a> . I save the images as strings with pickle in lmdb. Then I load them back. I found that it doesn’t speed up too much. Maybe pickle.loads() still costs too much time.</p>
<p>Now I have no idea to speed up the dataloader.  Any hints will help me much.</p>
<p>Thanks.</p>

jpeg729: <p>A database is a good optimisation when your data is a huge text file, but for images stored in individual files it is probably overkill and will add some unnecessary overhead.</p>
<p>Another point to consider is that <code>pickle</code> might be a little slow to save/load pytorch Tensors.</p>
<p>As you say, the image decoding seems to take most of the time, so I would suggest writing a small script that loads each <code>image_file.jpg</code> into a torch Tensor, and then uses <code>torch.save</code> to save the result back into a file named <code>image_file.pt</code> or something like that.</p>
<p>Then you would need to modify your loader to <code>torch.load</code> the .pt files instead of loading the image files.</p>

Andres_Asensio_Ramos: <p>In my experience, I would first build an HDF5 file with all your images, which you can build easily following the documentation of <code>h5py</code> on <a href="http://docs.h5py.org/en/latest/" rel="nofollow noopener">http://docs.h5py.org/en/latest/</a>. During training, build a class inheriting from <code>Dataset</code> which returns your images. Something along this line:</p>
<pre><code>class dataset_h5(torch.utils.data.Dataset):
def __init__(self, in_file):
    super(dataset_h5, self).__init__()

    self.file = h5py.File(in_file, 'r')
    self.n_images, self.nx, self.ny = self.file['images'].shape

def __getitem__(self, index):
    input = self.file['images'][index,:,:]
    return input.astype('float32')

def __len__(self):
    return self.n_images
</code></pre>
<p>Then you can build your loader with:</p>
<pre><code>self.train_loader = torch.utils.data.DataLoader(dataset_h5(train_file),
                                                    batch_size=16, shuffle=True)
</code></pre>
<p>My experience with this approach has been quite positive, with the GPU always at 100% even though I’m loading pretty heavy images.</p>

Hou_Qiqi: <p>Got it. Thank you~</p>
<p>Once I save preprocessed images as .pt format. It will load the tensor directly. How can I do random crop and resize? Need I convert them back to PIL.Image?</p>
<p>Thanks</p>

jpeg729: <p>I hadn’t thought of that problem… I can think of two approaches but I can’t tell you which will work the fastest.</p>
<ol>
<li>Convert each image into <code>.bmp</code> format instead of <code>.jpg</code>. Then use your original loader to load the <code>.bmp</code> format which will decompress much faster than <code>.jpg</code>.</li>
<li>Use <code>torchvision.transforms.ToPILImage</code> which I think should run pretty fast.</li>
</ol>

Hou_Qiqi: <p>Thank you.</p>
<p>I have implemented dataset_h5. It runs quite well when I set the “num_workers” as 1 or 2. I meet some problem when I set “num_worker” bigger than 2. It seems related to the h5py version. My h5py now is 2.7.1.</p>
<blockquote>
<p>File “/home/titan/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 281, in <strong>next</strong><br>
return self._process_next_batch(batch)<br>
File “/home/titan/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 301, in _process_next_batch<br>
raise batch.exc_type(batch.exc_msg)<br>
KeyError: ‘Traceback (most recent call last):\n  File “/home/titan/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 55, in _worker_loop\n    samples = collate_fn([dataset[i] for i in batch_indices])\n  File “/home/titan/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 55, in \n    samples = collate_fn([dataset[i] for i in batch_indices])\n  File “/home/titan/code/res-pytorch/AdobeData/AdobeData.py”, line 595, in <strong>getitem</strong>\n    fgimg = self.fgfile['img'][index, …]\n  File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper\n  File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper\n  File “/home/titan/anaconda3/lib/python3.6/site-packages/h5py/_hl/group.py”, line 167, in <strong>getitem</strong>\n    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)\n  File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper\n  File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper\n  File “h5py/h5o.pyx”, line 190, in h5py.h5o.open\nKeyError: 'Unable to open object (bad object header version number)'\n’</p>
</blockquote>
<p>My codes are like</p>
<blockquote>
<p>class AdobePatchDataHDF5(data.Dataset):</p>
<pre><code>def __init__(self, root, cropsize = 256, outputsize = 256):

    fgfile = h5py.File(root, 'r')

    self.root = root
    self.fgfile = fgfile
    self.cropsize = cropsize
    self.outputsize = outputsize


def __getitem__(self, index):
    # read image
    fgimg = self.fgfile['img'][index, ...]

    # random crop and resize, random flip with cv2

    # toTensors
    fgimg = fgimg.astype(np.float32) / 255.0 
    fgimg = torch.from_numpy(fg.transpose((2, 0, 1)))

    # norm [0, 1] to [-1, 1]

    return fgimg, label

def __len__(self):
    return self.fgfile['img'].shape[0]
</code></pre>
</blockquote>
<p>My train loader is like</p>
<pre><code class="lang-auto">    train_loader = torch.utils.data.DataLoader(
        AdobePatchDataHDF5(path), batch_size=256, shuffle= True
        num_workers=8, pin_memory=True, sampler=None)
</code></pre>

Andres_Asensio_Ramos: <p>What if you use the <code>swmr=True</code> (single write multiple reads) option when opening the HDF5 file?</p>

Hou_Qiqi: <p>Yeah. With swmr=True, it still gets the same bug.</p>

Andres_Asensio_Ramos: <p>It looks like HDF5 has some concurrency issues. My suggestion of using it is probably not appropriate when you use several workers. I often use one worker because my networks are computationally heavy and I’m not limited by the data iterator. Perhaps you should try other approaches like <code>zarr</code> (<a href="http://zarr.readthedocs.io/en/stable/" rel="nofollow noopener">http://zarr.readthedocs.io/en/stable/</a>) which have been designed to be thread-safe.</p>

melgor: <p>I’m not sure if the problem is same like my, but I’ve problem with reading a very big images (3k by 3k).<br>
If this is your case, it is my advice:</p>
<ol>
<li>Can you divide images to smaller sub-region, ex. 1k vs 1k? If yes, just make some crops of images to make it smaller. Then reading speed should be much faster.</li>
<li>You can use <a href="https://github.com/ajkxyz/jpeg4py" rel="nofollow noopener">jpeg4py</a>, library dedicated to encode big jpeg files much faster than PIL. Just  read image using this library, then transform it to PIL.</li>
<li>The fastest available option, which I found is using <strong>jpeg4py</strong> library together with OpenCV data-augmentation (so no PIL image). I used OpenCV technique from this <a href="https://github.com/pytorch/vision/pull/34" rel="nofollow noopener">pull request.</a>
</li>
</ol>

Hou_Qiqi: <p>After reading the some codes of torch.utils.data.dataloader, I find it does not work like caffe, which prefetches next batch data during the GPUs are working. I find a blog which tries to do it. I will try it.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="https://sagivtech.com/2017/09/19/optimizing-pytorch-training-code/" target="_blank" rel="nofollow noopener">Sagivtech</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/383;"><img src="https://sagivtech.com/wp-content/uploads/2017/09/PyTorch.png" class="thumbnail"></div>

<h3><a href="https://sagivtech.com/2017/09/19/optimizing-pytorch-training-code/" target="_blank" rel="nofollow noopener">Optimizing PyTorch training code</a></h3>

<p>Ben Levy and Jacob Gildenblat, SagivTech    

PyTorch is an incredible Deep Learning Python framework. It makes prototyping and debugging deep learning algorithms easier, and has great support for multi gpu training.
However, as always with...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>######################################################</p>
<p>After I read the blog, I made a mistake which the dataloader tries to prefetch the next batch data . But I find that it can <strong>NOT make full use of CPUs (it shows that it only uses about 60% ability of CPU)</strong>. In the blog it shows that the data precoessing takes less than 18% of time. Actually, if it could fully achieve its target and made full use of CPUs and the disk is fast enough, it should be near 0%, not 18%.</p>

taiky: <p><a class="mention" href="/u/hou_qiqi">@Hou_Qiqi</a><br>
have you finally fixed this annoying problem of “in h5py.h5o.open\nKeyError: ‘Unable to open object (bad object header version number)’\n’” ?</p>
<p>I have the same issue with you when I set num_workers greater than 2.</p>

Hou_Qiqi: <p><a class="mention" href="/u/taiky">@taiky</a><br>
Sorry, I don’t fix this bug. Now I write another process that prepares data on the fly when the gpus are runing. Hope that can help you.</p>

anton: <p>Check out a potential solution to HDF5 dataloader concurrency issues: <a href="https://stackoverflow.com/a/52249344/411907" rel="nofollow noopener">https://stackoverflow.com/a/52249344/411907</a></p>

yaysummeriscoming: <p>For anyone reading this, Nvidia DALI is a great solution:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/NVIDIA/DALI" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/1728152?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/NVIDIA/DALI" target="_blank" rel="nofollow noopener">NVIDIA/DALI</a></h3>

<p>A library containing both highly optimized building blocks and an execution engine for data pre-processing in deep learning applications - NVIDIA/DALI</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>It’s got simple to use Pytorch integration.</p>
<p>I was running into the same problems with the pytorch dataloader.  On ImageNet, I couldn’t seem to get above about 250 images/sec.  On a Google cloud instance with 12 cores &amp; a V100, I could get just over 2000 images/sec with DALI.  However in cases where the dataloader isn’t the bottleneck, I found that using DALI would impact performance 5-10%.  This makes sense I think, as you’re using the GPU to some of the decoding &amp; preprocessing</p>
<p>Edit: Dali also has a CPU only mode, meaning no GPU performance hit</p>

jasrk11: <p><a class="mention" href="/u/hou_qiqi">@Hou_Qiqi</a> Were you able to speed up your dataloader? Did you try preparing the data while GPUs are running?</p>

pb2: <p>Just found this thread a few days ago and implemented NVIDIA DALI to load my data while doing some transfer learning with Alexnet.</p>
<p>On an AWS p2.x8large instance (8 Tesla K80 GPUs), using DALI speeds up my epoch time from 480 s / epoch to 10 s / epoch. No need for any code that explicitly prepares data while the GPUs are running. One important thing to note is that if you’re using a DALI dataloader from an external source (i.e. if you have your image classes grouped by folder), you have to manually reset the dataloader using loader_name.reset() at the end of every training epoch. That’s not how the pytorch dataloader works, so it took me a while to realize that was what was going on here.</p>
<p>The only irritating thing I’ve found about DALI is that there is no immediately obvious way (to me, anyway) to convert pixel values from uint8 with a 0-255 range to float with a 0-1 range, which is needed for transfer learning with pytorch’s pretrained models. Dividing by 255.0 within a pipeline runs into data type mismatch, the ops.Cast() function only converts to float but doesn’t rescale, and all the various flavors of normalize functions don’t allow for it either. The only way I was able to do it was by manually scaling the mean and std values given by pytorch.</p>
<p>Other than that, agree that the pytorch integration is simple and fairly clean.</p>

KCHENPENGFEI: <p>Hi, I got the same problem. You can try to clear your cache to solve the problem and try many times.</p>

okman: <p>Hi, I also meet with this problerm, do you have sovle it ?</p>
<p>I want to train resnet50  on ImageNet, but the data loading is a bottleneck.</p>
<p>do you try the NVIDIA-DALI,   <a href="https://github.com/NVIDIA/DALI/tree/master/docs/examples/pytorch/resnet50" rel="nofollow noopener">resnet50-by-dali</a>  ??</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/triumph.png?v=6" title=":triumph:" class="emoji" alt=":triumph:"><img src="https://discuss.pytorch.org/images/emoji/apple/triumph.png?v=6" title=":triumph:" class="emoji" alt=":triumph:"><img src="https://discuss.pytorch.org/images/emoji/apple/triumph.png?v=6" title=":triumph:" class="emoji" alt=":triumph:"><img src="https://discuss.pytorch.org/images/emoji/apple/triumph.png?v=6" title=":triumph:" class="emoji" alt=":triumph:"></p>

aurooj: <p>I am facing the similar problem. However, I am more interested in knowing how to make my dataloader efficient.<br>
Ideally, I want to read multiple temporal blocks for a video, perform transformations like random crop, rescale, extract I3D features for them, concat them in a tensor and return with other information like text embeddings for question, multiple answers.</p>
<p>For now, I am preprocessing my videos and extracting features beforehand, which does not allow me to use multiple random crops and not a good way. The video frames are 60GB size in total. Any tips?</p>
<p>I am using Ubuntu 16.04, 24gb RAM, 245gb/959gb disk space is free right now, 1 Titan Xp GPU, Python 3.6.4, Pytorch 0.4.1, cuda 8.0.61, cudnn 7102.<br>
Dataset is on local disk.</p>