How to make __getitem__ more efficient in loading images?

onedeadmatch: <p>Analyzing the training of my model with the PyTorch profiler, I noticed that most of the time is spent by the CPU (unfortunately, the trace function does not show me any results, the tensorboard page remains blank).</p>
<p>I am more or less sure that the overhead is due to data loading. The dataset consists of two folders, namely one containing annotation .json files and the other jpg images. Considering that the annotations can be entirely loaded into RAM, while all the images cannot, how can the data loading be made more efficient?</p>
<pre><code class="lang-auto">   self.loader = torchvision.io.read_image

   def __getitem__(self, index: int) -&gt; Union[torch.Tensor, torch.Tensor, torch.Tensor]:
        actual_index = self.indices[index]
        annotation = self.annotations[actual_index]

        img_path = os.path.join(self.images_path, annotation["filename"])
        img = self.loader(img_path.replace('.json', ".jpg"))

        bboxes = annotation["bboxes"]
        mask = annotation["mask"]

        return img, bboxes, mask
</code></pre>
<p>I tried varying the batch size, but the gpu utilization remains almost the same (low).<br>
i have tried using multiple persistent workers, but the wait time between epochs is quite long.</p>
<p>I wonder if it is possible to pre-fetch images from files during traning, so while the GPU is busy. Or possibly, instead of using the <strong>getitem</strong> function, a method to which you can pass all batch indexes to load them together. In general, if there are more efficient methodologies.</p>

JuanFMontesinos: <p>Save the images as numpy arrays if you have enough space. Also if wanna go crazy, load batches instead of independent images and save them contiguous in memory so that you load whole sections of the disk for each batch.</p>

onedeadmatch: <p>Thank you very much for your reply! I have tried putting all the images into one HDF5 file, and I am still using the getitem function to always pick them up one by one. Unfortunately, the performance remained about the same, possibly the disk could be the bottleneck, it is an SSD. The images are 1080x1920 in size, maybe this could also be an additional bottleneck factor.</p>
<p>When you mean to load a batch of images at once, how do you think it should be done at the implementation level? I had already tried to load the batch of images in the <strong>call</strong> function, but pytorch lightning gives me error saying that the <strong>getitem</strong> function is missing.</p>
<p>Thank you very much again for your time!</p>
<pre><code class="lang-auto">iostat -xm 1
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c844b349c91c4fef0e370275760fbb891cc55ff1.png" data-download-href="https://discuss.pytorch.org/uploads/default/c844b349c91c4fef0e370275760fbb891cc55ff1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c844b349c91c4fef0e370275760fbb891cc55ff1_2_690x181.png" alt="image" data-base62-sha1="szEGoUmwckKkwLhiXzXPRAe0FtD" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c844b349c91c4fef0e370275760fbb891cc55ff1_2_690x181.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c844b349c91c4fef0e370275760fbb891cc55ff1_2_1035x271.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c844b349c91c4fef0e370275760fbb891cc55ff1_2_1380x362.png 2x" data-dominant-color="25244D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1605×422 45.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

JuanFMontesinos: <p>You can disable automatic batching when creating a Pytorch dataloader class.<br>
<a href="https://pytorch.org/docs/stable/data.html#disable-automatic-batching" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#disable-automatic-batching</a></p>
<p>So technically, you could load batches of images that are contiguous in memory. This should help with a bottleneck in your hard disk. But indeed the images are quite large. You might want to distribute them among several hard disks?</p>
<p>If at some point you find the CPU being a bottleneck, you could try nvidia dali</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/dali">
  <header class="source">
      <img src="https://dirms4qsy6412.cloudfront.net/assets/favicon-81bff16cada05fcff11e5711f7e6212bdc2e0a32ee57cd640a8cf66c87a6cbe6.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/dali" target="_blank" rel="noopener nofollow ugc">NVIDIA Developer</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://developer.download.nvidia.com/images/og-default.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://developer.nvidia.com/dali" target="_blank" rel="noopener nofollow ugc">NVIDIA Developer Data Loading Library (DALI)</a></h3>

  <p>An open-source library to accelerate deep learning training and inference.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
It incorporates gpu decoding of jpeg images.</p>