Faster the training of DL models by pre-transformation

Giang_Nguyen: <p>If you ever trained a model, you sometimes can see DataLoader (e.g. I am using pytorch Dataloader) could be the bottleneck because everytime you get an training sample from the dataset, the data transformation will be performed on-the-fly. Here I take the <strong>getitem</strong> func of DatasetFolder from <code>torchvision.datasets</code> as an example.</p>
<pre><code>    path, target = self.samples[index]
    sample = self.loader(path)
    if self.transform is not None:
        sample = self.transform(sample)
    if self.target_transform is not None:
        target = self.target_transform(target)

    return sample, target
</code></pre>
<p>I wonder can we pre-processed the images (e.g. ImageNet) in advance to tensors and save to disk. Then we modify the <code>__getitem__</code> function to get these tensors directly from disk. How efficient is this approach? Anyone has tried this solution before?</p>
<p>I think that maybe the loading from disk will burden and likely become a new bottleneck (instead of data transform we have before). Another thing is the size, for example, one ImageNet image takes 74 MB when being saved as tensors using standard transformation:</p>
<pre data-code-wrap="python3"><code class="lang-nohighlight">transforms.Compose([
                transforms.Resize(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
            ])
</code></pre>

tom: <aside class="quote no-group" data-username="Giang_Nguyen" data-post="1" data-topic="159908">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giang_nguyen/48/15828_2.png" class="avatar"> Giang_Nguyen:</div>
<blockquote>
<p>hink that maybe the loading from disk will burden and likely become a new bottleneck (instead of data transform we have before). Another thing is the size, for example, one ImageNet image takes 74 MB when being saved as tensors using standard transformation:</p>
</blockquote>
</aside>
<p>I think you are spot-on here. In my experience, decoding image files often is much faster than than the time it takes to load uncompressed images from disk.</p>
<p>That said, you might loom into pre-resizing to a comon (maybe a bit larger than the final) size, save as images, and just do ToTensor in the dataset and anything else on the GPU. That can give great speedups.</p>
<p>Best regards</p>
<p>Thomas</p>

Giang_Nguyen: <p>Hi Thomas. I would be great if you give me an example of doing anything else on gpu. Did you mean anything in transformation? I suppose you are mentioning ab the normalization, right?</p>
<p>So in this case per you suggested, the standard data transform will only contain ToTensor. Normalize will be called later once we move the tensors to gpu?</p>

tom: <p>So it also does other things, but the grapevine livedemo notebooks from this repo take a model and tune the dataloading:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/t-vi/lod2021">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/t-vi/lod2021" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b4ad8ce02581d94e2ee445523a466ff8428b87eaa74558e8855a0f7061843d1d/t-vi/lod2021" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/t-vi/lod2021" target="_blank" rel="noopener nofollow ugc">GitHub - t-vi/lod2021: PyTorch Tutorial at the LOD2021 conference</a></h3>

  <p>PyTorch Tutorial at the LOD2021 conference. Contribute to t-vi/lod2021 development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>There we end up loading the dataset into memory, which in general cannot be expected to work, but other considerations might apply to your case as well.</p>
<p>Beat regards</p>
<p>Thomas</p>