How to speed up training for large dataset

username: alexcruz0202
cooked: <p>Hello, everyone.<br>
I have large dataset with 3 million images for face related project.<br>
Torchvision Dataloader has been used for training. However training is very slow.<br>
So I have plan to use HDF5 or LMDB format dataset.<br>
What do you upvote in them?</p>

username: faizan_shaikh
cooked: <p>Hi <a class="mention" href="/u/alexcruz0202">@alexcruz0202</a>,</p>
<p>Could you maybe share the code you are using to load the data and train the model, so that people in the community might give suggestions on how to make the code more efficient and faster perhaps?</p>
<p>Hypothetically speaking, training might be slow because</p>
<ol>
<li>The data loading / preprocessing pipeline is not optimized for execution</li>
<li>DL model isn’t being trained on an efficient hardware such as a GPU</li>
</ol>
<p>If you could give more info around your overall pipeline, you would get an appropriate help from the community</p>

username: alexcruz0202
cooked: <p><a class="mention" href="/u/faizan_shaikh">@faizan_shaikh</a>. Sorry for late response.<br>
This is my codesnipet for dataloader.</p>
<pre><code class="lang-auto">class TrainDataset(Dataset):
    def __init__(self, root="", live_sub_dirs=[], fake_sub_dirs=[]):
        self.live_sub_dirs = live_sub_dirs
        self.fake_sub_dirs = fake_sub_dirs

        self.pos_filelist = get_total_list(root, live_sub_dirs)
        self.neg_filelist = get_total_list(root, fake_sub_dirs)

        count = len(self.neg_filelist) // len(self.pos_filelist)
        self.total_filelist = np.asarray(self.pos_filelist * count + self.neg_filelist, dtype=np.str)
        np.random.shuffle(self.total_filelist)

        self.transform = transforms.Compose([
            transforms.RandomCrop(128),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor()
        ])
        self.aug = strong_aug(0.5)

    def __getitem__(self, idx):
        img_path = self.total_filelist[idx]
        img = Image.open(img_path).convert("RGB")
        img = self.aug(image=np.array(img))["image"]
        img = self.transform(Image.fromarray(img))

        target = 0 if str('fake') in img_path.lower() else 1

        return img, torch.tensor(target, dtype=torch.long)
    
    def __len__(self):
        return len(self.total_filelist)
</code></pre>

username: faizan_shaikh
cooked: <aside class="quote no-group" data-username="alexcruz0202" data-post="3" data-topic="100452">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3ab097/48.png" class="avatar"> alexcruz0202:</div>
<blockquote>
<pre><code class="lang-auto">img = Image.open(img_path).convert("RGB")
img = self.aug(image=np.array(img))["image"]
img = self.transform(Image.fromarray(img))
</code></pre>
</blockquote>
</aside>
<p>Out of the overall code, this part seems much more computationally expensive, and could be made more efficient. If you don’t mind, could you clarify a few things?</p>
<ol>
<li>Is it necessary to convert the images to RBG? If yes, you can probably do it beforehand.</li>
<li>What is the use of <code>strong_aug</code> function? Can’t it incorporated in self.transform? Also, you could use the library “albumentations” to make faster data augmentations on the fly</li>
<li>Conversion of PIL image to numpy array and back seems counter intuitive, if they are a part of same Data augmentation pipeline. You could make it more seemless somehow</li>
</ol>
<p>Along with this, the part where you do random shuffling isn’t required if you rely on pytorch DataLoader’s “shuffle” argument</p>
<p>Hope this helps</p>

username: alexcruz0202
cooked: <p>I’m trying to check every lines of my dataloader.<br>
Thank you for your suggestion.</p>