Is there any way in which we can speed up the dataloader

username: aquorio15
cooked: <p>I am trying to build a model which will take two arrays from the dataloader. But the problem is the dataloader is extremely slow. To iterate over the entire sample it took around 3-4 hrs on a A100 GPU with num_workers=32. Is there any way in which I can speed up the dataloader? I have tried setting pin_memory=True, which is actually making it slower.<br>
This is my implementation</p>
<pre><code class="lang-auto">class VQADataset(torch.utils.data.Dataset):
    ans_vocab = {}
    
    def __init__(self, data_dir, qafile, img_dir, phase, img_def='images_resnet', raw_images=True, raw_audio=True, lang='French'):
        self.data_dir = data_dir 
        if phase == 'train':
            self.examples = open(os.path.join(data_dir, qafile), 'r').readlines()
        else:
            self.examples = open(os.path.join(data_dir, qafile), 'r').readlines()
        if phase == 'train':
            self.load_vocab(data_dir)
        self.img_dir = img_dir
        self.phase = phase
        self.raw_images = raw_images
        self.raw_audio = raw_audio
        self.lang = lang
        self.phase = phase
        self.unk_emb = 8000
        self.img_def =img_def
        
    def load_vocab(self, data_dir):
        ans_vocab_file = os.path.join(data_dir, 'ans_itos.tsv')
        for line in open(ans_vocab_file):
            parts = line.split('\t')
            VQADataset.ans_vocab[parts[0]] = parts[1]
    
    def __len__(self):
        return len(self.examples)

    def __getitem__(self, idx):
        audio_id, ques_id, ques, imgid, ans = self.examples[idx].strip().split('\t')
int(imgid)))
        img = np.load('{0}/features/{2}/{4}/COCO_{1}_{3:012d}.npy'.format(self.data_dir, self.img_dir, self.phase, int(imgid), self.img_def)).squeeze()
        audio_features = np.load(os.path.join(self.data_dir, 'features', 'w2v2_'+self.phase, self.lang, f"{self.lang}_{audio_id}.npy")).squeeze()
            
        return ques_id, ques, torch.tensor(audio_features), torch.tensor(img), imgid, int(ans)
</code></pre>
<p>My train dataset contains around 440000 samples and validation set contains around 200000 samples</p>
<pre><code class="lang-auto">train_dataloader = DataLoader(training_data, batch_size=256, num_workers=32)
</code></pre>
<p>Thank You</p>

username: alepistola
cooked: <p>I had and am having the same problem as you, using 2048x2048 images on Google Colab and analyzing the training process I noticed that the bottleneck is actually fetching the data from GDrive.</p>
<p>Things I have adopted temporarily are:</p>
<ul>
<li>set the device globally :</li>
</ul>
<pre><code class="lang-auto">device = "cuda" if torch.cuda.is_available() else "cpu"
torch.device(device)
</code></pre>
<ul>
<li>follow this rule to calculate the num_workers:<br>
<code>NUM_WORKERS = 4 * os.cpu_count()</code></li>
</ul>
<p>I believe, however, that to get maximum performance, you need to load all the data into memory beforehand, but I couldn’t find anything online.<br>
To speed up loading, an HDF5 dataset should be used and then a data loader wrapper created.<br>
Here is some useful information:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/mvsjober/pytorch-hdf5">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/mvsjober/pytorch-hdf5" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/697109f3a592537fef6864e578c99432c541ed75e9a87a23e299eac6b08e34a7/mvsjober/pytorch-hdf5" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/mvsjober/pytorch-hdf5" target="_blank" rel="noopener nofollow ugc">GitHub - mvsjober/pytorch-hdf5</a></h3>

  <p>Contribute to mvsjober/pytorch-hdf5 development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="quote" data-post="15" data-topic="25643">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/carsten_ditzel/48/8251_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-when-num-worker-0-there-is-bug/25643/15?fbclid=IwAR2jFrRkKXv4PL9urrZeiHT_a3eEn7eZDWjUaQ-zcLP6BRtMO7e0nMgwlKU">DataLoader, when num_worker &gt;0, there is bug</a> 
  </div>
  <blockquote>
    so then it seems to be a problem relating to hdf5 in general in combination with multirpocessing rather than with Pytorch itself? If so, it would be interesting so see, how this issue evolves when using the C++ frontend
  </blockquote>
</aside>