Dataloader is very slow when using shuffle and becomes normal while shuffle is False

horizon_hardyzl: <p>I’m having an issue while training large amounts of audios(about 2 million wav). Dataloader becomes very slow when shuffle is True .When I set Shuffle False, It becomes normal. I rewrite collate_func which can use torchaudio.load to get waveforms from batch(wav_paths).<br>
I wonder if it’s because many wavs locates different  folders  and it needs take time to find every wav?  what should I do if I want to use Shuffle?  Thanks!</p>

manudwd: <p>You can try</p>
<p><code>train_sampler = torch.utils.data.RandomSampler(train_set)</code></p>
<p>and pass the sampler to the dataloader as</p>
<pre><code class="lang-auto">train_loaders = torch.utils.data.DataLoader(train_set, 
                                                shuffle=False,
                                                sampler = train_sampler,
                                                batch_size=train_batch_size,
                                                num_workers=num_workers)
</code></pre>
<p>If that is still slow you can define a custom psuedo sampler by looking into the PyTorch RandomSampler.</p>

horizon_hardyzl: <p>Thanks for your reply very much ! But unfortunately ,I use RandomSampler and the problem still exists.<br>
Then I random the wav list in my wav_path_file manually,and set the <strong>Shuffle</strong> False , It’s still slow . So I think it’s the problem of the length of wavs . I use <strong>torch.nn.utils.rnn.pad_sequence</strong> to pad the wavs_tensor in collate_func . When <strong>Shuffle</strong> is True , the wavs’s length become so different so the <strong>pad_sequence</strong> has to pad more and cost more time, and it might be the reason ?<br>
More details:<br>
I set <strong>num_workers = 4</strong> and I find the workers pid becomes <strong>D</strong> . They seem to wait for something.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4.png" data-download-href="https://discuss.pytorch.org/uploads/default/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4_2_690x63.png" alt="image" data-base62-sha1="cqwYvwM5Mc2ETJtmkqkxetWFbwg" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4_2_690x63.png, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/5719dfe3f3dd9c085dd0ad0ccebcfbd9f137edb4.png 2x" data-dominant-color="3A3A3A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">865×80 75 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I try to use **ctrl+c" to interrupt the pids , and It says the pids hang to wait for something I don’t know.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89b2e18338873f8d936e3c0da5981575af7552e0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/89b2e18338873f8d936e3c0da5981575af7552e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89b2e18338873f8d936e3c0da5981575af7552e0_2_690x208.jpeg" alt="image" data-base62-sha1="jE8Fax3GfpLIcg2b9v6ol2f2eQM" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89b2e18338873f8d936e3c0da5981575af7552e0_2_690x208.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/9/89b2e18338873f8d936e3c0da5981575af7552e0.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/9/89b2e18338873f8d936e3c0da5981575af7552e0.jpeg 2x" data-dominant-color="141414"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">865×262 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I have searched for similar problems on the Internet but they don’t work . I would appreciate it for your reply! Thanks!</p>