How to fix randomness of dataloader in DDP?

username: Yi_wen_Chen
cooked: <p>Hi! I have a question about DistributedSampler.<br>
I’m using DDP and I hope that my data loader can generate precisely the same data pack for each training (but of course different for each GPU).<br>
However, even though I have set the <em>seed</em> and <em>shuffle</em> of DistributedSampler, I find the output data pack is not the same. So what’s the right way to fix the randomness of the data loader?</p>
<pre><code class="lang-auto">    train_loader = DataLoader(
        train_dset,
        batch_size=1,  
        shuffle=False,
        num_workers=8,
        pin_memory=True,
        sampler=DistributedSampler(train_dset, shuffle=True, seed=rank),
    )
    for batch_idx, data_pack in enumerate(train_loader):
        print(rank, batch_idx, data_pack)
</code></pre>

username: H-Huang
cooked: <p>cc <a class="mention" href="/u/ejguan">@ejguan</a> <a class="mention" href="/u/nivek">@nivek</a> for dataloader question.</p>

username: ejguan
cooked: <p>You should set same seed to <code>DistributedSampler</code> to get the get deterministic shuffling behavior.<br>
See document related to <code>seed</code> argument: <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.13 documentation</a></p>

username: Yi_wen_Chen
cooked: <p>Thanks for your reply! I have set the DistributedSampler as below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1ba9a08c5eca4a1827ed53d1d07307eec0f8dc79.png" data-download-href="https://discuss.pytorch.org/uploads/default/1ba9a08c5eca4a1827ed53d1d07307eec0f8dc79" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1ba9a08c5eca4a1827ed53d1d07307eec0f8dc79.png" alt="image" data-base62-sha1="3WIjBnX5mpOoWoLhVDNLO9Auah3" width="690" height="320" data-dominant-color="242423"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">816×379 31.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I found that the randomness of the input index of <strong>getitem</strong> has been fixed. But the output color is not. I found the reason is that I called np.random in <strong>getitem</strong>.<br>
I tried to fix this randomness of numpy by setting the seed of numpy.random. I have tried multiple times and found that:</p>
<ol>
<li>setting numpy.random.seed in <strong>getitem</strong>. Work, Both input index and output data is fixed.</li>
<li>setting  numpy.random.seed right after mp.spawn(). Not work, input index is fixed but output data is not fixed. I think the reason is I called numpy.random to initialize my network before data sampling. But this should not destroy the fixed randomness. I don’t know the reason.</li>
<li>setting numpy.random.seed before mp.spawn(). Not work, input index is fixed but output data is not fixed.</li>
</ol>
<p>So does this mean I should never use numpy.random but use torch.random in DDP? It’s so weird. Thanks!</p>

username: ptrblck
cooked: <p>Check if you would need to re-seed numpy in the <code>worker_init_fn</code> as described <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading">here</a>.</p>

username: Super_Kai
cooked: <p>You can fix the randomness of <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">DataLoader()</a> with <a href="https://pytorch.org/docs/stable/generated/torch.manual_seed.html" rel="noopener nofollow ugc">torch.manual_seed()</a> as shown below:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch.utils.data import DataLoader

torch.manual_seed(42)

my_tensor = torch.tensor([0, 1, 2, 3, 4, 5])

list(DataLoader(dataset=my_tensor, shuffle=True))
# [tensor([4]), tensor([5]), tensor([1]), tensor([0]), tensor([3]), tensor([2])]
</code></pre>