Is there a way to fix the random seed of every workers in dataloader?

username: _humandream
cooked: <p>transform = transforms.Compose([<br>
transforms.ColorJitter(0.1, 0.3, 0.3, 0.05),<br>
transforms.RandomHorizontalFlip(),<br>
my_folder.RandomRotate(),<br>
my_folder.RandomVerticalFlip(),<br>
transforms.ToTensor(),<br>
Cutout(n_holes = args.n_holes, length=args.cutout_length)<br>
])</p>
<p>…</p>
<p>train_loader = DataLoader(dataset=train_dataset, shuffle=False,<br>
batch_size=args.batch_size,  num_workers=24)</p>
<p>…</p>
<p>RANDOM_SEED = random.randint(0,1e+5)</p>
<p>torch.manual_seed(RANDOM_SEED)<br>
torch.cuda.manual_seed(RANDOM_SEED)<br>
torch.cuda.manual_seed_all(RANDOM_SEED)<br>
np.random.seed(RANDOM_SEED)<br>
random.seed(RANDOM_SEED)</p>
<p>for batch_idx, (data, _, _,) in enumerate(train_loader) :<br>
x1 = data<br>
print(x1[0])<br>
break</p>
<p>torch.manual_seed(RANDOM_SEED)<br>
torch.cuda.manual_seed(RANDOM_SEED)<br>
torch.cuda.manual_seed_all(RANDOM_SEED)<br>
np.random.seed(RANDOM_SEED)<br>
random.seed(RANDOM_SEED)</p>
<p>for batch_idx, (data, _, _,) in enumerate(train_loader) :<br>
x2 = data<br>
print(x2[0])<br>
break</p>
<p>I’m trying to make some tricky networks, and I need to get exactly the same data with same order twice. x1 and x2 become exactly the same if I set num_workers=0. but otherwise, It isn’t. (But using single worker is not an option for me.)</p>
<p>Is there a way to fix the random seed of whole workers?<br>
Thanks.</p>

username: ptrblck
cooked: <p>You could use the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">worker_init_fn</a> from <code>DataLoader</code> to set the worker’s seed.</p>

username: _humandream
cooked: <p>% It worked. Thanks very much!</p>
<p>Sorry, my mistakes. it still not working.<br>
I tried all methods suggested in here(<a href="https://github.com/pytorch/pytorch/issues/7068" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/7068</a>), but nothing worked.</p>

username: iceflame89
cooked: <p>import torch<br>
torch.manual_seed(0)</p>

username: kirk86
cooked: <p>An even dumber question, is it enough to set seed just in the main file or every file that is being imported in the main file should have their own seed?</p>

username: garibarba
cooked: <p>Hmmm, doesn’t this solution imply the same seed will be used at every epoch?</p>

username: ptrblck
cooked: <p>Yes, the worker seed would be the same, but this would also be the current behavior.<br>
In <a href="https://github.com/pytorch/pytorch/blob/ac79c874cefee2f8bc1605eed9a924d80c0b3542/torch/utils/data/dataloader.py#L715-L720">these lines of code</a> the seed is set as the <code>base_seed + i</code>, where <code>i</code> is the worker id.<br>
Inside the worker, the seed will be used <a href="https://github.com/pytorch/pytorch/blob/ac79c874cefee2f8bc1605eed9a924d80c0b3542/torch/utils/data/_utils/worker.py#L129-L130">here</a>.</p>
<p>Note that this would not force the same ordering of the data, since the sampler won’t use the same seed.<br>
The worker seed should be used for the current worker process, i.e. you could use the <code>worker_init_fn</code> to set the seed for all libraries you are using inside e.g. <code>Dataset.__getitem__</code>, such as numpy etc.</p>

username: garibarba
cooked: <p>Interesting. Thanks for your reply. Doesn’t this mean that the worker will always perform the same sequence of random transforms, each time on different data?</p>
<p>To be clear with what I mean:</p>
<blockquote>
<p>the sampler won’t use the same seed</p>
</blockquote>
<p>means that the sample distribution across workers is randomised.</p>
<p>But at the same time:</p>
<blockquote>
<p>the worker seed would be the same</p>
</blockquote>
<p>means that any random transformations performed by the worker will be the same.</p>
<p>I guess this is not problematic, since workers will rarely be used with a very low number of samples to augment, but it is unexpected.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="garibarba" data-post="8" data-topic="21687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/garibarba/48/3585_2.png" class="avatar"> garibarba:</div>
<blockquote>
<p>means that any random transformations performed by the worker will be the same.</p>
</blockquote>
</aside>
<p>No, that shouldn’t be the case and sorry for the “half-answer”.</p>
<p>The <code>base_seed</code> will be recreated in each call to <code>BaseDataLoaderIter.__init__</code> <a href="https://github.com/pytorch/pytorch/blob/ac79c874cefee2f8bc1605eed9a924d80c0b3542/torch/utils/data/dataloader.py#L341">here</a>, which would be called in each epoch. Once the <code>DataLoader</code> is empty, it will be recreated.</p>
<p>Each worker will thus get a new seed and would use it for the current epoch, such that the random transformation would be pseudo-random again.</p>
<p>You could however use the worker seed to force the same transformation as seen in this small code snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.arange(4).float().view(4, 1, 1, 1).repeat(1, 3, 224, 224)

    def __getitem__(self, index):
        x = self.data[index]
        seed = torch.utils.data.get_worker_info().seed
        print('worker seed {}'.format(seed))
        random.seed(seed)
        # apply random transform
        i, j, h, w = transforms.RandomCrop.get_params(
            x, output_size=(200, 200))
        print('sampled crop indices {}'.format((i, j, h, w)))
        return x

    def __len__(self):
        return len(self.data)


if __name__=='__main__':

    dataset = MyDataset()

    loader = DataLoader(dataset, batch_size=2, num_workers=2)
    for epoch in range(2):
        print('epoch {}'.format(epoch))
        for data in loader:
            pass

&gt; epoch 0
worker seed 7627862169094115211
sampled crop indices (16, 18, 200, 200)
worker seed 7627862169094115211
sampled crop indices (16, 18, 200, 200)
worker seed 7627862169094115212
sampled crop indices (11, 17, 200, 200)
worker seed 7627862169094115212
sampled crop indices (11, 17, 200, 200)
epoch 1
worker seed 3323761461075438105
sampled crop indices (23, 23, 200, 200)
worker seed 3323761461075438105
sampled crop indices (23, 23, 200, 200)
worker seed 3323761461075438104
sampled crop indices (4, 2, 200, 200)
worker seed 3323761461075438104
sampled crop indices (4, 2, 200, 200)
</code></pre>
<p>As you can see, each worker will sample its own random numbers in the current epoch.</p>
<p>Of course you could also skip the base seed completely by providing your own login in the <code>worker_init_fn</code>.</p>
<p>Let me know, if that clears things up.</p>

username: garibarba
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="21687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Each worker will thus get a new seed and would use it for the current epoch, such that the random transformation would be pseudo-random again</p>
</blockquote>
</aside>
<p>Thanks for your answer, now I think it’s clear.</p>
<p>The only thing bugging me now is that then, given how the seed is used</p>
<aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="21687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>the seed will be used <a href="https://github.com/pytorch/pytorch/blob/ac79c874cefee2f8bc1605eed9a924d80c0b3542/torch/utils/data/_utils/worker.py#L129-L130" rel="noopener nofollow ugc">here </a>.</p>
</blockquote>
</aside>
<pre><code class="lang-auto">random.seed(seed)
torch.manual_seed(seed)
</code></pre>
<p>the only sources of randomness for reproducible data augmentation should be <code>random</code> and <code>torch</code>. Right?</p>
<p>Otherwise one would need to set a seed with <code>worker_init_fn</code>, which would get us back to the pseudo-random samples repeating for each epoch.</p>

username: wesen
cooked: <p>Thanks for the clear explanation!</p>
<p>Four years later, I was wondering whether it’s still managed the same way. I’ve been trying to find the source code location, where a worker’s seed is set to the base seed (resampled at each epoch/when the iterator is regenerated), plus the worker id.</p>
<p>Here: <a href="https://github.com/pytorch/pytorch/blob/b539c61631fdda520d46cf653f1c30d3c332cf2c/torch/utils/data/dataloader.py#L1120" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/utils/data/dataloader.py at b539c61631fdda520d46cf653f1c30d3c332cf2c · pytorch/pytorch · GitHub</a><br>
it seems like the worker_loop is called only with the base_seed as seed.</p>
<p>Confusingly, running your code snippet, the workers <em>do</em> have different seeds (so it should be implemented the way you described it), but using the same seed twice (once per sample) still gives different crop indices (so your example doesn’t seem to work on my machine):</p>
<pre><code class="lang-auto">epoch 0
worker seed 5326672603560951676
sampled crop indices (2, 12, 200, 200)
worker seed 5326672603560951677
worker seed 5326672603560951676
sampled crop indices (9, 19, 200, 200)
sampled crop indices (21, 14, 200, 200)
worker seed 5326672603560951677
sampled crop indices (22, 22, 200, 200)

epoch 1
worker seed 2890333324556720703
sampled crop indices (4, 17, 200, 200)
worker seed 2890333324556720703
sampled crop indices (22, 19, 200, 200)
worker seed 2890333324556720704
sampled crop indices (22, 11, 200, 200)
worker seed 2890333324556720704
sampled crop indices (4, 13, 200, 200)
</code></pre>
<p>My PyTorch versions:<br>
torch==2.1.0+cu118<br>
torchvision==0.16.0+cu118</p>