How to set the same random seed for all workers?

username: odats
cooked: <p>Without setting a random seed the data loader returns the same random data for each epoch:<br>
<code>epoch 1: worker1-&gt;[2], worker2-&gt;[2], epoch 2:  worker1-&gt;[2], worker2-&gt;[2],...</code></p>
<p>When I set a random seed in worker_init_fn function I get random data for each worker:<br>
<code>epoch 1: worker1-&gt;[2], worker2-&gt;[4], epoch 2:  worker1-&gt;[7], worker2-&gt;[6],...</code></p>
<p>How to set a random seed to get a new random data each epoch but the same random data for each worker?<br>
<code>epoch 1: worker1-&gt;[2], worker2-&gt;[2], epoch 2:  worker1-&gt;[7], worker2-&gt;[7],...</code></p>
<pre><code class="lang-auto">def worker_init_fn(worker_id):                         
    print(torch.utils.data.get_worker_info().seed)
    print(torch.initial_seed())
    # set seed
    np.random.seed(int(torch.utils.data.get_worker_info().seed)%(2**32-1))

train_loader = DataLoader(DatasetRandom(), batch_size=2, num_workers=4, worker_init_fn=worker_init_fn)
</code></pre>
<p>I use Ignite for training.</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/odats">@odats</a> can you set a seed each epoch for that ?</p>
<pre><code class="lang-python">@trainer.on(Events.EPOCH_STARTED)
def set_epoch_seed():
    set_seed(trainer.state.epoch)
</code></pre>
<p>If this does not work for you, please provide a minimal code snippet to see the problem.</p>

username: odats
cooked: <p><code>set_seed</code> is no defined, where I can find this method?</p>

username: vfdev-5
cooked: <p>You can use <a href="https://github.com/pytorch/ignite/blob/master/ignite/utils.py#L142" rel="nofollow noopener">ignite.utils.manual_seed</a>, but I wanted to say that set the seed of your random generator.</p>

username: odats
cooked: <pre><code class="lang-auto">@trainer.on(Events.EPOCH_STARTED)
def set_epoch_seed():
    ignite.utils.manual_seed(trainer.state.epoch)
</code></pre>
<p>Yes, it works. But it has 2 issues:</p>
<ol>
<li>validation data loader returns the same random values as training loader</li>
<li>it always returns the same values (because seed values are always the same 1,2,3)</li>
</ol>

username: vfdev-5
cooked: <p>Please, provide a minimal code snippet to run it and see the problem in details</p>

username: odats
cooked: <aside class="quote no-group" data-username="odats" data-post="5" data-topic="92253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/odats/48/21970_2.png" class="avatar"> odats:</div>
<blockquote>
<p>Yes, it works. But it has 2 issues:</p>
<ol>
<li>validation data loader returns the same random values as training loader</li>
<li>it always returns the same values (because seed values are always the same 1,2,3)</li>
</ol>
</blockquote>
</aside>
<pre><code class="lang-auto">class DatasetRandom(Dataset):   
    def __len__(self):
        return 8
    def __getitem__(self, idx):
        #print(torch.utils.data.get_worker_info().seed)
        return (np.random.randint(1000, size=1), idx%2)

def worker_init_fn(worker_id):                         
    #print(torch.utils.data.get_worker_info().seed)
    #print(torch.initial_seed())
    np.random.seed(int(torch.utils.data.get_worker_info().seed)%(2**32-1))

@trainer.on(Events.EPOCH_STARTED)
def set_epoch_seed():
    #print('seed', torch.initial_seed())
    # manual_seed(trainer.state.epoch)
    manual_seed(int(torch.initial_seed())%(2**32-1))

train_loader = DataLoader(DatasetRandom(), batch_size=1, num_workers=2)
val_loader = DataLoader(DatasetRandom(), batch_size=1, num_workers=2, worker_init_fn=worker_init_fn)
</code></pre>
<ol>
<li>add worker_init_fn=worker_init_fn to val_loader</li>
<li>manual_seed(int(torch.initial_seed())%(2**32-1))</li>
</ol>

username: odats
cooked: <p>I have found issue and updated my code. I have posted working solution. Thank you for the support.</p>

username: vfdev-5
cooked: <blockquote>
<p>I have found issue and updated my code. I have posted working solution. Thank you for the support.</p>
</blockquote>
<p><a class="mention" href="/u/odats">@odats</a> great !</p>
<p>However, seeing your <code>DatasetRandom</code> implementation, it seems a bit weird to have the same random data each epoch as you said it in the very begining:</p>
<blockquote>
<p>Without setting a random seed the data loader returns the same random data for each epoch:<br>
<code>epoch 1: worker1-&gt;[2], worker2-&gt;[2], epoch 2:  worker1-&gt;[2], worker2-&gt;[2],...</code></p>
</blockquote>
<p>Probably, somewhere you should have some unwanted random seed synchonization. Which ignite version you are using, btw ?</p>

username: odats
cooked: <p>It is about pytorch. I thought Ignine has some elegant solution for this behavior. As you suggested add callback<code> @trainer.on(Events.EPOCH_STARTED)</code></p>
<pre><code class="lang-auto">class DatasetRandom(Dataset):    
    def __len__(self):
        return 2
    def __getitem__(self, idx):
        return (np.random.randint(1000, size=1), idx%2)

train_loader = DataLoader(DatasetRandom(), batch_size=1, num_workers=2)

for e in range(2): 
    print('epoch',e)
    for i, (images, labels) in enumerate(train_loader):
        print(i, images, labels)
</code></pre>
<pre><code class="lang-auto">epoch 0
0 tensor([[492]]) tensor([0])
1 tensor([[492]]) tensor([1])
epoch 1
0 tensor([[492]]) tensor([0])
1 tensor([[492]]) tensor([1])
</code></pre>
<p>Ignite: 0.4.1, PyTorch: 1.4.0</p>

username: vfdev-5
cooked: <blockquote>
<p>I thought Ignine has some elegant solution for this behavior.</p>
</blockquote>
<p>In v0.3.0 we had something similar to <code>set_epoch_seed</code> in the Engine automatically, but we found that it has a lot of side effects like you see with evaluator etc.</p>

username: vfdev-5
cooked: <p>Btw, seeing the implementation of DataLoader’s  worker loop : <a href="https://github.com/pytorch/pytorch/blob/23db54acdf455866206d2de36c33fb75e177cb4c/torch/utils/data/_utils/worker.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/23db54acdf455866206d2de36c33fb75e177cb4c/torch/utils/data/_utils/worker.py</a></p>
<p>A seed is setup for torch and python random (not numpy random) to randomize data each time dataloader iterator is created, so if you replace your <code>np.random.randint(1000, size=1)</code> by <code>random.randint(0, 1000)</code>, data will be random for each epoch.</p>

username: odats
cooked: <p>Thanks, but for numpy I should go with original solution?</p>

username: vfdev-5
cooked: <p>It depends on what is behind <code>np.randint</code> calls in your real case. I was thinking about data augmentations that can be parametrized with numpy, so if you have a control of that it would be more simple to replace randomness generation…</p>