How to fix all workers' seed via worker_init_fn for every iter?

dahyun: <p>I believe I fixed all workers’ seed as 0 via <code>worker_init_fn</code>, but it turned out all workers have different seeds indeed.</p>
<p><strong>Is there any way to fix all workers’ seed as a fixed constant for every iteration?</strong></p>
<p>In <code>dataset.py</code>,</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset
from torch.utils.data import DataLoader
from pytorch_lightning.core.datamodule import LightningDataModule

class MyDataset(Dataset):
   ...
    def __getitem__(self, idx):
        *# this is where the results printed*
        print(torch.utils.data.get_worker_info())

class MyDataModule(LightningDataModule):
   ...
    def worker_init_fn(self, worker_id):                                                                                                                                
        seed = 0                                                                                                                                                  
        torch.manual_seed(seed)                                                                                                                                   
        torch.cuda.manual_seed(seed)                                                                                                                              
        torch.cuda.manual_seed_all(seed)                                                                                          
        np.random.seed(seed)                                                                                                             
        random.seed(seed)                                                                                                       
        torch.manual_seed(seed)                                                                                                                                   
        return

    def val_dataloader(self):
        dataloader = DataLoader(dataset, \
                             batch_size=self.args.bsz, \
                             shuffle=False, \
                             num_workers=8, \        
                             worker_init_fn=self.worker_init_fn)
</code></pre>
<p>This code gives me workers with different seed <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> :</p>
<blockquote>
<p>WorkerInfo(id=2, num_workers=8, seed=900450186894289457, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=7, num_workers=8, seed=900450186894289462, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=3, num_workers=8, seed=900450186894289458, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=6, num_workers=8, seed=900450186894289461, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=5, num_workers=8, seed=900450186894289460, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=2, num_workers=8, seed=900450186894289457, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=7, num_workers=8, seed=900450186894289462, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=3, num_workers=8, seed=900450186894289458, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)<br>
WorkerInfo(id=0, num_workers=8, seed=900450186894289455, dataset=&lt;data.MyDataset object at 0x7f11d3613110&gt;)</p>
</blockquote>
<p><strong>Is there any way to fix all workers’ seed as a fixed constant for every iteration?</strong></p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>While the worker will use the <code>base_seed + worker_id</code> as its internal seed, your code should work nevertheless, since you are reseeding, if I’m not mistaken.<br>
Here is a small example, which also shows the different worker seeds, but the same result for <code>torch.randn</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, idx):
        # this is where the results printed
        print(torch.utils.data.get_worker_info())
        return torch.randn(1)

    def __len__(self):
        return(10)

def worker_init_fn(worker_id):                                                                                                                                
    seed = 0                
                                                                                                                                   
    torch.manual_seed(seed)                                                                                                                                   
    torch.cuda.manual_seed(seed)                                                                                                                              
    torch.cuda.manual_seed_all(seed)                                                                                          
    np.random.seed(seed)                                                                                                             
    random.seed(seed)                                                                                                       
    torch.manual_seed(seed)                                                                                                                                   
    return

    
dataset = MyDataset()
dataloader = DataLoader(dataset,
                        batch_size=2,
                        shuffle=False, 
                        num_workers=8,         
                        worker_init_fn=worker_init_fn)

for data in dataloader:
    print(data)
</code></pre>

dahyun: <p>It shows the same results for every run! Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1112d0d463daea3a9e34e0e9b528f57d3914827c.png" alt="image" data-base62-sha1="2r2qD8ykvEYvBa4mmPaonmDRLVG" width="512" height="52"><br>
But it still shows different seeds for different workers, which I believe all of them should have the same seed as I did so in <code>worker_init_fn</code>?</p>
<p>Thank you.</p>

gokulavasan: <p>That is because the worker_info is created (with the seed passed in from the DataLoader) before worker_init_fn is called:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/30625ae58228a220069c1153b45ea20addee37e4/torch/utils/data/_utils/worker.py#L241-L251">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/30625ae58228a220069c1153b45ea20addee37e4/torch/utils/data/_utils/worker.py#L241-L251" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/30625ae58228a220069c1153b45ea20addee37e4/torch/utils/data/_utils/worker.py#L241-L251" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/30625ae58228a220069c1153b45ea20addee37e4/torch/utils/data/_utils/worker.py#L241-L251</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="241" style="counter-reset: li-counter 240 ;">
          <li>_worker_info = WorkerInfo(id=worker_id, num_workers=num_workers,</li>
          <li>                          seed=seed, dataset=dataset)</li>
          <li></li>
          <li>from torch.utils.data import _DatasetKind</li>
          <li></li>
          <li>init_exception = None</li>
          <li></li>
          <li>try:</li>
          <li>    if init_fn is not None:</li>
          <li>        init_fn(worker_id)</li>
          <li></li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>