Reproducibility of data got from Dataloader with single/multiple threads

username: pepper8362
cooked: <p>Hi,</p>
<p>I am using <code>torch.utils.data.DataLoader</code> with a customized dataset with data randomization (e.g. random clipping). I find DataLoader seems to give different data with <code>num_workers=0</code> and with other <code>num_workers</code> values. An example code is as below:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import Dataset, DataLoader

def rnd_clip(d, l=20):     
     idx_st = np.random.randint(0, len(d)-l)  
     print(idx_st)   
     return d[idx_st:idx_st+l]

class DummyDataset(Dataset):
     def __init__(self):
          self.data = np.random.randn(4,100)

     def __len__(self):
          return len(self.data)

     def __getitem__(self, index):
          tmp_data = self.data[index]
          tmp_data = rnd_clip(tmp_data)
          return tmp_data

if __name__ == "__main__":
     seed = 1254
     num_workers = 16
     np.random.seed(seed)
     torch.random.manual_seed(seed)

     dataset = DummyDataset()
     dataloader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=num_workers)
     for batch_idx, data in enumerate(dataloader):
          print(data.std())
</code></pre>
<p>The printed std seems the same if <code>num_workers</code> is not 0. But if you set <code>num_workers</code> to 0, the printed std will be different. What’s the cause of this? And is there any way to fix it and get consistent data? Thanks!</p>

username: ptrblck
cooked: <p>You would need to seed <code>numpy</code> and other libs in the <code>worker_init_fn</code> as described <a href="https://pytorch.org/docs/stable/notes/randomness.html#dataloader">here</a>.</p>

username: pepper8362
cooked: <p>Thanks for your reply. I have tried to update the above demo with the instructions in your link. But I still get different results with <code>num_workers=0</code> and other values of <code>num_workers</code>. Please can you have a look on what’s wrong in my seed settings? Thanks! The updated code is as below (I changed the seed since this one leads to larger difference):</p>
<pre><code class="lang-auto">import torch
import random
import numpy as np
from torch.utils.data import Dataset, DataLoader

def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

def rnd_clip(d, l=20):     
     idx_st = np.random.randint(0, len(d)-l)  
     print(idx_st)   
     return d[idx_st:idx_st+l]

class DummyDataset(Dataset):
     def __init__(self):
          self.data = np.random.randn(4,100)

     def __len__(self):
          return len(self.data)

     def __getitem__(self, index):
          tmp_data = self.data[index]
          tmp_data = rnd_clip(tmp_data)
          return tmp_data

if __name__ == "__main__":
     seed = 1365
     num_workers = 0
     np.random.seed(seed)
     torch.random.manual_seed(seed)

     g = torch.Generator()
     g.manual_seed(seed)

     dataset = DummyDataset()
     dataloader = DataLoader(dataset, batch_size=16, shuffle=True, num_workers=num_workers, worker_init_fn=seed_worker, generator=g)
     for batch_idx, data in enumerate(dataloader):
          print(data.std())
</code></pre>

username: ptrblck
cooked: <p>Sorry, I might have misunderstood your actual use case.<br>
Are you trying to get the same random numpy values in a single process vs. multiple processes?<br>
In your single process code the numpy is seeded in the main process and will thus return the deterministic values in <code>np.random.randint(0, len(d)-l)</code> for all sequential calls.<br>
In your multi-processing use case you could seed numpy in each process using the same (this would be dangerous) or different seeds.<br>
Creating a mapping between both use cases might not be easily possible, since the execution order is different.</p>

username: pepper8362
cooked: <p>I see. Thank you very much!</p>