How to set two dataloaders, of the same dataset, to the same shuffle?

username: Deeply
cooked: <p>Clearly, we need to manipulate the random seeding to achieve this, I have tried the one below, but I am getting different combinations in the two dataloaders of the same data set.</p>
<pre><code class="lang-auto">def random_seeding(seed_value):    
    if seed_value&gt;0:
        np.random.seed(seed_value)
        torch.manual_seed(seed_value)   
        random.seed(seed_value)    
        if cuda: torch.cuda.manual_seed_all(seed_value)
</code></pre>
<p>Oh, and I am using <code>next(iter...)</code>  to get the batches from the two dataloaders, as follows:</p>
<pre><code class="lang-auto">img_batch1 = next(iter(dataloader_1))
img_batch2 = next(iter(dataloader_2))
</code></pre>
<p><code>img_batch1</code> is different from <code>img_batch2</code>!</p>
<p>NB. Each <code>dataloader</code> is using <code>num_workers=1</code><br>
NB2. Uasing <code>state = torch.get_rng_state()</code>,  before the first loader, then, <code>torch.set_rng_state(state)</code> before the second loader,  did not help neither</p>

username: ptrblck
cooked: <p>I think you need to set the seed in the <code>worker_init_fn</code> as described in the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">docs</a>:</p>
<blockquote>
<p>By default, each worker will have its PyTorch seed set to <code>base_seed + worker_id</code> , where <code>base_seed</code> is a long generated by main process using its RNG. However, seeds for other libraies may be duplicated upon initializing workers (w.g., NumPy), causing each worker to return identical random numbers. (See <a href="https://pytorch.org/docs/stable/notes/faq.html#dataloader-workers-random-seed">My data loader workers return identical random numbers</a> section in FAQ.) You may use <a href="https://pytorch.org/docs/stable/torch.html#torch.initial_seed"> <code>torch.initial_seed()</code> </a> to access the PyTorch seed for each worker in <code>worker_init_fn</code> , and use it to set other seeds before data loading.</p>
</blockquote>

username: Deeply
cooked: <p>I 've tested both dataloaders and they are still not synchronized correctly; again, I use<br>
<code>next(iter(..))</code>, as in</p>
<pre><code class="lang-auto">img_batch1 = next(iter(dataloader_1))  
img_batch2 = next(iter(dataloader_2))
</code></pre>
<p>I am still getting different images in <code>img_batch1</code> than in <code>img_batch2</code></p>
<p>Note. I had to set the number of workers to zero to get it working.</p>
<pre><code class="lang-auto">random_seed()
dataloader_1 = DataLoader(test_set1, 
batch_size= 5, 
shuffle=True, 
num_workers=0,
worker_init_fn = torch.initial_seed()
)
random_seed()
dataloader_2 = DataLoader(test_set2, 
batch_size= 5, 
shuffle=True, 
num_workers=0,
worker_init_fn = torch.initial_seed()
)
</code></pre>

username: MariosOreo
cooked: <p>Hello <a class="mention" href="/u/deeply">@Deeply</a> ,</p>
<p>I think <code>torch.manaul_seed()</code> can guarantee the same shuffle sequence in each data_loader separately, but it cannot perform well between data_loaders. And I think <code>worker_init_fn</code> works in worker-wise in a dataloader, it can guarantee the random operations (like <code>transformations</code>) to the same in each process of a dataloader, so I think setting <code>worker_init_fn</code> to the same is not a proper solution.</p>
<p>If I am wrong, please correct me.</p>
<p>According to the <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L144" rel="nofollow noopener">source</a>, when you set <code>shuffle=True</code> it will create a <code>RandomSampler</code>. In random_sampler it calls <code>torch.randint</code> to implement the random indices sequence, it is what the problem is. By setting <code>torch.manaul_seed</code> can guarantee the same sequence in each data_loader separately but why not all the data_loader?</p>
<p>Another method, you could pass the same <code>RandomSampler</code> to your dataloaders so that they will get the same random sequence I believe.</p>

username: Deeply
cooked: <p>Thanks for sharing your thoughts on this issue. Allow me to say that <code>torch.manual_seed</code> did not help at all, neither did <code>worker_init_fn</code>, not sure if I am doing something wrong.  And; you are correct, passing <code>RandomSampler</code> would solve the issue. As a simple solution to my problem, However, I enabled the <code>dataset</code> class to return two instances of the same image instead of one, each transformed differently.</p>

username: bluelancer
cooked: <p>Hi is there any follow up for this?</p>
<p>This I think have to do with reproducibility…</p>