Random seed is reset inside DataLoader? [maybe bug]

username: Alex_I
cooked: <p>I’m using an IterableDataset inside a DataLoader (multiple workers).  Some of the stuff in my IterableDataset code calls numpy.random functions.  I noticed after a while that in each epoch, the sequence of values returned by the random functions is exactly the same!  In other words, every worker is (somehow) reset to the same random seed at the beginning of the epoch (or when it is created).  So if (for example) the worker tried to do random image crops with positions from numpy.random, they are the same crops for each image for every epoch.</p>
<p>How/where is the seed set?  Is this expected behavior?</p>
<p>WHY does this happen?  I would expect the numpy.random seed in each worker to act the same as in a new process, unless numpy.random.seed is explicitly called by the user code.</p>
<p>(I was not doing anything explicit to set the seed, using either numpy or torch calls, or anything to make torch deterministic.  This seems to just be the default behavior - torch modifyint numpy to make it deterministic without being requested to by the user)</p>
<p>Simple code to reproduce:</p>
<pre><code class="lang-auto">import torch
import numpy as np
from torch.utils.data import DataLoader, Dataset

class TestIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self):
        super(TestIterableDataset).__init__()

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        for n in range(10):
            yield(worker_info.id, np.random.randint(1000000))
            
ds = TestIterableDataset()

for worker_id, number in DataLoader(ds, batch_size=4, num_workers=2):
    print(worker_id, number)

# This prints the same result every time it is run, and the same sequence from each worker:
# tensor([0, 0, 0, 0]) tensor([ 68669, 230721, 801136, 274196])
# tensor([1, 1, 1, 1]) tensor([ 68669, 230721, 801136, 274196])
# tensor([0, 0, 0, 0]) tensor([617084, 429589, 436968, 718987])
# tensor([1, 1, 1, 1]) tensor([617084, 429589, 436968, 718987])
# tensor([0, 0]) tensor([150977,  59469])
# tensor([1, 1]) tensor([150977,  59469])

</code></pre>

username: Alex_I
cooked: <p>Opened issue here: <a href="https://github.com/pytorch/pytorch/issues/41329" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/41329</a><br>
and apparently it’s not a bug, just a gotcha - a pretty well documented one at that.</p>
<p>Numpy random number generator keeps its seed over a fork(), and DataLoader starts worked processed using a fork, without doing anything special about numpy.random.  I’d say it seems like much more of a numpy issue - keep seeds over a fork by default is a kind of wtf behavior, but okay.  Easy enough to work around, just re-seed in worker_init_fn or at the beginning of <strong>iter</strong> works fine.</p>

username: dmitryako
cooked: <p>That does not solve the randomness problem inside numpy+pytorch. For example, you will get exactly the same numpy random numbers in each epoch! Does anyone know a “clean” solution to this? I have to  call (from inside dataset.<strong>getitem</strong>) a very messy (possibly wrong) code to make every worker have different numpy random seed at different epochs.<br>
def set_cuda_rand_seed():<br>
worker = torch.utils.data.get_worker_info()<br>
new_seed = np.random.randint(0, 2 ** 32 - 1)<br>
if worker is not None:<br>
new_seed = worker.seed<br>
new_seed = int(new_seed) % (2 ** 32 - 1)<br>
new_seed = int(new_seed)<br>
random.seed(new_seed)<br>
np.random.seed(new_seed)  # todo &lt;— bug using num_workers&gt;0<br>
return new_seed</p>