Does __getitem__ of dataloader reset random seed?

thnkim: <p>If I add a following code to <strong>getitem</strong> of <a href="https://github.com/pytorch/vision/blob/master/torchvision/datasets/cifar.py" rel="nofollow noopener">cifar.py</a> in torchvision,</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        ...
        # doing this so that it is consistent with all other datasets
        # to return a PIL Image
        img = Image.fromarray(img)

        if index == 0:    # outputs a random number for debugging
            print(np.random.uniform(-1, 1))

        if self.transform is not None:
            img = self.transform(img)
        ...
</code></pre>
<p>The line <code>print(np.random.uniform(-1, 1))</code> always outputs the same value for every epoch.</p>
<p>I tried to make my own dataloader without torchvision, but I observed the same issue.<br>
I never called <code>numpy.random.seed()</code>.</p>
<p>Interesting thing is, when I tried</p>
<pre><code class="lang-auto">kwargs = {'num_workers': 1, 'pin_memory': True} if args.cuda else {}
train_loader = torch.utils.data.DataLoader(
    dataset('../data', train=True, download=True, transform=transform),
    batch_size=args.batch_size, shuffle=True, **kwargs)

for data in enumerate(train_loader):
    numpy.random.uniform(-1, 1)    # This dummy line was intentionally added
    pass
</code></pre>
<p>With the dummy line above (the first line in for loop) makes <strong>getitem</strong> generate non-repetitive values.<br>
Without the line, I get the same value.</p>
<p>Thank you.</p>

smth: <p>the workers are re-created on every epoch. Hence i think you are seeing the RNG start again with the default seed at the beginning of every epoch.</p>

smth: <p>This is the small (hopefully easy to follow function):<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L129" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L129" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L129</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="119" style="counter-reset: li-counter 118 ;">
<li>        if re.search('[SaUO]', elem.dtype.str) is not None:</li>
<li>            raise TypeError(error_msg.format(elem.dtype))</li>
<li>
</li>
<li>        return torch.stack([torch.from_numpy(b) for b in batch], 0)</li>
<li>    if elem.shape == ():  # scalars</li>
<li>        py_type = float if elem.dtype.name.startswith('float') else int</li>
<li>        return numpy_type_map[elem.dtype.name](list(map(py_type, batch)))</li>
<li>elif isinstance(batch[0], int_classes):</li>
<li>    return torch.LongTensor(batch)</li>
<li>elif isinstance(batch[0], float):</li>
<li class="selected">    return torch.DoubleTensor(batch)</li>
<li>elif isinstance(batch[0], string_classes):</li>
<li>    return batch</li>
<li>elif isinstance(batch[0], collections.Mapping):</li>
<li>    return {key: default_collate([d[key] for d in batch]) for key in batch[0]}</li>
<li>elif isinstance(batch[0], collections.Sequence):</li>
<li>    transposed = zip(*batch)</li>
<li>    return [default_collate(samples) for samples in transposed]</li>
<li>
</li>
<li>raise TypeError((error_msg.format(type(batch[0]))))</li>
<li>
</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>On each <code>__iter__</code> the DataLoader returns a new one of these iterators: <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L302">https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L302</a></p>
<p>The iterator creates a new <code>self.workers</code> everytime it’s created: <a href="https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L151-L155">https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py#L151-L155</a></p>

thnkim: <p>Thank you!</p>
<p>and I found I should not use numpy.random.seed() for multi-threading:</p>
<blockquote>
<p>‘each subprocess will inherit the RNG state from its parent, meaning that you will get identical random variates in each subprocess’</p>
</blockquote>
<p>in in <a href="https://stackoverflow.com/questions/31057197/should-i-use-random-seed-or-numpy-random-seed-to-control-random-number-gener" class="inline-onebox" rel="noopener nofollow ugc">python - Should I use `random.seed` or `numpy.random.seed` to control random number generation in `scikit-learn`? - Stack Overflow</a>)</p>

Marvin: <p>Is this meaning that the same data augmentation is applied to data at different workers? If the batch size equals num_workers, there is very high chance that this batch has been transformed with exactly the same operations (rotation, translation, flip, etc).</p>
<p>If this is the case, would calling  <code>np.random.seed()</code> and <code>random.seed()</code> before each transformation make those transforms in a batch different?</p>

thnkim: <p>Well, the number of workers was 1.<br>
I mean the i-th iteration of for-loop in the code below generates the same random value.<br>
I created data-set once, created data-loader at every epoch.<br>
When I used random.*, no problem, but I had the issue when I used numpy.random.*.</p>
<pre><code class="lang-auto">for epoch in range(100):
    kwargs = {'num_workers': 1, 'pin_memory': True} if args.cuda else {}
    train_loader = torch.utils.data.DataLoader(
        dataset('../data', train=True, download=True, transform=transform),
        batch_size=args.batch_size, shuffle=True, **kwargs)
    #
    for data in enumerate(train_loader):
        do_something()
</code></pre>

smth: <p><a class="mention" href="/u/thnkim">@thnkim</a> it is because numpy’s RNG is not forkable.<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/29854398/seeding-random-number-generators-in-parallel-programs" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1702972/overcomer" target="_blank" rel="nofollow noopener">
    <img alt="overcomer" src="https://i.stack.imgur.com/6ejwQ.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/29854398/seeding-random-number-generators-in-parallel-programs" target="_blank" rel="nofollow noopener">Seeding random number generators in parallel programs</a>
</h4>

<div class="tags">
  <strong>python, numpy, random, multiprocessing</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1702972/overcomer" target="_blank" rel="nofollow noopener">
    overcomer
  </a>
  on <a href="https://stackoverflow.com/questions/29854398/seeding-random-number-generators-in-parallel-programs" target="_blank" rel="nofollow noopener">05:59PM - 24 Apr 15</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Instead, add this line to the top of your main script (and you need to use  python 3)</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
mp.set_start_method('spawn') 
</code></pre>

ignacio-rocco: <p>Hi, my fix was to pass a random seed to each worker in the following way:</p>
<p>In DataLoaderIter:</p>
<pre><code>self.workers = [
            multiprocessing.Process(
                target=_worker_loop,
                args=(self.dataset, self.index_queue, self.data_queue, self.collate_fn, np.random.randint(0, 4294967296, dtype='uint32')))
            for _ in range(self.num_workers)]
</code></pre>
<p>In the worker loop:</p>
<pre><code>def _worker_loop(dataset, index_queue, data_queue, collate_fn, rng_seed):
global _use_shared_memory
_use_shared_memory = True

np.random.seed(rng_seed)  # local random seed for each worker
torch.set_num_threads(1)
while True:
    r = index_queue.get()
    if r is None:
        data_queue.put(None)
        break
    idx, batch_indices = r
    try:
        samples = collate_fn([dataset[i] for i in batch_indices])
    except Exception:
        data_queue.put((idx, ExceptionWrapper(sys.exc_info())))
    else:
        data_queue.put((idx, samples))
</code></pre>
<p>Then, setting a global seed with numpy.random.seed makes the code reproducible, while keeping the random numbers diverse across workers.</p>

Rishi_Rawat: <p>My Soln:</p>
<p>Somewhere else:<br>
import datetime</p>
<p>in the dataset class<br>
def _ <em>call</em> _ :<br>
numpy.random.seed(datetime.datetime.now().second + datetime.datetime.now().millisecond)<br>
…</p>
<p>seems to be working ok</p>

Royi: <p>New user question.<br>
I use shuffle in the Data Loader with number of workers set to 0.<br>
I set at the beginning of the script <code>torch.manual_seed(seedNumber)</code>.</p>
<p>I was expecting that form now on results with the same parameters to be reproducible yet they are not.<br>
Is there a way to set the seed number of the random number generator used by the shuffle property?</p>
<p>I also use Dropout Layer, will the seed number set it as well?</p>
<p>Thank You.</p>

SimonW: <p>You may also need to set <code>torch.backends.cudnn.deterministic = True</code>.</p>

Royi: <p>Is there a place which describe exactly what’s needed to able to reproduce results on any system?</p>

ZhengRui: <p>I met same issue today, in my dataset’s <code>__getitem__</code>, i used <code>torch.multinomial</code> or <code>np.random.choice</code> to draw some samples, and in the dataloader i specified 8 workers, and in every continuous 8 batches, the drawed samples are the same.</p>

SimonW: <p><a class="mention" href="/u/zhengrui">@ZhengRui</a> umPy’s RNG is duplicated on forking multiprocess. See dataloader’s <code>worker_init_fn</code> option for how to solve it.</p>

ZhengRui: <p><a class="mention" href="/u/simonw">@SimonW</a>, thanks for the quick reply, i just read the documentation and now creating an virtualenv for pytorch3.1, was using pytorch2. Seems i don’t need to use <code>woker_init_fn</code> option, each worker already generate different sequences due to <code>base_id+worker_id</code> used to seed each worker internally.</p>

SimonW: <p>Yeah if you are using <code>torch.multinomial</code> then it is good already <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> But you need to set seed if you use numpy.</p>

jia_lee: <p>Hi, I’m new to pytorch. Do you mean we do not need to reset the seed for every worker if we use pytorch Dataloader?</p>

jia_lee: <p>Hi，if we use random rather than np.random, do we need to reset the random seed for every worker when we use the multi-process in Dataloader?</p>

SimonW: <p>You shouldn’t set random seed in getitem, and should only set the numpy one in worker_init_fn if you use numpy. See <a href="https://pytorch.org/docs/master/notes/faq.html#my-data-loader-workers-return-identical-random-numbers" rel="nofollow noopener">https://pytorch.org/docs/master/notes/faq.html#my-data-loader-workers-return-identical-random-numbers</a></p>

jia_lee: <p>Thank you for your help! Your hint is useful.  After some googling, if we use python random package (not NumPy.random), we don’t need to reset the random seed any more.</p>