Reproducibility with all the bells and whistles

humble_D: <p>Had to dig up a lot of docs and discussions to finally figure out the right set of code to make a pytorch pipeline completely reproducible.</p>
<p>There are two things here, firstly you would want to use a seed that will be used to seed pytroch, numpy and python right at the start of main process. Be it training, be it inference.</p>
<p>You would want to use this function:</p>
<pre><code class="lang-auto">def seed_all(seed):
    if not seed:
        seed = 10

    print("[ Using Seed : ", seed, " ]")

    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    numpy.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
</code></pre>
<p>Just call it right at the start and you are good to go.</p>
<p>Now comes the part where you use worker processes in DataLoader to speed up training. Each worker process is a separate realm and has no relation to the seed that you have used in your main process except for the fact that internally PyTorch seeds itself for you using <strong>“base_seed + worker_id”</strong> and base_seed is generated based on the seed you used in the main process. So we are still predictable until this part. Keep in mind that whether u define a <strong>worker_init_fn</strong> or not, PyTorch will get seeded.</p>
<p>The reason why you would want to implement the worker_init_fn is so that you can seed the other libraries that PyTorch does not seed by default. For example numpy and python random module. As you may know, Albumentations library uses python random module and thus making sure that each worker has a unique yet predictable seed is important for randomness. So you would want to use this (Use the worker_seed to seed everything else except PyTorch which has been seeded for u):</p>
<pre><code class="lang-auto">def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)
</code></pre>
<p>you would define this function in the global scope and use it like so:</p>
<pre><code class="lang-auto">DataLoader(
  train_dataset,
  batch_size=batch_size,
  num_workers=4,
  worker_init_fn=seed_worker,
  shuffle=True
)
</code></pre>
<p>I am training on a single GPU and this ensures AtoZ reproducibility. The only catch here is you will get different results if you change the number of workers for obvious reasons.<br>
I hope this post reaches people on time so that they don’t waste precious GPU compute trying to figure out where on earth the randomness comes from.</p>

ptrblck: <p>Thanks for sharing this post!<br>
Would you be interested on adding some of this information to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Randomness docs</a>? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

humble_D: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I would love to. Thanks for the encouragement.</p>

ptrblck: <p>Sounds good! Please create a feature request <a href="https://github.com/pytorch/pytorch/issues">here</a>, explain your use case a bit (or link to your initial post) and propose some changes. Once it’s reviewed, you should be good to go to create the PR! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

humble_D: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/38541" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/38541" target="_blank" rel="nofollow noopener">Doc update regarding predictability of experiments using Seeds and Workers</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-15" data-time="06:39:59" data-timezone="UTC">06:39AM - 15 May 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/AakashMallik" target="_blank" rel="nofollow noopener">
          <img alt="AakashMallik" src="https://avatars1.githubusercontent.com/u/19551058?v=4" class="onebox-avatar-inline" width="20" height="20">
          AakashMallik
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
The current official docs don't have specific examples regarding the use of seeds and worker_init_fn in order to attain a...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


milan_kalkenings: <p>Thank you a lot, it works for me!<br>
I have one follow up question:</p>
<p>i get the same train_eval-results when i do:</p>
<pre><code class="lang-auto">set_seed(10)  # does everything you mention above
train_eval_loop()
set_seed(10)
train_eval_loop()
</code></pre>
<p>but not when I do</p>
<pre><code class="lang-auto">set_seed(10)
train_eval_loop()
train_eval_loop()
</code></pre>
<p>I don’t really get why, because I just set the same settings once again.<br>
Can anyone explain this to me?</p>

ptrblck: <p>In your first approach you are resetting the seeds before calling <code>train_eval_loop</code>, which makes sure that the random operations in <code>train_eval_loop</code> will sample the same random numbers and thus would yield the same results.<br>
In the second approach, you are seeding the code only once, so that multiple calls into <code>train_eval_loop</code> will use the pseudorandom number generator in its current state and could yield different results. However, rerunning the second approach in multiple sessions should again yield the same results.</p>
<p>Usually you would stick to the second approach only, since you don’t want to sample the same “random” numbers in each training iteration (they would not be random anymore).</p>

vict0rsch: <p>A bit late to the party but how would you do this, given Numpy’s recommendations are to drop the legacy global <code>np.random.seed()</code> in favor of <code>rng = np.random.default_rng(seed)</code> and then using <code>rng</code> ?</p>
<p>I get that you can do</p>
<pre><code class="lang-auto">def seed_all(seed):
    if not seed:
        seed = 10

    print("[ Using Seed : ", seed, " ]")

    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    rng = np.random.default_rng(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

    return rng
</code></pre>
<p>But how would you go about <code>seed_worker()</code> ?</p>
<p><a href="https://numpy.org/doc/stable/reference/random/generator.html" rel="noopener nofollow ugc">Numpy: Random Generator</a></p>

Eloy_Schultz: <p>Could you explain why we use <code>torch.initial_seed()%2**32</code> to set the seed of the worker?<br>
If I print the <code>worker_seed</code> in <code>seed_worker()</code>, I get a different result each time the workers are initialized at the start of each epoch.<br>
Now the <code>worker_seed</code> is consistent in between each time I execute my training script, but it differs between each epoch.</p>
<p>Why is this?</p>

ptrblck: <p>The seeds should be changed between epochs to avoid applying the same “random” transformations in each epoch.</p>