Dataloaders multiprocess with torch.manual_seed

username: Digbalay_Bose
cooked: <p>Hi,<br>
I am currently fixing the seed values globally in my script using the following snippet:</p>
<pre><code class="lang-auto">seed_value=123457
np.random.seed(seed_value) # cpu vars
torch.manual_seed(seed_value) # cpu  vars
random.seed(seed_value) # Python
torch.cuda.manual_seed(seed_value)
torch.cuda.manual_seed_all(seed_value)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>Further, I am using multiple processes in my DataLoader using num_workers &gt; 0 to load frames for a video. Does the above snippet assign a fixed seed <code>worker_id+seed_value </code> to every worker in an epoch?<br>
I am also using random augmentations as a part of my data loading pipeline like RandomResizedCrop, RandomHorizontal Flipping from torchvision transforms. If the worker seed is fixed at  <code>worker_id+seed_value </code>, does that mean at each epoch, the data will go through the same set of augmentations?</p>
<p>If someone can clarify this, it would be of great help. Thanks!</p>

username: tom
cooked: <p>From the <a href="https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading" rel="noopener nofollow ugc">documentation</a>:</p>
<blockquote>
<p>By default, each worker will have its PyTorch seed set to <code>base_seed + worker_id</code>, where <code>base_seed</code> is a long generated by main process using its RNG (thereby, consuming a RNG state mandatorily). However, seeds for other libraries may be duplicated upon initializing workers (e.g., NumPy), causing each worker to return identical random numbers. (See <a href="https://pytorch.org/docs/stable/notes/faq.html#dataloader-workers-random-seed" rel="noopener nofollow ugc">this section</a> in FAQ.).</p>
</blockquote>
<p>So unless you reset the RNG between to epochs, the creation of the base seed from the RNG (which isn’t completely obvious to me that it is a good idea for perfect randomness) ensures that you get a new random seed every time.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: I would always recommend experimentally verifying things like this. Even just <code>b1 = next(iter(dl)) ; b2 = next(iter(dl))</code> and inspecting the results.</p>

username: Digbalay_Bose
cooked: <p>Many thanks for the reply !</p>
<p>I cross-checked the generated batches using consecutive calls of <code>next(iter(dl))</code>. The batches are different.<br>
So the <code>base_seed</code> generated by the main process is different than the seed fixed manually using <code>torch.manual_seed</code>? Also, if I interpret it, for each epoch, the workers will be assigned a new seed <code>base_seed + worker_id</code>. And the data augmentations will be based on the new worker seed?</p>

username: tom
cooked: <aside class="quote no-group" data-username="Digbalay_Bose" data-post="3" data-topic="123044">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/digbalay_bose/48/38989_2.png" class="avatar"> Digbalay_Bose:</div>
<blockquote>
<p>So the <code>base_seed</code> generated by the main process is different than the seed fixed manually using <code>torch.manual_seed</code>?</p>
</blockquote>
</aside>
<p>Yes, the base_seed is generated from drawing a random number, so it is dependent on / defined by the RNG state (and thus the manual_seed) but not identical to it.</p>
<aside class="quote no-group" data-username="Digbalay_Bose" data-post="3" data-topic="123044">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/digbalay_bose/48/38989_2.png" class="avatar"> Digbalay_Bose:</div>
<blockquote>
<p>Also, if I interpret it, for each epoch, the workers will be assigned a new seed <code>base_seed + worker_id</code>. And the data augmentations will be based on the new worker seed?</p>
</blockquote>
</aside>
<p>One thing to keep in mind is that if enabled, the shuffeling of the dataset (i.e. which things to stick in a minibatch) is the done in the parent.<br>
The randomness in the dataset (and potentially the collation function) like augmentations (in the dataset) are indeed from in the worker based on the new worker seed. All this assuming that you use PyTorch’s random functions (which you should and you need to be super-careful to properly initialize your RNG if you don’t).</p>
<p>All this said, there is a good argument to be made to do the random augmentation after batching on the GPU if possible. Solves not only any confusion around randomness but is likely much more efficient, too.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Digbalay_Bose
cooked: <p>Thank you <a class="mention" href="/u/tom">@tom</a> for clarifying the doubts.</p>
<p>In the <code>torch.utils.DataLoader</code> argument, we can pass a function declaration to <code>worker_init_fn</code>. Is it advisable to use  <code>worker_init_fn</code> to access the worker’s current seed and seed the other accompanying libraries like random based on the same seed ?</p>
<pre><code class="lang-auto">    worker_seed = torch.initial_seed() % 2**32
    numpy.random.seed(worker_seed)
    random.seed(worker_seed)

DataLoader(
    train_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    worker_init_fn=seed_worker
)
```. (As mentioned in https://pytorch.org/docs/stable/notes/randomness.html#dataloader)</code></pre>

username: tom
cooked: <p>Well, so it depends on</p>
<ul>
<li>Will you use other libraries’ random functions?</li>
<li>Is there a chance you might be using them inadvertently?</li>
<li>Will others grab your code and do funny things with it and then claim it’s your fault they didn’t get proper randomness?</li>
</ul>
<p>If any of these is yes or at least might be yes, it is a good idea to initialize randomness. If it is all firmly no, it just adds uninteresting boilerplate to your code.</p>
<p>Personally, I think that mixing RNGs from different libraries is not a great idea (and in fact, trying to be clever around RNGs is usually a good way to shoot yourself into the foot unless you exactly know what you are doing).<br>
Imagine you do this with two libraries that have identical RNGs. Now you seed them to identical states. By some accident you draw random integers in some range from both of them in sync. They will always be identical. Now you combine them, say, by taking the difference. Instead of getting a random number with a symmetric triangular distribution on [-1, 1] which you would get from independent RVs you now get all 0s.<br>
Now this is an obvious and improbable example, but more subtle interactions do exist and happen where people do not expect them.</p>
<p>(For another variation of the “don’t try to be clever” thing: here is a link why, while understanding the motivation, I am skeptical of the “seed with a random number” approach: <a href="https://www.johndcook.com/blog/2016/01/29/random-number-generator-seed-mistakes/" class="inline-onebox" rel="noopener nofollow ugc">Random number generator seed mistakes &amp; how to seed an RNG</a> .)</p>
<p>Best regards</p>
<p>Thomas</p>