Fix seed for data loader

username: cbd
cooked: <p>I am little bit confuse with the data loader and number of workers. Lets say I have 100 images in my dataset and used shuffle=True. I run the code for 20 epochs. In each epoch it randomly shuffle the 100 images.</p>
<p>(1) If i run the code again for 20 epochs, how to make it follows the same shuffle as previous run?<br>
(2) Lets the number of workers=2 and batch_size =8. Is it mean each worker will contribute to 4 images in batch?<br>
(3) If batch_size=1 and number of workers=2, Is it actually require one worker as batch_size=1? If no how the data process?<br>
(4) If number of worker=0, code works fine. Where actually it contribute?</p>

username: ptrblck
cooked: <ol>
<li>You could reset the seed via <code>torch.manual_seed</code> either before starting the new epoch or probably also by recreating the <code>DataLoader</code>s and seed the workers in the <code>worker_init_fn</code>.</li>
<li>Each worker will load the entire batch of 8 samples.</li>
<li>I don’t understand this question so could you explain it a bit more?</li>
<li>
<code>num_workers=0</code> will load the data in the main process and will not spawn background processes.</li>
</ol>

username: cbd
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could reset the seed via <code>torch.manual_seed</code></p>
</blockquote>
</aside>
<p>Thanks. If i re run for 20 epoch, it shuffle as it do for first run. Now consider 2 cases.<br>
<strong>Case 1:</strong> Model run for total 20 epochs.<br>
<strong>Case 2:</strong> Let say training stop at epoch-6 and model is saved at epoch-5. When i load the epoch-5 saved model and start continue training, it follow the shuffling of data of epoch-6 as epoch-1, epoch-7 as epoch-2 and so on. It does not follow the shuffling of epoch-6 as in Case 1. I need same shuffling as in  Case 1 even if i start continue training with saved model.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>num_workers=0</code> will load the data in the main process and will not spawn background processes</p>
</blockquote>
</aside>
<p>If i am not wrong, num_workers initiate background process and keep dataset of mentioned batch_size ready to speed up the process. Now please correct me, if at a time one batch is input to model in case of single GPU, there is no need to specify more than 1 number of workers. Is it?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I don’t understand this question so could you explain it a bit more?</p>
</blockquote>
</aside>
<p>I think your answer:2 is applied to question:3. Each worker create batch_size=1.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="cbd" data-post="3" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Let say training stop at epoch-6 and model is saved at epoch-5. When i load the epoch-5 saved model and start continue training, it follow the shuffling of data of epoch-6 as epoch-1, epoch-7 as epoch-2 and so on. It does not follow the shuffling of epoch-6 as in Case 1. I need same shuffling as in Case 1 even if i start continue training with saved model.</p>
</blockquote>
</aside>
<p>To to able to “reset” the seed the original training method could already set the seed in each epoch so that you could just use the same workflow. I.e. training in epoch 0 uses <code>seed=0</code>, training in epoch 1 uses <code>seed=1</code>, etc.</p>
<aside class="quote no-group" data-username="cbd" data-post="3" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>If i am not wrong, num_workers initiate background process and keep dataset of mentioned batch_size ready to speed up the process. Now please correct me, if at a time one batch is input to model in case of single GPU, there is no need to specify more than 1 number of workers. Is it?</p>
</blockquote>
</aside>
<p>Depending on your system it could be beneficial to create more processes. Even if a single batch is ready, the other workers would still preload the next batch and add them to the queue. You can experment with different number of workers and see where the sweet spot for your system and workload is.</p>
<aside class="quote no-group" data-username="cbd" data-post="3" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Each worker create batch_size=1.</p>
</blockquote>
</aside>
<p>If <code>batch_size=1</code> is used in the <code>DataLoader</code>, then yes.<br>
Otherwise each worker will create an entire batch (i.e. it will load and process <code>batch_size</code> samples):</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Each worker will load the entire batch of 8 samples.</p>
</blockquote>
</aside>

username: cbd
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>To to able to “reset” the seed the original training method could already set the seed in each epoch so that you could just use the same workflow. I.e. training in epoch 0 uses <code>seed=0</code>, training in epoch 1 uses <code>seed=1</code>, etc.</p>
</blockquote>
</aside>
<p>The question is when <strong>i run the model for 20 epoch(training not stop in beetween)</strong> it shuffle the images in each epoch. I want same shuffling if training stop and load the save model of previous epoch. Now problem comes when i load the saved model and start training. Let say training stop after saving the model of epoch-5.</p>
<p>Now if i load the saved model and start training from epoch-6, <strong>shuffling is as per epoch-1</strong>. Epoch-7 follows the shuffling as per epoch-2 and so on. What i want is shuffling of Epoch-6 as <strong>i run the model for 20 epoch(training not stop in beetween)</strong>. In short if i start training after loading saved model, it reset the seed even if i start from epoch-6. I hope question is clear now.</p>

username: ptrblck
cooked: <p>I am understanding the question and don’t think there is a way other than to:</p>
<ul>
<li>already seed the data loading during training and just doing the same when resuming the training</li>
<li>rerunning the training.</li>
</ul>
<p>Even if you already set a global seed during the training once, one might think you could just iterate the <code>DataLoader</code> for <code>x</code> epochs before continuing the training in epoch <code>x+1</code>. However, while this approach might work if random operations are only used in the data loading, all calls to the PRNG in the model (e.g. in dropout layers) are missed and you thus won’t get the same results.</p>

username: cbd
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="146711">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you could just iterate the <code>DataLoader</code> for <code>x</code> epochs before continuing the training in epoch <code>x+1</code>.</p>
</blockquote>
</aside>
<p>Thanks. I think i am near to what i want. Can u tell me where i can plug the code 1 in my code 2 below.</p>
<pre><code class="lang-auto">Code 1: 
for _ in range(epochs_to_restore):
    for batch in loader:
        pass

</code></pre>
<pre><code class="lang-auto">Code 2: 
for epoch in range(0,num_epoch):

   for i, data in enumerate(dataset):</code></pre>

username: ptrblck
cooked: <p>You can run code 1 right before starting the training in code 2, if you’ve set the global seed in both use cases at least once or if your model does not contain any random ops.</p>