Why random functions return same values during the training based on Ignite?

username: odats
cooked: <p>I use Ignite for training my model.<br>
np.random.randint, torch.randint… called from  <span class="mention">@trainer.on</span>(Events.EPOCH_COMPLETED(every=1)) return the same values.<br>
It looks like a seed is hardcoded. Why do they have it and how to cancel it?</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/vfdev-5">@vfdev-5</a>, who is the main author of the library.</p>

username: vfdev-5
cooked: <p><a class="mention" href="/u/odats">@odats</a> this is done to have reproducble trainings = user can resume the training from a checkpoint (based on epoch or iteration) and have “almost” the same training behaviour. Please, see this : <a href="https://pytorch.org/ignite/concepts.html#resume-training" rel="nofollow noopener">https://pytorch.org/ignite/concepts.html#resume-training</a></p>
<p>To alter the seed, it is possible to specify the seed as the argument in the run:</p>
<pre><code class="lang-python">trainer.run(dataloader, seed=1234)
</code></pre>
<p>That being said we think to change this behaviour and master/nightly releases already setup the seed using torch default random generator : <a href="https://github.com/pytorch/ignite/pull/799" rel="nofollow noopener">https://github.com/pytorch/ignite/pull/799</a></p>
<p>Hope this helps. Feel free to ask other questions.</p>
<p>PS. <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the notification ! I should have missed the email somehow…</p>

username: odats
cooked: <p>I have some randomness in Dataset get item. I do random shuffling of values. It looks like seed is not hardcoded there and I get true random data for each training step. Can you please confirm?</p>

username: vfdev-5
cooked: <blockquote>
<p>I have some randomness in Dataset get item. I do random shuffling of values. It looks like seed is not hardcoded there and I get true random data for each training step. Can you please confirm?</p>
</blockquote>
<p>Sorry, I do not get what you would like me to confirm.<br>
On ignite’s side, seed is used to make random state “synchronization” at each dataloader restart (it corresponds in most of the cases to the epoch size). In this way, for a given iteration/epoch the dataflow can be the same (for a given seed). More precisely it is something like</p>
<pre><code class="lang-auto">for e in range(num_epochs):
    set_seed(seed + e)
    do_single_epoch_iterations(dataloader)
</code></pre>
<p>Hope this answers your question</p>

username: odats
cooked: <p>I mean in my custom dataset I have a code:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
   def __getitem__(self, idx):
      item = self.dataset.iloc[idx]
      new_item = np.random.permutation(item)
      return new_item
</code></pre>
<p>By default each time it returns random seed. My question is: will my dataset, for each epoch, return new randomly permuted examples?</p>
<p>I have found some information:<br>
However, while resuming from iteration, random data augmentations are not synchronized in the middle of the epoch and thus batches remaining until the end of en epoch can effectively be different of those from the initial run.</p>

username: vfdev-5
cooked: <p>IMO, there is no problems with your <code>getitem</code>’s permutations, they all will be generated randomly as supposed.</p>
<blockquote>
<p>By default each time it returns random seed.</p>
</blockquote>
<p><code>new_item = np.random.permutation(item)</code> it returns a random value not random state’s seed, right ?</p>
<blockquote>
<p>I have found some information:<br>
However, while resuming from iteration, random data augmentations are not synchronized in the middle of the epoch and thus batches remaining until the end of en epoch can effectively be different of those from the initial run.</p>
</blockquote>
<p>This info is about when you try to resume/restart the training from a checkpoint that saved trainer’s state in the middle of epoch (e.g. I store checkpoints every 1000 iterations and my epoch is 5432 iterations).</p>

username: odats
cooked: <p>Thank you for clarification. Everything is clear now. It would be nice to have some documentation about this behaviour.</p>

username: vfdev-5
cooked: <p>I agree we need to better update the documentation.</p>