Does PyTorch change its internal seed during training?

BramVanroy: <p>I am trying to make my training code as deterministic and reproducible as possible. When running the same training code multiple times, and always re-initialising the model, I get different results - even if I set the seeds manually, before all runs start. I found that when I reset the seed on every training run, all runs <em>do</em> end up with the same result. This seems to indicate that torch (or numpy or Python’s random) internally <em>does</em> change its seed.</p>
<pre><code>def set_seed():
    torch.manual_seed(3)
    torch.cuda.manual_seed_all(3)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(3)
    random.seed(3)
    os.environ['PYTHONHASHSEED'] = str(3)

for i in range(10):
    set_seed()
    model = init_model()
    model.train()
    model.test()
</code></pre>
<p>The code above always produces the same result for the test set, as expected. But when I only set the seed once, i.e. outside the loop, results vary over the iterations. What causes this?</p>

ybj14: <p>uh… random generation is a sequential process so every time you generate a new random number the seed changes. Thus before every training begins you need to reset the seed, i.e. put set_seed() in the loop.</p>

BramVanroy: <p>I’m not sure we understand each other. Let’s say that I put <code>set_seed(5)</code> outside the range loop, I would expect all ten runs to have the same result. This is not the case. When I put <code>set_seed(5)</code> inside the range loop, it does work as expected. Does that mean the seed changes over time?</p>
<p>I did find that manual seed returns a <a href="https://pytorch.org/docs/stable/torch.html#torch.manual_seed" rel="nofollow noopener">generator</a>, which does explain this behaviour.</p>

ptrblck: <p>As <a class="mention" href="/u/ybj14">@ybj14</a> said, the pseudo-random number generator uses the seed as its initial seed and generates all sequential numbers based on this initial seed.<br>
That doesn’t mean that every “random” number will have the exact same value (which would create a useless random number generator), but that the sequence of random numbers are the same.<br>
Have a look at this example:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
print(torch.randn(2))
print(torch.randn(2))
print(torch.randn(2))

torch.manual_seed(2809)
print(torch.randn(2))
print(torch.randn(2))
print(torch.randn(2))
</code></pre>
<p>As you can see, <code>torch.randn</code> will yield new random numbers if sequentially called. After resetting the seed, you’ll get the same sequence of random numbers.</p>
<p>In your case, the model initialization can be seen as a call to the random number generator, which will yield different results. If you want your model to have exactly the same parameters, set the seed before initialization or reload the <code>state_dict</code> of a reference model.</p>

BramVanroy: <p>Thank you, that does make sense indeed!</p>

Vincent24: <p>Hello everybody!<br>
I did as above and had the problem</p>
<p>Code:<br>
def seed_torch(seed=0):<br>
random.seed(seed)#<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)#<br>
np.random.seed(seed)#<br>
torch.manual_seed(seed)#<br>
torch.cuda.manual_seed(seed)#<br>
torch.cuda.manual_seed_all(seed)# # if you are using multi-GPU.<br>
torch.backends.cudnn.benchmark = False#<br>
torch.backends.cudnn.deterministic = True#</p>
<p>seed_torch()<br>
net_basnet.train()<br>
for i, data in enumerate(train_dataloader):<br>
seed_torch()<br>
…<br>
…</p>
<p>seed_torch()<br>
net_basnet.eval()<br>
seed_torch()<br>
with torch.no_grad():<br>
for i, data in enumerate(val_dataloader):<br>
seed_torch()<br>
…</p>
<p>I have the same results of training loss. However, I have different results of validation loss, even putting seed_torch() everywhere in loops. Please help me solve this problem!!!. Thank you!</p>

s_n: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>“the pseudo-random number generator uses the seed as its initial seed and generates all <strong>sequential numbers</strong> based on this initial seed.”--------------------- Is it possible to print these <strong>sequential numbers</strong> in python somehow?</p>

ptrblck: <p>You can print these values directly, e.g. via:</p>
<pre><code class="lang-python">torch.manual_seed(2809)
for _ in range(10):
    print(torch.randn(1))
</code></pre>
<p>After re-seeding you would see the same values again.</p>

s_n: <p>Actually, my code is pretty big and I can’t run a for loop like this. I just wanted to print the value of RNG at some particular point in my code.</p>

ptrblck: <p>I’m not sure I understand what “value of RNG” is.<br>
The pseudorandom number generator can be seeded and will then output defined pseudorandom values.<br>
If you want to check which value would be sampled in the 1000th iteration from <code>torch.randn</code>, you could just call this method.</p>