PyTorch gives deterministic result inside a file, but different between files

Karls: <p>I know that my issue may feel similar to others regarding repeatability of results, but I will try to def the statement that it is different. Why? Let me explain:<br>
I use <strong>Optuna</strong> to optimize hyperparameters. Let’s consider this simplified objective function:</p>
<pre><code class="lang-auto"># For repetability
torch.manual_seed(7)
random.seed(7)
np.random.seed(7)
optuna_seed = optuna.samplers.TPESampler(seed=10)

def objective(trial: optuna.trial.Trial):
    optimizer_name = trial.suggest_categorical("optimizer_name", fun_train_h,)
    lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
    ANN_instance = model_func(optimizer_name,lr,train_dataloader,val_dataloader)
    accuracy_val=loss_eval(ANN_instance,val_dataloader)
    filename=f"/trial_{trial.number}-loss={accuracy_val:.5f}.pth"
    torch. Save(model, filename)
    return  accuracy_val
</code></pre>
<p>So the algorithm determine best optimizer type from the list, and the best value of learning rate. It also save all models to file. So I can reload it (I will skip determining the best file, for clarity, it is the same script as optimization):</p>
<pre><code class="lang-auto">ANN_instance=torch.load(best_checkpoint_path)
ANN_instance.eval()

accuracy_train=loss_eval(ANN_instance,train_dataloader)*100
accuracy_val=loss_eval(ANN_instance,val_dataloader)*100
accuracy_test=loss_eval(ANN_instance,test_dataloader)*100
</code></pre>
<p>It gives me results:<br>
Accuracy over Train: 29.73<br>
Accuracy over Validation: 20.51<br>
Accuracy over Test: 18.81<br>
A bit poor but it is just dirty test, so never mind. I can restart the file multiple times, always the same result, fully deterministic. This is end of <strong>File1</strong></p>
<p>Now in <strong>File2</strong> I want to repeat the learning process to register few additional metrics (not implemented yet) so I don’t reload checkpoint, I start learning again, making sure I have random seeds fixed. The same hyperparametres, the very same dataloaders.</p>
<pre><code class="lang-auto"># For repetability
torch.manual_seed(7)
random.seed(7)
np.random.seed(7)
#Results from previous file
optimizer="AdamW"
lr=0.0005
ANN_instance = model_func(optimizer_name,lr,train_dataloader,val_dataloader)
ANN_instance.eval()

accuracy_train=loss_eval(ANN_instance,train_dataloader)*100
accuracy_val=loss_eval(ANN_instance,val_dataloader)*100
accuracy_test=loss_eval(ANN_instance,test_dataloader)*100
</code></pre>
<p>And I get:<br>
Accuracy over Train: 15.05<br>
Accuracy over Validation: 14.8<br>
Accuracy over Test: 14.07<br>
Again I can restart the file as many times as I want, and I will get the same result. But different than in <strong>File1</strong></p>
<p>My question is why?</p>
<h1><a name="edit-i-think-i-figured-it-myself-1" class="anchor" href="#edit-i-think-i-figured-it-myself-1"></a>Edit: I think I figured it myself:</h1>
<pre><code class="lang-auto">def objective(trial: optuna.trial.Trial):
    torch.manual_seed(7)
    random.seed(7)
    np.random.seed(7)
</code></pre>
<p>This fixes the issue. But why global seed is not respected and I have to repeat it inside Optuna call?</p>

ptrblck: <p>The global seed is used but most likely the order of calls into the pseudo-random number generator differs between the two scripts.</p>

Karls: <p>Hi, thanks for the answer. What do you mean by the order of calls? The random seed is defined on the very top of the file, so it is executed before calling Optuna study.</p>

ptrblck: <p>Setting the seed is one requirement but to get deterministic outputs you would also need to make sure the order of calls into the PRNG is equal in both scripts, which is not always trivial. E.g. take this code snippet demonstrates the effect:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.manual_seed(2809)
#nn.Linear(10, 10)
print(torch.randn(10))
</code></pre>
<p>This script will output different results for the <code>randn</code> call depending if the linear layer was initialized or not even though it’s never actually used since an additional call into the PRNG was performed.</p>

Karls: <p>It makes sens, I even encountered similar issue when unpacking <code>torch_data.DataLoader</code> leads to change in the function result, despite the loader is not used directly in training algorithm but just to report data to Tensorboad. Is there any way to suppress this unwanted behavior other than calling manual seed multiple times?</p>

ptrblck: <p>This is also expected as described in <a href="https://discuss.pytorch.org/t/does-iterating-over-unshuffled-dataloader-change-random-state/173137/2">this post</a> as the <code>_BaseDataLoaderIter</code> is creating a <code>base_seed</code> by calling into the PRNG.</p>
<aside class="quote no-group" data-username="Karls" data-post="5" data-topic="196936">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/958977/48.png" class="avatar"> Karls:</div>
<blockquote>
<p>Is there any way to suppress this unwanted behavior other than calling manual seed multiple times?</p>
</blockquote>
</aside>
<p>It’s not unwanted behavior as it’s the expected behavior from any PRNG. Creating the same value for specific calls, such as <code>torch.randn(10)</code>, would not be random anymore and the PRNG would be broken. Each call into it needs to increase its offset to guarantee values are random.</p>
<p>Depending on this behavior is indeed not trivial so you would either need to rerun exactly the same code or would re-seed the code risking numbers might be repeated e.g. if you are reusing the same seed.</p>