Reproducibility not possible despite following PyTorch guidelines

papoo13: <p>I cannot reproduce my results even when I set the seeds following PyTorch guidelines. This is my seeding configuration:</p>
<pre><code class="lang-auto">def configure_random_seed(args):
    with logger.LoggingBlock("Random Seeds", emph=True):
        # python
        seed = args.seed
        random.seed(seed)
        logging.info("Python seed: %i" % seed)
        # numpy
        seed += 1
        np.random.seed(seed)
        logging.info("Numpy seed: %i" % seed)
        # torch
        seed += 1
        torch.manual_seed(seed)
        logging.info("Torch CPU seed: %i" % seed)
        # torch cuda
        seed += 1
        torch.cuda.manual_seed_all(seed)
        torch.cuda.manual_seed(seed)
        logging.info("Torch CUDA seed: %i" % seed)

        torch.backends.cudnn.deterministic = True
        torch.backends.cudnn.benchmark = False
</code></pre>
<p>I also set:</p>
<pre><code class="lang-auto">def seed_worker(worker_id):
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)
</code></pre>
<pre><code class="lang-auto"> gpuargs = {
            "num_workers": args.num_workers,
            "pin_memory": True,
            "worker_init_fn": seed_worker} if 'cuda' in args.device else {}

train_loader = DataLoader(
                train_dataset,
                batch_size=args.batch_size,
                shuffle=True,
                drop_last=False,
                **gpuargs)
</code></pre>
<p>Despite this, I still get widely different results (5% accuracy difference). While reading the PyTorch guidelines, I read this part which I am not sure if I understand fully.</p>
<blockquote>
<p>“However, some applications and libraries may use NumPy Random Generator objects, not the global RNG (<a href="https://numpy.org/doc/stable/reference/random/generator.html" class="inline-onebox" rel="noopener nofollow ugc">Random Generator — NumPy v1.26 Manual</a>), and those will need to be seeded consistently as well.”</p>
</blockquote>
<p>Maybe I need to set something different? anyone knows how to take care of NumPy Random Generator? I appreciate your guidance.</p>
<p>P.S. I’m working with a Bayesian network (Variational BNN) in a continual learning setting.</p>

ptrblck: <p>Did you also set <code>torch.use_deterministic_algorithms(True)</code> as mentioned in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>?<br>
Seeding might not be enough to get deterministic and reproducible outputs if the algorithms themselves produce the non-deterministic results.</p>

papoo13: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="155793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.use_deterministic_algorithms(True)</code></p>
</blockquote>
</aside>
<p>I’m setting torch <code>backends.cudnn.deterministic = True</code>. This is different from what you have mentioned?</p>

papoo13: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="155793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>torch.use_deterministic_algorithms(True)</code></p>
</blockquote>
</aside>
<p>and one more question regarding <code>torch.use_deterministic_algorithms(True)</code>, where should I set it up? so, I have many modules in my code where I am importing torch. Should I add <code>torch.use_deterministic_algorithms(True)</code> after <code>import torch</code>?</p>

ptrblck: <p>Yes, it’s a different setting which picks deterministic algorithms for native kernels, checks that the cublas workspace size was properly set etc.</p>
<p>Set it right after the import.</p>

papoo13: <p>Thank you for your reply. I imported <code>torch.use_deterministic_algorithms(True)</code> after <code>import torch</code> in all the modules that imported torch in my code. I get the following error:</p>
<blockquote>
<p>RuntimeError: Deterministic behavior was enabled with either <code>torch.use_deterministic_algorithms(True)</code> or <code>at::Context::setDeterministicAlgorithms(true)</code>, but this operation is not deterministic because it uses CuBLAS and you have CUDA &gt;= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility" class="inline-onebox" rel="noopener nofollow ugc">cuBLAS</a></p>
</blockquote>
<p>The line of code where the error comes from is: <code>F.linear(input_means, self.weight, self.bias)</code><br>
I tried setting up the first environment variable before python <code>main.py</code> but hasn’t worked so far.<br>
Any ideas on this? does it mean that there is no deterministic equivalent to <code>F.linear()</code></p>
<p>I need to add that my CUDA driver version is 510.73.05 and CUDA_Version is 11.6.</p>

ptrblck: <p>If you are still seeing the error after trying to set the env variable, it might have been too late in the script.<br>
Set it as an external env variable or during the launch:</p>
<pre><code class="lang-auto">CUBLAS_WORKSPACE_CONFIG=:4096:8 python script.py
</code></pre>

papoo13: <p>Thank you very much. I added the argument before python script.py. It worked and then I was getting an error regarding the non-deterministic functions so I added  <code>torch.use_deterministic_algorithms(True, warn_only=True)</code>.<br>
Now, I get this warning instead:</p>
<blockquote>
<p>UserWarning: nll_loss2d_forward_out_cuda_template does not have a deterministic implementation, but you set ‘torch.use_deterministic_algorithms(True, warn_only=True)’. You can file an issue at <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/pytorch · GitHub</a> to help us prioritize adding deterministic support for this operation. (Triggered internally at  /opt/conda/conda-bld/pytorch_1646755897462/work/aten/src/ATen/Context.cpp:79.)</p>
</blockquote>
<p>and my result is not identical. I am running a continual learning application (prior based: using variational inference) and in two runs I get accuracies for task 1 and task 2 as follows:<br>
Task 1: Run1: 94.8%, Run2: 96.40%<br>
Task2: Run1:83.80%, Run2: 81.00%<br>
and this gap continues for the upcoming tasks.</p>
<p>I wonder whether this is normal in light of having one non-deterministic function, here nll_loss2d_forward_out_cuda?</p>

ptrblck: <aside class="quote no-group" data-username="papoo13" data-post="8" data-topic="155793">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f19dbf/48.png" class="avatar"> papoo13:</div>
<blockquote>
<p>I wonder whether this is normal in light of having one non-deterministic function, here nll_loss2d_forward_out_cuda?</p>
</blockquote>
</aside>
<p>It’s hard to tell as it would depend on the overall stability of your training.<br>
I.e. assuming you would get deterministic results you could rerun the script with different seeds and compare how the model would perform in the end.<br>
Since you are already using non-deterministic methods, you wouldn’t have a baseline but the test might still be interesting.</p>
<p>Also, in case you are not on the latest PyTorch version, install <code>1.12.0</code> or the nightly release to check if this operation has a deterministic version now.</p>