Random seed with external GPU

username: Karoly_Poka
cooked: <p>Hi all,</p>
<p>I bought a new Palit GeForce RTX 3070 GPU, to speed up my deep learning projects. My laptop is a Dell Latitude 5491 with an Nvidia GeForce MX130 and Intel UHD Graphics 630. I am using the GeForce RTX 3070 in a Razer Core X via Thunderbolt 3.0.</p>
<p>I would like to make my pytorch training reproducible, so I am using:<br>
torch.manual_seed(1)<br>
np.random.seed(1)<br>
random.seed(1)<br>
torch.cuda.manual_seed(1)<br>
torch.cuda.manual_seed_all(1)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
<p>Symptom: When the device=“cuda:0” its addressing the MX130, and the seeds are working, I got the same result every time. When the device=“cuda:1” its addressing the RTX 3070 and I dont get the same results. Seems like with the external GPU the random seed is not working. When device=“cuda” its automatically uses the RTX 3070 and no reproducibility.<br>
I am working with num_workers=0 and worker_init_fn=np.random.seed(1) in the dataloder. So practically changing the executor GPU has effect on the random seed. I dont want to, and I am not using both GPU-s in parallel.</p>
<p>How can I make the work with external GPU reproducible? I would very appreciate any help. Thanks in advance!</p>
<p>Pytorch version: 1.7.0<br>
Cuda toolkit: 11.0.221<br>
Anaconda version: 2020.07</p>
<p>According to NVIDIA-SMI:<br>
Cuda version: 11.1<br>
Driver version: 457.09</p>

username: ptrblck
cooked: <p>Additionally to the already used arguments, you could also set <code>torch.set_deterministic(True)</code> as described in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a> so that an error is raised in case a non-deterministic method is used.</p>

username: Karoly_Poka
cooked: <p>Thank you for the tip it solved my problem! Yes with <code>torch.set_deterministic(True)</code> I’ve  got the following error:<br>
<strong>RuntimeError: Deterministic behavior was enabled with either torch.set_deterministic(True) or at::Context::setDeterministic(true), but this operation is not deterministic because it uses CuBLAS and you have CUDA &gt;= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility" rel="noopener nofollow ugc">https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility</a></strong></p>
<p>So as it turned out the <code>CUBLAS_WORKSPACE_CONFIG=:16:8</code> or <code>CUBLAS_WORKSPACE_CONFIG=:4096:2</code> environment variable settings can solve the problem, and make the training reproducible if you have CUDA &gt;=10.2 version.</p>
<p>My internal MX130 probably used an older CUDA version which was not shown in the nvidia-smi, and probably that’s why it worked in that case, but not worked with the eGPU.</p>
<p>PS.: some pytorch layers can’t even work deterministically for example: <code>nn.AdaptiveAvgPool1d()</code> . But in my case I just had to remove the <code>torch.set_deterministic(True)</code> setting and there was no error and the particular layer had no effect on the deterministic results.</p>

username: JDE65
cooked: <p>Great advise, but where/how do you define the environment variable CUBLAS_WORKSPACE_CONFIG ?<br>
THX in advance <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: ptrblck
cooked: <p>You can set it as an env variable in your terminal either via <code>export</code> or by prepending it to your application execution:</p>
<pre><code class="lang-auto">CUBLAS_WORKSPACE_CONFIG=:16:8 python script.py args
</code></pre>

username: anu
cooked: <p>hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I am a Windows user and got this error from my terminal that says <em>‘CUBLAS_WORKSPACE_CONFIG’ is not recognized as an internal or external command, operable program, or batch file</em>. Is there a way to put such a command on a Windows terminal? Thanks in advance!</p>

username: ptrblck
cooked: <p>I’m not sure how exactly the “Windows terminal” works, but I think you can set env variables in Windows in the “Advanced system settings”.</p>

username: vickyx398
cooked: <p>Hi there,<br>
I want to perform Class Activation Mapping on my model.<br>
Setting <code>CUBLAS_WORKSPACE_CONFIG=:16:8</code> does not work for me.<br>
I am using an AdaptiveConcatPool1d Layer, which seems to not work deterministically at all. Am I right?<br>
Is there a way to get reproducible results for a Class Activation Map and still use that layer?<br>
I need it for my bachelor thesis…<br>
Thanks in advance!</p>

username: ptrblck
cooked: <p><code>AdaptiveConcatPool1d</code> doesn’t seem to be a (core) PyTorch layer, so I don’t know which operations are used internally and if they have a deterministic mode.<br>
In any case, did you set <code>torch.use_deterministic_algorithms(True)</code> and if so, did you receive an error?</p>

username: vickyx398
cooked: <p>Yes, I did. The error said to set <strong>CUBLAS_WORKSPACE_CONFIG=:4096:8</strong>, which didn’t change anything.<br>
However, I found a workaround by setting torch.manual_seed(0) before getting the feature maps and weights for the class activation map.</p>

username: dorbittonn
cooked: <p>Had the exact same case like you</p>

username: papoo13
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="102260">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>CUBLAS_WORKSPACE_CONFIG=:16:8 python script.py args</code></p>
</blockquote>
</aside>
<p>Is there an equivalent to writing this when I am using visual Studio Code. I need to somehow add this to my environment in the launch.json file but so far I haven’t been able to get it to work.</p>

username: ptrblck
cooked: <p>I’m not familiar enough with Visual Studio and how the <code>launch.json</code> file is used. You could try to set it in your Python script right at the beginning before importing torch and before creating any CUDA context. Note however, that users are often running into issues when trying to set env variables inside the script, as it can be too late and the context might have already been initialized.<br>
If you want to always enable it, you might just want to <code>export</code> it.</p>

username: papoo13
cooked: <p>Thank you very much. I try the first option and the check the results with different seeds to see if it has taken effect.<br>
I am not sure what you mean by exporting it. Could you please explain it a bit more?</p>

username: ptrblck
cooked: <p><code>export CUBLAS_WORKSPACE_CONFIG=:16:8</code> will set this env variable in your terminal and you wouldn’t need to add it to each command (similar to how e.g. <code>LD_LIBRARY_PATH</code> is set).</p>

username: papoo13
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="15" data-topic="102260">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>wouldn’t need to add it to each command</p>
</blockquote>
</aside>
<p>Thank you very much!</p>

username: papoo13
cooked: <p>One more question,<br>
I am running my code on a certain conda environment. should I export the variable when my conda environment is activated?</p>

username: ptrblck
cooked: <p>You can <code>export</code> it in your terminal before executing the Python script, but it doesn’t matter if the conda environment is already activated or not. Note that <code>export</code> will only set this env variable in the current terminal. If you want to save it for all terminals (and also to be active after a reboot) you can write it to your <code>.bashrc</code>.</p>

username: Walid_Moustafa
cooked: <p>Yes, there is.<br>
use:</p>
<pre><code class="lang-auto">import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"
</code></pre>