Inconsistent reproducibility

username: louisabraham
cooked: <p>I have a deterministic code that gives the same result between runs on the same machine.</p>
<p>I tried to launch it on two machines:</p>
<ul>
<li>my macbook pro m1 with intel python 3.8 and torch=1.9.0</li>
<li>google colab with python 3.7 and torch=1.9.0+cu102</li>
</ul>
<p>It gives different results in most cases.</p>
<p>However, if I set the optimizer to Adam and use 10 epochs at most, the results are the same.<br>
Even weirder, if I use CUDA on colab (which needs <code>torch.use_deterministic_algorithms(False)</code> but the code still gives the same results every time), then I can do 109 epochs at most that give the same results between the two machines.<br>
If I use SGD, then using CPU on colab works for 9 epochs at most and GPU doesn’t work when any epoch is done.</p>
<p>I couldn’t make a MRO, <a href="https://colab.research.google.com/drive/1r-IjlK3Be64zAlUq8MlPXgSdwd_rucF1?usp=sharing" rel="noopener nofollow ugc">here is the code</a>. It is hard to dive down because I make most operations using a package. But anyone can probably try to run it and report their results that are different. Also, when relaunching I cannot get the outputs to match as I explain above.</p>
<p>The notebook gives <code>rrmse: 1.179134459111933</code> and my computer gives <code>rrmse: 1.1791344531807701</code>.</p>
<p>Does anyone have an idea?</p>

username: ptrblck
cooked: <p>There is no guarantee to get deterministic results on different hardware architectures, since e.g. the pseudorandom number generator implementation could be different.</p>
<aside class="quote no-group" data-username="louisabraham" data-post="1" data-topic="131335">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/louisabraham/48/20573_2.png" class="avatar"> louisabraham:</div>
<blockquote>
<p>(which needs <code>torch.use_deterministic_algorithms(False)</code></p>
</blockquote>
</aside>
<p>I don’t quite understand this claim. Since you are not using deterministic algorithms, you shouldn’t expect to get deterministic results.</p>

username: louisabraham
cooked: <p>Oh, right, I didn’t read the <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">reproducibility docs</a> right:</p>
<blockquote>
<p>Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different <strong>platforms</strong></p>
</blockquote>
<p>About <code>use_deterministic_algorithms</code>, here is my traceback. Looks like linear is non deterministic on some cuda versions:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     94 
     95     def forward(self, input: Tensor) -&gt; Tensor:
---&gt; 96         return F.linear(input, self.weight, self.bias)
     97 
     98     def extra_repr(self) -&gt; str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1845     if has_torch_function_variadic(input, weight):
   1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-&gt; 1847     return torch._C._nn.linear(input, weight, bias)
   1848 
   1849 

RuntimeError: Deterministic behavior was enabled with either `torch.use_deterministic_algorithms(True)` or `at::Context::setDeterministicAlgorithms(true)`, but this operation is not deterministic because it uses CuBLAS and you have CUDA &gt;= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility
</code></pre>
<p>Anyway, you already answered my main question, thanks!</p>

username: ptrblck
cooked: <p>Yes, as the error message explains you would need to set the cublas workspace sizes to these fixed values to get deterministic results in newer cublas releases.</p>