Why is torch.mm non deterministic?

Megh_Bhalerao: <p>Hello,<br>
I get the following error (this error has been referenced many times on the internet, but putting it here anyways) -</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 79, in &lt;module&gt;
    main()
  File "main.py", line 69, in main
    distillation = Distillation(train_loader, train_dataset, model_wt_path, config_dict, args.seed) # passing the mean and stddev of the dataset for the images, this is more specific to images when done in this context
  File "/x0/megh98/projects/ddist/distillation.py", line 44, in __init__
    train_acc = self.run_validation(self.train_loader)
  File "/x0/megh98/projects/ddist/distillation.py", line 66, in run_validation
    logits = self.net(img)
  File "/x0/megh98/anaconda3/envs/ddist/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/x0/megh98/projects/ddist/models/networks.py", line 67, in forward
    out = self.classifier(out)
  File "/x0/megh98/anaconda3/envs/ddist/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/x0/megh98/anaconda3/envs/ddist/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: Deterministic behavior was enabled with either `torch.use_deterministic_algorithms(True)` or `at::Context::setDeterministicAlgorithms(true)`, but this operation is not deterministic because it uses CuBLAS and you have CUDA &gt;= 10.2. To enable deterministic behavior in this case, you must set an environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or CUBLAS_WORKSPACE_CONFIG=:16:8. For more information, go to https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility
</code></pre>
<p>when I set - <code>torch.use_deterministic_algorithms(True)</code>. This is no surprise. When I went to the docs here - <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html" class="inline-onebox" rel="noopener nofollow ugc">torch.use_deterministic_algorithms — PyTorch 1.12 documentation</a> , it mentions that <code>torch.mm</code> is a non deterministic operation. So, I have the following 2 questions -</p>
<ol>
<li>Why would a simple matrix multiplication such as <code>torch.nn</code> be a non-deterministic operation?</li>
<li>From the error traceback it seems like the non-determinism is coming from the <code>F.linear</code> function, which after looking at the pytorch GitHub seems to use torch.matmul which inturn uses torch.mm atleast according to my understanding - and this is what is leading to the non determinism. Am I right in my understanding here?<br>
Thanks!</li>
</ol>

smth: <p>the reasoning is given in both the error message and the documentation and it points to this link: <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility" class="inline-onebox">cuBLAS :: CUDA Toolkit Documentation</a></p>

Megh_Bhalerao: <p>Thanks <a class="mention" href="/u/smth">@smth</a> - yes, I did read the explanation in the link earlier, but it seems like the explanation is a general one where they mention stuff about cuda streams etc. But, I wanted to know what is specific about the <code>torch.mm</code> function that makes it non-deterministic, is it related the explanation given here - <a href="https://docs.nvidia.com/cuda/cublas/index.html#batching-kernels" class="inline-onebox" rel="noopener nofollow ugc">cuBLAS :: CUDA Toolkit Documentation</a> - is it something related to the parallelism of the matrix multiplies?</p>

smth: <p>yes, the non determinism comes from torch.mm calling cublas and cublas being non deterministic in these cases</p>

Megh_Bhalerao: <p>Hi, sorry to open this issue again, but I am still not completely clear as to why torch.mm is non deterministic. I understand that there are some backend cuda operations that are non deterministic, but a matrix multiply must be deterministic right? Other wise if I do AB where A and B are matrices, and I do the same operation again, I will get 2 different results which does not make sense?<br>
Am I missing some crucial point here? Please let me know and thanks again for your time.</p>

ptrblck: <p>Depending on the algorithm and its implementation a matmul kernel could be non-deterministic. Especially if you are using workspaces to stored intermediate results, algorithms using split-k, etc.</p>