Independence of matrix rows in matmul

username: yiftach
cooked: <p>Hi,<br>
According to linear algebra, the result of multiplying a matrix by a matrix can be computed separately by rows (or by columns). Why are then the two following result not exactly equivalent? Do I need to enable any other flag/choose a different backend?</p>
<pre><code class="lang-auto">import random
import os
import numpy as np
import torch

seed = 42
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = True

model = torch.randn(1000,1000)
inputs = torch.randn(2,1000)
with torch.inference_mode():
    a = (inputs @ model)[0]
    b = inputs[0] @ model
    print(torch.all(torch.isclose(a,b , 1e-3)))  # True
    print(torch.all(torch.isclose(a,b , 1e-4)))  # False
</code></pre>

username: ptrblck
cooked: <p>Different algorithms can be picked depending on the input shape of your workload.<br>
My system returns the same results, but your CPU might take an optimized code paths for one of the operations. Generally you cannot expect to see bitwise-identical results due to the limited floating point precision and would need to set <code>torch.use_deterministic_algorithms</code> to <code>True</code> if needed.</p>

username: yiftach
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="175598">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.use_deterministic_algorithms</p>
</blockquote>
</aside>
<p>Adding <code>torch.use_deterministic_algorithms(True)</code> does not change the results I’m getting, but increasing floating point precision does make both equal (I’m running on a colab notebook for reproducibility).<br>
Any explanation for why that would be the case? I would think both accumulations (for computing the first output and the second output) happen independently, but apparently not the same thing is happening (even apparently without any non-deterministic algorithms).</p>

username: ptrblck
cooked: <p>Sorry for the unclear description. Enabling deterministic algorithms should return bitwise-identical and deterministic results for the <em>same workload</em> which is defined by e.g. the shapes of the used tensors. Different algorithms can still be picked for differently shaped tensors and a small example also shows this for simple reductions:</p>
<pre><code class="lang-python">x = torch.randn(100, 100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0).sum(0)
print(s1 - s2)
# tensor(0.0002)
</code></pre>

username: yiftach
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, I appreciate your answers.<br>
What surprises me and to my understanding differentiates between both examples, is that in the original example the two computations, at least in theory, do not even share any intermediate result (theoretically none of the scalars generated while computing <code>(inputs @ model)[1]</code> takes part in the computation of <code>(inputs @ model)[0]</code> so I do not see where the inconsistency comes from.<br>
In the example you give, I can guess that the difference results from a different order of accumulating the coordinates of <code>x</code> (but I could be wrong here as well) - is that what happens in my example as well? That regardless of the additional <code>inputs @ model)[1]</code>, the different coordinates of <code>inputs @ model)[0]</code> are accumulated in a different order than they are in the <code>inputs[0] @ model</code>, resulting in a different rounding error?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="yiftach" data-post="5" data-topic="175598">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yiftach/48/58678_2.png" class="avatar"> yiftach:</div>
<blockquote>
<p>(theoretically none of the scalars generated while computing <code>(inputs @ model)[1]</code> takes part in the computation of <code>(inputs @ model)[0]</code> so I do not see where the inconsistency comes from.</p>
</blockquote>
</aside>
<p>But this is also not what you are comparing.<br>
In your example you are executing a large workload, slicing it afterwards, and a small workload by slicing the inputs:</p>
<pre data-code-wrap="python"><code class="lang-python">a = (inputs @ model)[0]
b = inputs[0] @ model
</code></pre>
<p>Both approaches could take different algorithms, which can use a different order of operations, which will result in the expected error due to the limited floating point precision.</p>
<p>I would recommend to profile the code and to verify my claims:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.randn(1000,1000).cuda()
inputs = torch.randn(2,1000).cuda()

a = (inputs @ model)[0]
b = inputs[0] @ model
print((a - b).abs().max())
# tensor(2.8610e-05, device='cuda:0')


from torch.profiler import profile, record_function, ProfilerActivity


with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    a = (inputs @ model)[0]
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
#                                                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
#                                            aten::matmul         0.41%       9.000us        98.46%       2.175ms       2.175ms       0.000us         0.00%      21.000us      21.000us             1  
#                                                aten::mm        85.60%       1.891ms        98.05%       2.166ms       2.166ms      21.000us       100.00%      21.000us      21.000us             1  
# void gemmSN_NN_kernel&lt;float, 256, 4, 2, 8, 2, 4, fal...         0.00%       0.000us         0.00%       0.000us       0.000us      21.000us       100.00%      21.000us      21.000us             1  
# cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFla...        11.18%     247.000us        11.18%     247.000us     247.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                        cudaLaunchKernel         1.27%      28.000us         1.27%      28.000us      28.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                            aten::select         1.00%      22.000us         1.09%      24.000us      24.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                        aten::as_strided         0.09%       2.000us         0.09%       2.000us       2.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                   cudaDeviceSynchronize         0.45%      10.000us         0.45%      10.000us      10.000us       0.000us         0.00%       0.000us       0.000us             1  
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
# Self CPU time total: 2.209ms
# Self CUDA time total: 21.000us


with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], record_shapes=True) as prof:
    b = inputs[0] @ model
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
#                                                    -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
#                                                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls  
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
#                                            aten::matmul         0.17%      19.000us        97.55%      10.962ms      10.962ms       0.000us         0.00%      10.000us      10.000us             1  
#                                                aten::mm         0.61%      68.000us        97.20%      10.922ms      10.922ms      10.000us       100.00%      10.000us      10.000us             1  
# std::enable_if&lt;!(false), void&gt;::type internal::gemvx...         0.00%       0.000us         0.00%       0.000us       0.000us      10.000us       100.00%      10.000us      10.000us             1  
#                                            aten::select         0.18%      20.000us         0.21%      24.000us      24.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                        aten::as_strided         0.04%       4.000us         0.04%       4.000us       2.000us       0.000us         0.00%       0.000us       0.000us             2  
#                                         aten::unsqueeze         0.07%       8.000us         0.07%       8.000us       8.000us       0.000us         0.00%       0.000us       0.000us             1  
# cudaOccupancyMaxActiveBlocksPerMultiprocessorWithFla...         0.04%       5.000us         0.04%       5.000us       5.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                        cudaLaunchKernel        96.55%      10.849ms        96.55%      10.849ms      10.849ms       0.000us         0.00%       0.000us       0.000us             1  
#                                          aten::squeeze_         0.08%       9.000us         0.12%      13.000us      13.000us       0.000us         0.00%       0.000us       0.000us             1  
#                                       aten::as_strided_         0.04%       4.000us         0.04%       4.000us       4.000us       0.000us         0.00%       0.000us       0.000us             1  
# -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  
# Self CPU time total: 11.237ms
# Self CUDA time total: 10.000us
</code></pre>
<p>As you can see two different kernels are used: <code>gemmSN_NN_kernel</code> and <code>internal::gemvx</code>.</p>

username: yiftach
cooked: <p>This is again very useful, I did not know you could see these internal calls with a profiler and this is exactly what I was missing.<br>
Thanks a lot!</p>