Why matrix multiplication returns inconsistent results with different sizes?

meowmix: <p>When I run the following code</p>
<pre><code class="lang-auto">a = torch.normal(0,1,(2,5000))
b = torch.normal(0,1,(5000,17))
(a@b)[:, :5] - (a@b[:,:5])
</code></pre>
<p>I get the output<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/946e56718e32f7efec510070ee61c755940a0cce.png" data-download-href="https://discuss.pytorch.org/uploads/default/946e56718e32f7efec510070ee61c755940a0cce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/946e56718e32f7efec510070ee61c755940a0cce_2_690x122.png" alt="image" data-base62-sha1="lb53U9o7RWFqhBCA77DRelKHAXc" width="690" height="122" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/946e56718e32f7efec510070ee61c755940a0cce_2_690x122.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/946e56718e32f7efec510070ee61c755940a0cce_2_1035x183.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/4/946e56718e32f7efec510070ee61c755940a0cce.png 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1288×228 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In other words, multiplying two matrices and taking a smaller portion of the result isn’t the same as multiplying the first matrix with the corresponding smaller portion of the second matrix, when mathematically it should be the same.</p>
<p>I noticed that for the above example, the errors persist with subsetting different numbers of columns (i.e. other than 5) until 13, at which point the errors disappear.</p>
<p>This is problematic for me because I am trying to implement a version of attention that performs scatter operations to avoid needing padding all input sequences to the same length. Instead of a batched matrix multiply on each individual sequence, I perform a matrix multiply on a matrix containing the sequences concatenated together. But because of this error, I cannot reproduce pytorch’s attention to within the tolerance I desire, (1e-4 max discrepancy between mine and pytorch attention’s outputs with same random inputs/parameters)</p>

ptrblck: <p>The errors are expected due to the limited floating point precision. Different algorithms can be picked for different input shapes, which might then change the order of operation and cause these numerical mismatches.</p>

Chard_Lee: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> , I just ask a quick question regarding this matter. So, this issue cannot be fixed even though I follow this reproducibility document (<a href="https://pytorch.org/docs/stable/notes/randomness.html" class="inline-onebox" rel="noopener nofollow ugc">Reproducibility — PyTorch 2.6 documentation</a>)?</p>
<p>Specifically, setting like:<br>
export CUBLAS_WORKSPACE_CONFIG=:16:8<br>
torch.manual_seed(0)<br>
np.random.seed(0)<br>
random.seed(0)<br>
torch.backends.cudnn.benchmark = False<br>
torch.use_deterministic_algorithms(True)<br>
torch.utils.deterministic.fill_uninitialized_memory = True</p>

ptrblck: <p>You can achieve deterministic reproducibility for the <em>same</em> workloads in PyTorch. However, changing e.g. the batch size creates a new workload which will return reproducible results when executed again, but it’s not guaranteed it will return bitwise identical results to other workloads (e.g. with different batch sizes).</p>

Chard_Lee: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , I learnt lots from you!</p>