Two tensors produced from the '@' operation are not eqaul under the measurement of '.eq()'

Alex_Shen: <p>Hi, I’m quite new to PyTorch and I came cross the ‘@’ operation, which does matrices multiplication for the last two dimensions. (referrence: <a href="https://discuss.pytorch.org/t/how-does-the-sign-work-in-this-instance/11232"> How does the @ sign work in this instance?</a>).</p>
<p>As far as I understand, given two tensors A (bsz, d1, d2), and B (bsz, d2, d3),  (A@B)[i] should equal to (A[i])@(B[i]); thus, these two results from ‘@’ should be identical when call the ‘.eq()’ measurement (a simple code exmaple is shown below with i=0).</p>
<pre><code class="lang-auto">import torch
torch.set_printoptions(profile="full")

A = torch.rand(16, 3, 40)
B = torch.rand(16, 40, 3)

C = (A@B)[0]
D = (A[0])@(B[0])

print('C=(A@B)[0] is:')
print(C)
print('D=(A[0])@(B[0]) is:')
print(D)
print('C.eq(D) is:')
print(C.eq(D))
</code></pre>
<p>However, what I got from the ‘C.eq(D)’ line was not as I expected to be all ‘True’, and the printed values of C, D are identical as I expected.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3f2bbdfed3d625e62ed8375fce837997b3ed367.png" alt="RK7W83FM%`IHT%VP$}OBFU" data-base62-sha1="wwwrkdu4wHzL8E2c8ABsVqknOS3" width="278" height="210"><br>
I understand comparing the values with our raw eyes is not the best way to decide whether two tensors are equal, but the results of ‘.eq()’ is really confusing.</p>
<p>And I find that when I declare A, B to be <strong>integers</strong>, C=(A@B)[0] always equals to D=(A[0])@(B[0]) under ‘.eq()’ measurement, so here are my questions:</p>
<ol>
<li>
<p>why C.eq(D) is <strong>not</strong> all ‘True’ when declaring A, B to be <strong>float</strong>? And is all ‘True’ if A, B declared as <strong>integers</strong>?</p>
</li>
<li>
<p>Regarding Q1, I suspect it has something to do with the datatypes, however I can’t find anything related to this difference on PyTorch Docs, help on this perspective is very much appreciated!</p>
</li>
<li>
<p>Regaiding Q3, if the ‘.eq()’ behaviour is caused by the nature of datatype <strong>float</strong> vs. <strong>integer</strong>, why the printed values, shown in the picture, are the same from raw eye observation (especially after I called: <em>torch.set_printoptions(profile=“full”)</em>)?</p>
</li>
</ol>
<p>I’m using PyTorch 1.12.0 and thanks in avdance!</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="Alex_Shen" data-post="1" data-topic="161622" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_shen/48/49123_2.png" class="avatar"> Alex_Shen:</div>
<blockquote>
<p>given two tensors A (bsz, d1, d2), and B (bsz, d2, d3),  (A@B)[i] should equal to (A[i])@(B[i])</p>
</blockquote>
</aside>
<p>Because of floating-point round-off error, this will not be true in general.</p>
<p>The two computations are <em>mathematically</em> equivalent, but pytorch chooses<br>
to perform various arithmetic operations in differing orders that can lead to<br>
differing round-off error.</p>
<aside class="quote no-group">
<blockquote>
<ol>
<li>why C.eq(D) is <strong>not</strong> all ‘True’ when declaring A, B to be <strong>float</strong>? And is all ‘True’ if A, B declared as <strong>integers</strong>?</li>
</ol>
</blockquote>
</aside>
<p>Integer operations don’t suffer the same round-off error that floating-point<br>
operations do.</p>
<aside class="quote no-group">
<blockquote>
<ol start="3">
<li>Regaiding Q3, if the ‘.eq()’ behaviour is caused by the nature of datatype <strong>float</strong> vs. <strong>integer</strong>, why the printed values, shown in the picture, are the same from raw eye observation (especially after I called: <em>torch.set_printoptions(profile=“full”)</em>)?</li>
</ol>
</blockquote>
</aside>
<p>You will need to print the values out to close to machine accuracy to see the<br>
difference.  Try <code>torch.set_printoptions (precision = 20)</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Alex_Shen: <p>Hi Frank!</p>
<p>Thank you very much on the explanation of the round-off error, and I indeed noticed the difference after <em>torch.set_printoptions (precision = 20)</em>.</p>
<p>However I’m still a little confused about the statement that: “<em><strong>pytorch chooses to perform various arithmetic operations in differing orders that can lead to differing round-off error</strong></em>”. I agree that the different in the orders of operations can lead to different round-offs; thus, inequality when calling ‘.eq()’, but I can’t understand why processing in batch can lead to such differences (the code below is a pseudocode that I think how ‘@’ deals with batch).</p>
<pre><code class="lang-auto">given: A=(bsz, d1, d2), B=(bsz, d2, d3), we are to calculate A@B
init: C=()
for i in (1, 2, ..., bsz):
    C.append(A[i]@B[i])
return C
</code></pre>
<p>From my understanding, I think when ‘@’ deals with batching, it just computes ‘A[i]<span class="mention">@B</span>[i]’ bsz times and stacks the results together, and thus should yield identical outcomes for ‘A[i]<span class="mention">@B</span>[i]’ and ‘A@B[i]’. However as this is not the truth in practice, can you please tell me where I got it wrong on how ‘@’ deals with batching? Thx in advance!</p>

KFrank: <p>Hi Alex!</p>
<aside class="quote no-group" data-username="Alex_Shen" data-post="3" data-topic="161622" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_shen/48/49123_2.png" class="avatar"> Alex_Shen:</div>
<blockquote>
<p>I think when ‘@’ deals with batching, it just computes ‘A[i]<span class="mention">@B</span>[i]’ bsz times and stacks the results together</p>
</blockquote>
</aside>
<p>Conceptually this is a good way to picture what is happening, but<br>
it doesn’t match the exact details of pytorch’s actual computation.</p>
<p>Under the hood pytorch has a lot of optimized code to make sure<br>
it uses the floating-point pipelines as efficiently as possible.  (Note,<br>
modern cpus have floating-point pipelines, so these comments apply<br>
to cpus as well as gpus.)</p>
<p>The pytorch code will often reorder the computation in order to keep<br>
the floating-point pipelines as full as possible when, for example, the<br>
size of the input changes.  In your example, you are moving from a<br>
batch matrix multiply to a single matrix multiply.  This could easily<br>
trigger such a reordering.</p>
<p>You will also see this kind of reordering when you move from the cpu<br>
to gpu, when you move from one hardware platform to another, when<br>
you change pytorch versions, and so on.  (You will also see this when<br>
you move from single precision to double precision, although this is less<br>
likely to be noticed because the single-precision and double-precision<br>
results won’t be exactly the same even when the computations are<br>
performed in the same order.)</p>
<p>Best.</p>
<p>K. Frank</p>

Alex_Shen: <p>Hi Frank,</p>
<p>Thank you very much for your responsive and detailed reply! Now I get to know a little deeper about the mechanics of how the computation is done, which is very helpful for me to understand some other inconsistencies!</p>