Matmul instability

amity137: <p>Hey folks, have been haunted by this for some time. Now have an isolated repro. c1 and c2 below should be absolutely equal below but that’s not the case. Can anyone please explain why? The behavior is same on CPU and GPU.</p>
<pre><code class="lang-auto">import torch

d1=10
d2=2
d3=50
d4=150

torch.set_printoptions(20)

e=torch.rand((d1,d2,d3))
f=torch.rand((d3,d4))
r1=torch.matmul(e.view(-1,d3),f).view(d1,d2,d4)
r2=torch.matmul(e[:,-1,:].unsqueeze(1).view(-1,d3),f).view(d1,1,d4)

c1=r1[:,-1,:]
c2=r2[:,-1,:]
print(torch.equal(c1,c2))
print(torch.max(torch.abs(c1-c2)))
print(torch.min(torch.abs(c1-c2)))
</code></pre>

tom: <p>That (1e-6-ish difference) the numerical precision of floating point. There isn’t much to do about it (unless you want to use double or “enhanced precision techniques” for particularly critical things when you get that to 1e-14ish).</p>
<p>Best regards</p>
<p>Thomas</p>

amity137: <p>Thanks Thomas. So are you saying that if we do a series of flops in the same order, we’d always get the same result (till 20 digits). But if we change the order of flops, while keeping the process identical logically, the final result would be different?</p>

Yaroslav_Bulatov: <p>You lose precision in matmul and could get different results even if you run your operations in the exact same order, here’s an explanation  – <a href="https://www.nag.com/content/wandering-precision" rel="nofollow noopener">https://www.nag.com/content/wandering-precision</a></p>
<p>You could use the following routine to find largest error of a matmul (see this <a href="https://github.com/pytorch/pytorch/issues/27016#issuecomment-536209410" rel="nofollow noopener">thread</a> for derivation)</p>
<pre><code class="lang-auto">def float32_matmul_error(a, b):
    a = a.type(torch.float64)
    b = b.type(torch.float64)
    n = max(a.shape+b.shape)
    u = np.finfo(np.dtype('float32')).eps
    gamma=(n*u)/(1-n*u)
    error=torch.norm(a.flatten())*torch.norm(b.flatten())*gamma
    return error
</code></pre>

KFrank: <p>Hi Amity!</p>
<aside class="quote no-group" data-username="amity137" data-post="3" data-topic="71101" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/amity137/48/12361_2.png" class="avatar"> amity137:</div>
<blockquote>
<p>So are you saying that if we do a series of flops in the same order, we’d always get the same result (till 20 digits). But if we change the order of flops, while keeping the process identical logically, the final result would be different?</p>
</blockquote>
</aside>
<p>Yes, this is true (for any reasonable definition of “do a series<br>
of flops in the same order,” and excluding perverse hardware<br>
implementations).</p>
<p>Consider this:</p>
<p>Mathematically, addition is <em>commutative</em>, i.e. <code>x + y = y + x</code>.<br>
It is also <em>associative</em>: <code>x + (y + z) = (x + y) + z</code>.</p>
<p>Floating-point addition can be (and should be) commutative.  But<br>
there is no reasonable way to make floating-point addition<br>
associative.</p>
<p><code>1.0 + ((-1.0) + 1.e-20) = 0.0</code> with floating-point arithmetic,<br>
while <code>(1.0 + (-1.0)) + 1.e-20 = 1.e-20</code>.  This is because<br>
<code>(-1.0) + 1.e-20 = (-1.0)</code>, as neither floats nor doubles have<br>
enough precision to represent <code>(-1.0) + 1.e-20</code> as a number<br>
different from <code>-1.0</code>.  There’s really no way around this.</p>
<p>So if you change the order of operations in <code>x + y + z</code>, you<br>
get the same answer mathematically, but not necessarily  with<br>
floating-point arithmetic.</p>
<p>Hello Yaroslav !</p>
<aside class="quote no-group quote-modified" data-username="Yaroslav_Bulatov" data-post="4" data-topic="71101">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yaroslav_bulatov/48/7017_2.png" class="avatar"> Yaroslav_Bulatov:</div>
<blockquote>
<p>You lose precision in matmul and could get different results even if you run instructions in the exact same order, here’s an explanation – <a href="https://www.nag.com/content/wandering-precision" rel="noopener nofollow ugc">https://www.nag.com/content/wandering-precision</a></p>
</blockquote>
</aside>
<p>This isn’t correct (at least for what one would normally mean by<br>
“run instructions in the exact same order”), and your nag reference<br>
doesn’t support this.  Quoting from your nag reference:</p>
<blockquote>
<p>The problem is that by messing with the order of the accumulations, you are quite possibly changing the final result, simply due to rounding differences when working with finite precision computer arithmetic.</p>
</blockquote>
<p>They talk about logic at a higher level (in this case the compiler)<br>
choosing to <em>change</em> the order of the floating-point operations,<br>
causing the result to change (at the level of floating-point precision).</p>
<p>Best.</p>
<p>K. Frank</p>

Yaroslav_Bulatov: <p>Sure, at some point there’s a change in what processor actually does. The NAG example is interesting because most people assume same same inputs + same binary + single thread = same arithmetic result, but instead the result depends on memory alignment which can change between invocations of the binary. Even harder to guarantee same result once your op starts using multiple cores</p>

amity137: <p>Thanks a lot folks for your detailed replies.</p>