A question about the accuracy of enisum calculation

I-Love-U: <pre><code class="lang-python"># 1
a = torch.rand(4, 5, 6, 7)
b = torch.rand(4, 6, 7, 9, 6, 7)
torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b))
tensor(False)

# 2
a = torch.randint(low=1, high=10, size=(4, 5, 6, 7))
b = torch.randint(high=4, size=(4, 6, 7, 9, 6, 7))
torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b))
tensor(True)

# 3
a = torch.rand(4, 5, 6, 7)
b = torch.rand(4, 6, 7, 9, 6, 7)
c = torch.bmm(a.reshape(4, 5, 6*7), b.reshape(4, 6*7, -1)).reshape(4, 5, 9, 6, 7).sum(dim=2)
torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == c)
tensor(True)
torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b))
tensor(False)
torch.all(torch.isclose(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)), torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b)))
tensor(True)
</code></pre>
<p>I want to use the code <code>torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b)</code>, but there is a strange result.<br>
Its result is different from <code>torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b))</code> when the tensors are the float type. I think this is due to the error of floating point calculation…<br>
However, for the float tensors, the result of <code>torch.bmm(a.reshape(4, 5, 6*7), b.reshape(4, 6*7, -1)).reshape(4, 5, 9, 6, 7).sum(dim=2)</code> is the same as <code>torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b))</code>. I wonder <strong>why this is not affected by floating-point computing</strong>.</p>

InnovArul: <aside class="quote no-group" data-username="I-Love-U" data-post="1" data-topic="106090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i-love-u/48/8664_2.png" class="avatar"> I-Love-U:</div>
<blockquote>
<pre><code class="lang-auto">torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == torch.einsum('bdhw,bhwnxy-&gt;bdxy', a, b))
tensor(False)
</code></pre>
</blockquote>
</aside>
<p>While comparing <code>float/double</code> tensors, use <code>torch.allclose()</code>, instead of <code>==</code>.</p>
<p>Another related note about float vs. double:</p><aside class="quote quote-modified" data-post="6" data-topic="106075">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/discrepancy-between-theory-and-practice/106075/6">Discrepancy between theory and practice</a> 
  </div>
  <blockquote>
    Hi, 
Running on current colab, this is what I see: 
The same thing as you: a difference of ~1e-6 difference for float. But adding torch.set_default_dtype(torch.double) at the beginning, it goes down to ~1e-15. <a class="mention" href="/u/innovarul">@InnovArul</a> did you set it properly to double? 
So it looks like it is expected loss of precision because of floating point numbers. 
Also note that the deeper your network is gonna be, the larger this difference is gonna be as most operation amplify a small difference that happened at the …
  </blockquote>
</aside>

<p>By default, pytorch data type is <code>torch.float</code>. The absolute tolerance level for float operations is around <code>10^-6</code>. If you set default data type to <code>double</code> using <code>torch.set_default_dtype(torch.double)</code>, the tolerance level is <code>&lt;= 10^-15</code>.</p>

I-Love-U: <p>Thanks for your reply.<br>
This does work, but I wonder why the two instructions still have the same result without considering the accuracy error.</p>
<pre><code class="lang-python">c = torch.bmm(a.reshape(4, 5, 6*7), b.reshape(4, 6*7, -1)).reshape(4, 5, 9, 6, 7).sum(dim=2)
torch.all(torch.einsum('bdnxy-&gt;bdxy', torch.einsum('bdhw,bhwnxy-&gt;bdnxy', a, b)) == c)
tensor(True)
</code></pre>