Little difference in the result of sparse calculation and dense calculation

alireza7: <p>I do a matrix multiplication of  r1 = x*w1  when w1 is a dense tensor.</p>
<p>Then i create an sparse copy of  w1   : w2 = w1.to_sparse() and calculate : r2=x*w2</p>
<p>now r1 and r2 slightly differ</p>
<pre><code class="lang-auto">
x = torch.randn(700,500, dtype = torch.float32)


w2 = torch.rand(500,600 , dtype = torch.float32 )
w1 = w2.to_sparse()


print( torch.abs( w2 - w1).sum()   )
&gt;&gt;&gt;tensor(0.)
r1 = torch.sparse.mm( w1.t()  , x.t()  ).t()

r2= torch.mm(x , w2)


print( torch.abs(  (r2 - r1)  ).sum()   )
&gt;&gt;&gt;tensor(1.4281)
</code></pre>
<p>when i changed dtype to float64 this difference became low</p>
<pre><code class="lang-auto">x = torch.randn(700,500, dtype = torch.float64)


w2 = torch.rand(500,600 , dtype = torch.float64 )
w1 = w2.to_sparse()


print( torch.abs( w2 - w1).sum()   )
&gt;&gt;&gt;tensor(0., dtype=torch.float64)

r1 = torch.sparse.mm( w1.t()  , x.t()  ).t()

r2= torch.mm(x , w2)


print( torch.abs(  (r2 - r1)  ).sum()   )
&gt;&gt;&gt;tensor(2.6097e-09, dtype=torch.float64)
</code></pre>
<p>Why there is such a difference?<br>
Is there any considerable problem?<br>
How does it effect learning a network with sparse tensors?</p>

tom: <aside class="quote no-group" data-username="alireza7" data-post="1" data-topic="95189">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/919ad9/48.png" class="avatar"> alireza7:</div>
<blockquote>
<pre><code class="lang-auto">print( torch.abs(  (r2 - r1)  ).sum()   )
</code></pre>
</blockquote>
</aside>
<p>This can be expected. On an item by item basis, you have a typical floating point difference for fp32:</p>
<pre data-code-wrap="python"><code class="lang-python">(r1-r2).abs().max()
tensor(6.4850e-05)
</code></pre>
<p>As you have 4.2e5 items, you end up with a the difference you see. So everything is as expected here, but if you depend on the difference, you probably want fp64 for at least part of your computation.</p>
<p>Best regards</p>
<p>Thomas</p>

alireza7: <p>I have got that this is due to round-off error.<br>
but the thing is in the following code this error is 0.</p>
<pre><code class="lang-auto">
x = torch.randn(700,500, dtype = torch.float32)


w2 = torch.rand(500,600 , dtype = torch.float32 )
w1 = w2.to_sparse()
w1 = w1.to_dense()
w1 = w1.clone()
print( torch.abs( w2 - w1).sum()   )
&gt;&gt;&gt;tensor(0.)


r1= torch.mm(x , w1)
r2= torch.mm(x , w2)


print( torch.abs( 1* (r2 - r1)  ).sum()   )
&gt;&gt;&gt;tensor(0.)
</code></pre>
<p>The thing that I want to know is that what mechanism is underlying torch.sparse.mm() compare to torch.mm()  which cause this error?</p>
<p>As I know if we have a1=a2 (dtype = float32 ) and all 32 bits of a1 be identical to a2<br>
and also have b1=b2 (dtype = float32 ) and all 32 bits of b1 be identical to b2 and we do an identical floating-point operation by an FPU and get c1=a1*b1  and c2=a2*b2 we expect that c1=c2 and all 32 bits of c1 be identical to c2.</p>

tom: <p>Well, so summation results depend on the order of summation for floating point. The guarantees that you get the same order are very limited and non-existent between different implementations (or different number of available cores or …).</p>
<p>Best regards</p>
<p>Thomas</p>