The sum() function is inaccurate when the CPU evaluates the float32 type with precision

username: Zebulun
cooked: <p>version: pytorch==1.3.0<br>
code：</p>
<pre><code class="lang-auto">import torch
input = torch.randn((2, 128, 10, 6), dtype=torch.float32)
out = input.sum()
print("%3.10f" % out.data)
&lt;&lt; 0.0181007385
</code></pre>
<p>Compute comparison tests using the same data：<br>
sum：0.0179971529<br>
numpy： 0.017990112.  (RE=0.04%)<br>
torch cuda：0.0179862976.  (RE=0.06%)<br>
torch cpu：0.0181007385  （RE=0.57%）</p>
<p>The error of summation of float32 type data is normal, and the precision is close compared with other computed data. However, the calculation results of the Torch CPU vary greatly. How to determine the accuracy of the data?</p>
<p>Is it because the accumulated errors are processed by other computing libraries, while the Torch CPU does not have higher errors? Or is the processing of TorchCPU a special process?<br>
Where can I see the implementation of the cumulative calculation?</p>

username: utkuumetin
cooked: <p>Hi! I’m not very familiar and good with the subject, but maybe this issue page can help you:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/18427" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/18427" target="_blank" rel="noopener nofollow ugc">the acc of torch.float32 is not good</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-03-25" data-time="11:50:22" data-timezone="UTC">11:50AM - 25 Mar 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-03-25" data-time="11:56:29" data-timezone="UTC">11:56AM - 25 Mar 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/sun7-she" target="_blank" rel="noopener nofollow ugc">
          <img alt="sun7-she" src="https://avatars3.githubusercontent.com/u/18720482?v=4" class="onebox-avatar-inline" width="20" height="20">
          sun7-she
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">environment:
python3
pytorch 0.4.1
run the code1:
import torch
import numpy as np
n=4
c=3
h=32
w=32
inputs = torch.FloatTensor(n,c,h,w).fill_(0)
inputs = np.asarray(inputs)
inputs_sum = 0.0
for j in range(c):
 for k in range(h):
...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: KFrank
cooked: <p>Hi Zebulun!</p>
<aside class="quote no-group" data-username="Zebulun" data-post="1" data-topic="105223" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zebulun/48/24958_2.png" class="avatar"> Zebulun:</div>
<blockquote>
<p>Compute comparison tests using the same data：<br>
sum：0.0179971529<br>
numpy： 0.017990112.  (RE=0.04%)<br>
torch cuda：0.0179862976.  (RE=0.06%)<br>
torch cpu：0.0181007385  （RE=0.57%）</p>
</blockquote>
</aside>
<p>As Utku suggested with his link, the imprecision you see is expected<br>
as it is consistent with 32-bit floating-point round-off error.</p>
<p>You are summing over about 15,000 normally distributed values, so<br>
the expected size of your sum is about 124.  Let’s treat your numpy<br>
result as the exact result.  Then your cpu result is off in the fourth<br>
place after the decimal point, so relative to the expected value, this<br>
is fully consistent with round-off error.</p>
<p>However, there is something fishy going on with your numbers.  The<br>
value you get for your sum is <em>improbably</em> small.  As I mentioned<br>
above, the expected size of your sum is about 124, so a value of<br>
0.018 is <em>very</em> unlikely – about one chance in ten thousand.  I can’t<br>
think of any good reason this should happen, and it would be pushing<br>
your luck to suggest that luck explains it.</p>
<p>Here is a script that goes through the calculations relevant to these<br>
two points:</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np

tdim = (2, 128, 10, 6)
sumcpu = 0.0181007385
sumnpy = 0.017990112

nSum = np.product (tdim)
expected_size = np.sqrt (nSum)

relerr = (sumcpu - sumnpy) / expected_size

print ('nSum =', nSum, ', expected_size =', expected_size)
print ('relerr =', relerr)

import math

prob_small = math.erf (abs (sumnpy) / (math.sqrt (2.0) * expected_size))
print ('prob_small =',  prob_small)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt;
&gt;&gt;&gt; tdim = (2, 128, 10, 6)
&gt;&gt;&gt; sumcpu = 0.0181007385
&gt;&gt;&gt; sumnpy = 0.017990112
&gt;&gt;&gt;
&gt;&gt;&gt; nSum = np.product (tdim)
&gt;&gt;&gt; expected_size = np.sqrt (nSum)
&gt;&gt;&gt;
&gt;&gt;&gt; relerr = (sumcpu - sumnpy) / expected_size
&gt;&gt;&gt;
&gt;&gt;&gt; print ('nSum =', nSum, ', expected_size =', expected_size)
nSum = 15360 , expected_size = 123.93546707863734
&gt;&gt;&gt; print ('relerr =', relerr)
relerr = 8.92613733644216e-07
&gt;&gt;&gt;
&gt;&gt;&gt; import math
&gt;&gt;&gt;
&gt;&gt;&gt; prob_small = math.erf (abs (sumnpy) / (math.sqrt (2.0) * expected_size))
&gt;&gt;&gt; print ('prob_small =',  prob_small)
prob_small = 0.00011581860221173588
</code></pre>
<p>Two things:</p>
<p><code>relerr = 8.92613733644216e-07</code></p>
<p>32-bit round-off error is about 1.e-7.  You are summing over about<br>
15,000 values of order one, so the round-off error could easily<br>
accumulate to this level.</p>
<p><code>prob_small = 0.00011581860221173588</code></p>
<p>When you sum over 15,000 (independent) normally distributed values<br>
(so each value has standard deviation one), the sum comes from a<br>
gaussian distribution that has a standard deviation of about 124 (and<br>
a mean of zero).  You can calculate the probability of such a sum<br>
having a magnitude smaller that 0.018 using the so-called <em>error<br>
function,</em> and that probability is very small – about 1% of 1%.</p>
<p>Best.</p>
<p>K. Frank</p>