Numerical error between batch and single instance computation

smutahoang: <p>I found that there is some insignificant numerical error between the result obtained from batch mode computation and those obtained from iterating over the instances in the batch. For example:</p>
<pre><code class="lang-auto">batch_size = 32
input_size = 128
output_size = 256
linear = torch.nn.Linear(input_size,output_size)
x = torch.rand(batch_size, input_size)
output = linear(x)
output1 = linear(x[:1])
output2 = linear(x[:(batch_size//2)])

print(10e6*torch.max(torch.abs(output[:1] - output1)))
print(10e6*torch.max(torch.abs(output[:(batch_size//2)] - output2)))
</code></pre>
<p>The above code may print out something like:</p>
<pre><code class="lang-auto">tensor(1.4901, grad_fn=&lt;MulBackward&gt;) # a non-zero value, i.e., there is some numerical error
tensor(0., grad_fn=&lt;MulBackward&gt;) # a zero value, i.e, there is no error
</code></pre>
<p>Where does the above error come from? Any help is highly appreciated.</p>

albanD: <p>Hi,</p>
<p>The precision of a floating point number is around 1e-6. So anything smaller is going to be noise.<br>
Also floating point operations are not really commutative or associative and will create very small error if you do them in a different order.<br>
So this is expected behavior I’m afraid. You can use double precision numbers if you require more precision.</p>

smutahoang: <p>Thanks for the great answer.</p>

KFrank: <p>Hi Tuan Anh (and Alban)!</p>
<aside class="quote no-group quote-modified" data-username="smutahoang" data-post="1" data-topic="56735" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smutahoang/48/16189_2.png" class="avatar"> smutahoang:</div>
<blockquote>
<p>I found that there is some insignificant numerical error between the result obtained from batch mode computation and those obtained from iterating over the instances in the batch.</p>
<p>…</p>
<p>Where does the above error come from? Any help is highly appreciated.</p>
</blockquote>
</aside>
<p>I do find this a bit odd.</p>
<p>It is true, as Alban points out, you can’t really complain and call<br>
this an error.  The results of the two versions of the computation<br>
do agree to within 32-bit floating-point round-off error.</p>
<p>But floating-point computations don’t just give you random<br>
differences (and they’re not allowed to).  Again, as Alban notes,<br>
doing a floating-point computation in a different (but mathematically<br>
equivalent) order can lead to a round-off error difference in the result.</p>
<p>(A minor quibble:  Floating-point operations are not associative,<br>
but are commutative – at least in any sensible world, e.g., IEEE.)</p>
<p>My problem is that I can’t cook up any good reason the two versions<br>
of the computation should be being done in different orders.</p>
<p>For a little more fun, I tweaked your sample code to run on my decrepit<br>
pytorch 0.3.0 installation, and added to it a few more tests.</p>
<p>The two highlights</p>
<p>When done with 64-bit doubles, no difference arises.</p>
<p>When done with 32-bit floats, a difference only arises when batches<br>
of size 1, 2, and 3 are passed into <code>linear</code>.</p>
<p>(When run on the gpu instead of the cpu I get very similar results,<br>
except that the non-zero difference only shows up for a batch of<br>
size 1.)</p>
<p>My complete script and output for these tests appear below.</p>
<p>Now it is true that our complaint will not stand up in a court of<br>
floating-point law, but I am curious where this behavior comes<br>
from and if there is really any good reason for it.  Perhaps some<br>
experts could chime in with more insight.</p>
<p>There must be a sensible reason, no?</p>
<p>Surely pytorch is not – dare I say it? – random …</p>
<p>Best.</p>
<p>K. Frank</p>
<p>Script:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

torch.manual_seed (2019)

gpu = False
# gpu = True

print ('gpu =', gpu)

batch_size = 32
input_size = 128
output_size = 256
linear = torch.nn.Linear(input_size,output_size)
x = torch.autograd.Variable (torch.rand(batch_size, input_size))
if gpu:
    linear.cuda()
    x = x.cuda()
output = linear(x)
output1 = linear(x[:1])
output2 = linear(x[:(batch_size//2)])

print(10e6*torch.max(torch.abs(output[:1] - output1)))
print(10e6*torch.max(torch.abs(output[:(batch_size//2)] - output2)))

for  n in range (1, 5):
    outputn = linear (x[:n])
    print ('n =', n, ', diff =',10e6*torch.max(torch.abs(output[:1] - outputn[:1])))
    print ('n =', n, ', diff =',10e6*torch.max(torch.abs(output1 - outputn[:1])))

diffcount = 0
for  n in range (1, batch_size):
    outputn = linear (x[:n])
    maxdiff = torch.max(torch.abs(output[:1] - outputn[:1])).data[0]
    if  maxdiff != 0.0:
        print  ('n =', n, ', maxdiff =', maxdiff)
        diffcount += 1
print ('diffcount =', diffcount)

dlinear = torch.nn.Linear(input_size,output_size)
dlinear.weight = torch.nn.parameter.Parameter (linear.weight.data.double())
dlinear.bias = torch.nn.parameter.Parameter (linear.bias.data.double())
dx = x.double()
doutput = dlinear(dx)
doutput1 = dlinear(dx[:1])
doutput2 = dlinear(dx[:(batch_size//2)])

print(10e6*torch.max(torch.abs(doutput[:1] - doutput1)))
print(10e6*torch.max(torch.abs(doutput[:(batch_size//2)] - doutput2)))

ddiffcount = 0
for  n in range (1, batch_size):
    doutputn = dlinear (dx[:n])
    maxdiff = torch.max(torch.abs(doutput[:1] - doutputn[:1])).data[0]
    if  maxdiff != 0.0:
        print  ('n =', n, ', maxdiff =', maxdiff)
        ddiffcount += 1
print ('ddiffcount =', ddiffcount)
</code></pre>
<p>Output (set to run on cpu):</p>
<pre><code class="lang-plaintext">0.3.0b0+591e73e
gpu = False
Variable containing:
 2.9802
[torch.FloatTensor of size 1]

Variable containing:
 0
[torch.FloatTensor of size 1]

n = 1 , diff = Variable containing:
 2.9802
[torch.FloatTensor of size 1]

n = 1 , diff = Variable containing:
 0
[torch.FloatTensor of size 1]

n = 2 , diff = Variable containing:
 2.6822
[torch.FloatTensor of size 1]

n = 2 , diff = Variable containing:
 2.3842
[torch.FloatTensor of size 1]

n = 3 , diff = Variable containing:
 2.3842
[torch.FloatTensor of size 1]

n = 3 , diff = Variable containing:
 2.0117
[torch.FloatTensor of size 1]

n = 4 , diff = Variable containing:
 0
[torch.FloatTensor of size 1]

n = 4 , diff = Variable containing:
 2.9802
[torch.FloatTensor of size 1]

n = 1 , maxdiff = 2.980232238769531e-07
n = 2 , maxdiff = 2.682209014892578e-07
n = 3 , maxdiff = 2.384185791015625e-07
diffcount = 3
Variable containing:
 0
[torch.DoubleTensor of size 1]

Variable containing:
 0
[torch.DoubleTensor of size 1]

ddiffcount = 0
</code></pre>

albanD: <p>My explanation based on the original code is that depending on the amount of computation to be done, different algorithms can be used.<br>
If we focus only on cpu, we have flags like <a href="https://github.com/pytorch/pytorch/blob/2ce8c83f677fbd144c45d0b500a62c392b01c8fe/torch/csrc/jit/fuser/cpu/resource_strings.h#L79" rel="nofollow noopener">this</a> one that make the choice between using a single thread or OpenMP for multithreaded computations.<br>
I think this is why you see a difference in the original code between the full operation and the one with one sample (mono-core algorithm used), again the other one where if you use half the batch or the whole batch (multi-core algorithm used).</p>
<p>Of course these different algorithms will give rise to different rounding.<br>
A similar argument can be made on GPU where the grid/block sizes are decided based on the input size.</p>
<p>Is that a more satisfying explanation? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

KFrank: <p>Hello Alban!</p>
<aside class="quote no-group quote-modified" data-username="albanD" data-post="5" data-topic="56735" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>My explanation based on the original code is that depending on the amount of computation to be done, different algorithms can be used.<br>
If we focus only on cpu, we have flags like <a href="https://github.com/pytorch/pytorch/blob/2ce8c83f677fbd144c45d0b500a62c392b01c8fe/torch/csrc/jit/fuser/cpu/resource_strings.h#L79" rel="noopener nofollow ugc">this</a> one that make the choice between using a single thread or OpenMP for multithreaded computations.</p>
<p>…</p>
<p>Is that a more satisfying explanation? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Yes, I’ll buy that.  Certainly bumping over to an OpenMP algorithm<br>
(or some other size-dependent change of algorithm) would be<br>
expected to change the details of the floating-point result.</p>
<p>(One specific detail doesn’t seem to add up:  Your quoted<br>
<code>#define OMP_THRESHOLD 100000</code> seems to be too large to trigger<br>
OpenMP for Tuan Ahn’s example, at least for my naive estimates of<br>
what <code>totalElements</code> might be.)</p>
<p>Thanks.</p>
<p>K. Frank</p>

albanD: <p>Yes, this is most likely not even used in this example. This was just an example of a place where the underlying implementation is dependent on the input size <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Also the underlying libraries we use like MKL and OpenBLAS will have their own thresholds. And the cuda libraries will have their own set of threshold (you can actually play with cudnn algorithm selection for conv by setting torch.backends.cudnn.benchmark and torch.backends.cudnn.deterministic).<br>
I’m afraid it’s beyond my knowledge which thresholds are hit in this particular case.</p>
<p>If you really want to know, you can trace down the call stack to see exactly which function is used. And what are the conditions on that function. I would be interested to know the answer if you do that! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>