Reproducibility and floating point arithmetics with(out) AVX512

username: TMat
cooked: <p>Hello,</p>
<p>I am trying to have reproducible calculation with pytorch across multiple computers and I managed to do almost that except that I did not manage to have the same results on a computer that support AVX512 and one that does not.</p>
<p>Basically, if I try to execute the following code</p>
<pre><code class="lang-auto">import numpy as np
import torch
import torch.nn as nn
import random
torch.use_deterministic_algorithms(True)

seed = 42

random.seed(seed)
torch.manual_seed(seed)
np.random.seed(seed)

device = torch.device("cpu")

def layer_init(layer, std=np.sqrt(2), bias_const=0.0):
    torch.nn.init.orthogonal_(layer.weight, std)
    torch.nn.init.constant_(layer.bias, bias_const)
    return layer

network = nn.Sequential(layer_init(nn.Linear(100, 100)),
                        layer_init(nn.Linear(100,1), std=1.0),).to(device)
with torch.no_grad():
    action = network(torch.rand(size=(100,)).to(device))

print(format(action.cpu().numpy()[0].item(), '.60g'))
</code></pre>
<p>Result on AVX512 computer is <code>0.4804637432098388671875</code> on non-AVX512<br>
computer I get <code>0.4804628789424896240234375</code>, a difference of order 1e-5.</p>
<p>I understand that this difference is expected due to floating point calculation, my problem is that there should be 0 difference when specifying that AVX512 not be used (i.e. using <code>ATEN_CPU_CAPABILITY=avx2</code>), if the accelerations used are the same on both computers there should not be any difference!<br>
However, even when asking to use only AVX2 the two computers still give different results. Do you have any idea on how to not use AVX512 on an AVX512 compatible computer in order to be reproducible?</p>
<p>This is a continuation of <a href="https://github.com/pytorch/pytorch/issues/155423." rel="noopener nofollow ugc">this github issue</a> in which I already explained my problem and was advised to comehere.</p>
<p>Thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="TMat" data-post="1" data-topic="220674">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/71c47a/48.png" class="avatar"> TMat:</div>
<blockquote>
<p>my problem is that there should be 0 difference when specifying that AVX512 not be used (i.e. using <code>ATEN_CPU_CAPABILITY=avx2</code>), if the accelerations used are the same on both computers there should not be any difference!</p>
</blockquote>
</aside>
<p>Could you leave no to this claim in the AVX docs or explain why this should be the case?</p>

username: TMat
cooked: <p>It is not in the doc, this just seems to be obvious to me that same code and libraries and environment should give same result, but maybe it is not obvious. Reproducibility is indeed hard to achieve in general.</p>
<p>But then my question is why are the results different?</p>
<p>For now, the only way I found to reproduce the results was to create a vm with AVX512 deactivated and then I manage to get reproducibility across several computers, so I believe it is not a hardware problem.</p>
<p>Due to this experiment with a vm, I believe this means that AVX512 is still used even though <code>ATEN_CPU_CAPABILITY=avx2</code>  is set, is it normal?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="TMat" data-post="3" data-topic="220674">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/71c47a/48.png" class="avatar"> TMat:</div>
<blockquote>
<p>It is not in the doc, this just seems to be obvious to me that same code and libraries and environment should give same result, but maybe it is not obvious.</p>
</blockquote>
</aside>
<p>I would expect the opposite: different HW specs and features could use different algorithms with the same instruction set, so unless bitwise-identical results are guaranteed and documented for different HW platforms I would not expect it.</p>