Matmul gives different results depending on operating system

Yaroslav_Bulatov: <p>Below is an example of matrix multiplication that incurs precision loss in float32. Any idea why the result is different when I run it on Mac vs Linux?</p>
<pre><code class="lang-auto">import torch
x = torch.tensor([[11041., 13359, 15023, 18177], [13359, 16165, 18177, 21995], [15023, 18177, 20453, 24747], [18177, 21995, 24747, 29945]])
y = torch.tensor([[29945., -24747, -21995, 18177], [-24747, 20453, 18177, -15023], [-21995, 18177, 16165, -13359], [18177, -15023, -13359, 11041]])
print(x@y)
</code></pre>
<p>On Linux I’m seeing the following result, both GPU and CPU (<a href="https://colab.research.google.com/drive/1q9O-qLvI-bS1lxBWhcNA7oRs8puhhGmU#scrollTo=OIc6mDHPme2c" rel="nofollow noopener">colab</a>)</p>
<pre><code class="lang-auto">[[ 33.  17.   1.   1.]
 [ 11.  27.  -5.  -5.]
 [ 11.  -5.  27. -21.]
 [ -7.   9.   9.  25.]]
</code></pre>
<p>Whereas on mac laptop, I see the following</p>
<pre><code class="lang-auto">[[28.,  0., -4.,  0.],
 [ 0., 28.,  0., -4.],
 [20.,  0., 20.,  0.],
 [ 0., 20.,  0., 20.]]
</code></pre>

albanD: <p>Hi,</p>
<p>The expected result is <code>16 * Identity</code> right?<br>
I guess using different blas libraries that potentially use different algorithms for mm will lead to different results. Running on different gpus / different number of cpu cores  (for larger matrices) would give different results as well.</p>

Yaroslav_Bulatov: <p>Yes, 16*Identity is the correct answer. The curious part was that CPU result is identical to GPU result when running on Volta, so it must be using the same algorithm</p>

albanD: <p>Actually on my machine, a binary install gives me the same thing as your mac while a source install gives the same thing as your linux (with everything else being the same). Do you use OpenBLAS on your linux and mkl on your mac by any chance?</p>

Yaroslav_Bulatov: <p>For both Linux and Mac, I installed PyTorch latest official conda instructions for PyTorch 1.2. Is there a way to check if it’s using MKL?</p>
<p>BTW, this came out when trying to debug Hessian calculation of f(X)=sum(Y*Y) with Y=AXB, all three matrices are 2x2 initialized with entries 1,2,3,…,12 . Looks like numerical cancellation can be an issue even for tiny examples.</p>

albanD: <p>You can use the <code>get_env_info()</code> function from the <code>torch.utils.collect_env</code> module to get some informations.</p>
<p>To be sure what is used, the best way is to open python. <code>import torch</code>. Copy the path to the main c library with <code>torch._C.__file__</code>. Then in your terminal, run <code>ldd path_to_that_library</code>. It will list the shared libraries it links to.</p>
<p>In my case, for the source install, there is the line: <code>libopenblas.so.0 =&gt; MY_OPENBLAS_INSTALL_PATH/lib/libopenblas.so.0</code>. So this one definitely use my own openblas install.</p>
<p>For the binary install, no such line exist, but if you check <code>libtorch.so</code>, it jumps from 95MB in a source install to 230MB for a binary one. That’s because the mkl library is statically linked into it. You can see this by first getting the path to <code>libtorch.so</code> from the <code>ldd</code> command above, then run <code>nm -gC path_to_libtorch.so | grep mkl</code> to see all the symbols associated with mkl (there is a lot of them, this will spam heavily your terminal).</p>
<p>Note that running that same command on the libtorch.so from the source install does not see any mkl symbols (only mkldnn ones that are unrelated).</p>
<p>Now if both your computers are using the mkl library, maybe there is different handling of mac machines in mkl?</p>

Yaroslav_Bulatov: <p>Thanks for the tip! It looks like conda install brings in MKL, whereas default AWS AMI images don’t include MKL.</p>
<p>I observed a couple of other results for this problem (curiously, always integer valued) on different configurations, looks like the real issue is the numerics of the underlying example</p>