CPU/GPU results inconsistent with matrix multiplication

zvant: <p>I always thought 32-bits floats should be sufficient for most ML calculations. I tested the actual precision of a simple matrix multiplication operation on NumPy, PyTorch CPU, and PyTorch CUDA. Here is my code:</p>
<pre><code class="lang-auto">import numpy as np
import torch

np.random.seed(0)
M = np.random.randn(1000, 1000).astype(np.float64)

for T_np, T_cuda in [(np.float64, torch.float64), (np.float32, torch.float32)]:
    print('')
    M_np, M_pt, M_cuda = M.astype(T_np), torch.from_numpy(M).type(T_cuda), torch.from_numpy(M).type(T_cuda).cuda()
    M_pt.requires_grad = M_cuda.requires_grad = False
    print(M_np.dtype, M_np.shape, M_pt.dtype, M_pt.size(), M_cuda.dtype, M_cuda.size(), flush=True)

    MM = np.matmul(M_np, M_np)
    print(MM[0, : 5], flush=True)

    MM_pt = torch.matmul(M_pt, M_pt)
    print(MM_pt[0, : 5], flush=True)

    diff = np.absolute(M_np - M_pt.numpy())
    print('error:', diff.sum().item(), diff.mean().item(), diff.std().item())
    diff = np.absolute(MM - MM_pt.numpy())
    print('error:', diff.sum().item(), diff.mean().item(), diff.std().item())

    MM_cuda = torch.matmul(M_cuda, M_cuda)
    print(MM_cuda[0, : 5], flush=True)

    diff = np.absolute(M_np - M_cuda.cpu().numpy())
    print('error:', diff.sum().item(), diff.mean().item(), diff.std().item())
    diff = np.absolute(MM - MM_cuda.cpu().numpy())
    print('error:', diff.sum().item(), diff.mean().item(), diff.std().item())
</code></pre>
<p>On my windows machine with an RTX30 GPU, the output is:</p>
<pre><code class="lang-auto">float64 (1000, 1000) torch.float64 torch.Size([1000, 1000]) torch.float64 torch.Size([1000, 1000])
[-20.90456457  63.66601763   4.60530742 -38.41815213 -50.70685021]
tensor([-20.9046,  63.6660,   4.6053, -38.4182, -50.7069], dtype=torch.float64)
error: 0.0 0.0 0.0
error: 0.0 0.0 0.0
tensor([-20.9046,  63.6660,   4.6053, -38.4182, -50.7069], device='cuda:0',
       dtype=torch.float64)
error: 0.0 0.0 0.0
error: 1.8810475736069643e-08 1.8810475736069643e-14 1.685592703582923e-14

float32 (1000, 1000) torch.float32 torch.Size([1000, 1000]) torch.float32 torch.Size([1000, 1000])
**[-20.904562  63.666027   4.605306 -38.418148 -50.706852]**
**tensor([-20.9046,  63.6660,   4.6053, -38.4181, -50.7069])**
error: 0.0 0.0 0.0
error: 0.0 0.0 0.0
**tensor([-20.8954,  63.6586,   4.5888, -38.4053, -50.7082], device='cuda:0')**
error: 0.0 0.0 0.0
error: 7389.58349609375 0.007389583624899387 0.005598879884928465
</code></pre>
<p>It shows that for 64-bits floats, results for CUDA tensors are consistent with both CPU tensors and NumPy matrices. But for 32-bits floats, CUDA tensors give quite different results, while CPU tensors are same to NumPy results.</p>
<p>I have another machine with GTX1080Ti running Ubuntu. Same code, the results are:</p>
<pre><code class="lang-auto">float64 (1000, 1000) torch.float64 torch.Size([1000, 1000]) torch.float64 torch.Size([1000, 1000])
[-20.90456457  63.66601763   4.60530742 -38.41815213 -50.70685021]
tensor([-20.9046,  63.6660,   4.6053, -38.4182, -50.7069], dtype=torch.float64)
error: 0.0 0.0 0.0
error: 0.0 0.0 0.0
tensor([-20.9046,  63.6660,   4.6053, -38.4182, -50.7069], device='cuda:0',
       dtype=torch.float64)
error: 0.0 0.0 0.0
error: 2.4674370506863396e-08 2.4674370506863395e-14 2.4656188542088475e-14

float32 (1000, 1000) torch.float32 torch.Size([1000, 1000]) torch.float32 torch.Size([1000, 1000])
[-20.904562  63.666027   4.605306 -38.418148 -50.706852]
tensor([-20.9046,  63.6660,   4.6053, -38.4181, -50.7069])
error: 0.0 0.0 0.0
error: 0.0 0.0 0.0
tensor([-20.9046,  63.6660,   4.6053, -38.4181, -50.7068], device='cuda:0')
error: 0.0 0.0 0.0
error: 10.073770523071289 1.0073770681628957e-05 9.016605872602668e-06
</code></pre>
<p>The difference between CUDA/CPU results is much smaller this time. Both machines runs PyTorch 1.10 with CUDA toolkit 11.3.</p>
<p>From the results, the difference comes from the matrix multiplication operation, instead of copying tensors from RAM to GPU. For Windows, the error is really high for 32-bits floats. I think the results are not very reliable anymore. I tested matrix adding too, but there was no error at all. Can anyone familiar with the underlying GPU implementation of PyTorch explain? Thank you!</p>

KFrank: <p>Hi Zvant!</p>
<aside class="quote no-group quote-modified" data-username="zvant" data-post="1" data-topic="136427" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zvant/48/43824_2.png" class="avatar"> zvant:</div>
<blockquote>
<p>with an RTX30 GPU<br>
…<br>
But for 32-bits floats, CUDA tensors give quite different results<br>
…<br>
I have another machine with GTX1080Ti</p>
</blockquote>
</aside>
<p>Depending on your  GPU, nvidia might be switching you over by<br>
default to the misleading (dishonestly?) named “tf32” floating-point<br>
arithmetic.  (tf32 is essentially half-precision floating-point.)</p>
<p>You can try turning tf32 off with:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.backends.cuda.matmul.allow_tf32 = False
</code></pre>
<p>See the following thread and the github issue <a class="mention" href="/u/tom">@tom</a> references in it:</p>
<aside class="quote quote-modified" data-post="1" data-topic="135248">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/4bbf92/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bug-matmul-seems-to-cast-to-float16-internally/135248">Bug? matmul seems to cast to float16 internally</a> 
  </div>
  <blockquote>
    What’s going on here? Comments will be appreciated. 
[[1.0]] multiplied by [[1.0001]] incorrectly results in 1., whereas [1.0] multiplied by [1.0001] correctly results in 1.0001 on Win10. 
It seems as if the internal accumulator is only float16. It works with float64, or without using CUDA. 
Cannot reproduce on Ubuntu machine. 
Code 
import torch 
dtype = torch.float32 
A = torch.tensor([[1.]], dtype=dtype).cuda() 
B = torch.tensor([[1.0001]], dtype=dtype).cuda() 
test1 = torch.matmul(A, B) 
A =…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

zvant: <p>I set the flag to False and the error level is now reasonable. Never knew PyTorch on Ampere GPUs has this behaviour. Guess in most cases tf32 should be enough. Most networks have all kinds of regularizations anyway. Thank you!</p>

tom: <p>While I have reservations about PyTorch enabling tf32 by default, note that this</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="136427">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>tf32 is essentially half-precision floating-point</p>
</blockquote>
</aside>
<p>is a bit of an oversimplification, I’d probably describe it as “tf32 has the dynamic range of fp32 but the relative precision of fp16”. For many applications, dynamic range of fp16 has been troublesome (so half overflowing to +/-inf or underflowing to 0), this is why AMP does the gradient scaling for backpropagation. In place where fp16 fails due to the lack of relative precision, tf32 won’t cut the mustard, either.<br>
If tf32s characteristics are acceptable, it does seem to offer a formidable speed-up.</p>
<p>Best regards</p>
<p>Thomas</p>