Bug? matmul seems to cast to float16 internally

tjak: <p>What’s going on here? Comments will be appreciated.</p>
<p>[[1.0]] multiplied by [[1.0001]] incorrectly results in 1., whereas [1.0] multiplied by [1.0001] correctly results in 1.0001 on Win10.</p>
<p>It seems as if the internal accumulator is only float16. It works with float64, or without using CUDA.</p>
<p>Cannot reproduce on Ubuntu machine.</p>
<p><strong>Code</strong><br>
import torch</p>
<p>dtype = torch.float32<br>
A = torch.tensor([[1.]], dtype=dtype).cuda()<br>
B = torch.tensor([[1.0001]], dtype=dtype).cuda()<br>
test1 = torch.matmul(A, B)</p>
<p>A = torch.tensor([1.], dtype=dtype).cuda()<br>
B = torch.tensor([1.0001], dtype=dtype).cuda()<br>
test2 = torch.matmul(A, B)</p>
<p>print(test1)<br>
print(test2)</p>
<p>print(torch.version.cuda)<br>
print(torch.<strong>version</strong>)</p>
<p><strong>Output</strong><br>
tensor([[1.]], device=‘cuda:0’)<br>
tensor(1.0001, device=‘cuda:0’)<br>
Cuda v11.3<br>
Torch v1.10.0+cu113</p>

tjak: <p>It’s running on an RTX 3080.</p>

KFrank: <p>Hi Thomas!</p>
<aside class="quote no-group" data-username="tjak" data-post="1" data-topic="135248" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/4bbf92/48.png" class="avatar"> tjak:</div>
<blockquote>
<p>It seems as if the internal accumulator is only float16. It works with float64, or without using CUDA.</p>
<p>Cannot reproduce on Ubuntu machine.</p>
</blockquote>
</aside>
<p>FYI, I also cannot reproduce this on Ubuntu using a<br>
“GeForce GTX 1050 Ti”.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; dtype = torch.float32
&gt;&gt;&gt; A = torch.tensor([[1.]], dtype=dtype).cuda()
&gt;&gt;&gt; B = torch.tensor([[1.0001]], dtype=dtype).cuda()
&gt;&gt;&gt; test1 = torch.matmul(A, B)
&gt;&gt;&gt; A = torch.tensor([1.], dtype=dtype).cuda()
&gt;&gt;&gt; B = torch.tensor([1.0001], dtype=dtype).cuda()
&gt;&gt;&gt; test2 = torch.matmul(A, B)
&gt;&gt;&gt; print(test1)
tensor([[1.0001]], device='cuda:0')
&gt;&gt;&gt; print(test2)
tensor(1.0001, device='cuda:0')
&gt;&gt;&gt; print(torch.version.cuda)
10.2
&gt;&gt;&gt; print(torch.__version__)
1.9.0
&gt;&gt;&gt; print (torch.cuda.get_device_name (0))
GeForce GTX 1050 Ti
</code></pre>
<pre><code class="lang-plaintext">$ uname -a
Linux server 5.11.0-38-generic #42~20.04.1-Ubuntu SMP Tue Sep 28 20:41:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

tom: <p>I can confirm this happening for me, too.</p>
<p>Workaround: <code>torch.backends.cuda.matmul.allow_tf32 = False</code>, but it doesn’t look good. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=10" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Best regards</p>
<p>Thomas</p>

tom: <p>So I ran this by people who know more than me (i.e. everyone else), and <a class="mention" href="/u/ngimel">@ngimel</a> kindly explained that this is the expected behaviour for tf32 arithmetic, which only uses 10 bits of mantissa.</p>
<p>There is an issue that discusses the loss of precision here: <a href="https://github.com/pytorch/pytorch/issues/61890" class="inline-onebox" rel="noopener nofollow ugc">The matrix multiplication operator can't get correct results on 3090 !! · Issue #61890 · pytorch/pytorch · GitHub</a></p>
<p>Best regards</p>
<p>Thomas</p>

tjak: <p>Excellent answer, thank you very much for taking your time. The workaround you posted solved everything. And it explains what is going on.</p>
<p>I think it’s false advertising that someone named the format tf32, though. Should be tf19 really.</p>
<p>Best regards,<br>
Thomas</p>

tom: <p>Hi Thomas,</p>
<p>in the meantime Natalia opened a RFC issue: <a href="https://github.com/pytorch/pytorch/issues/67384" class="inline-onebox" rel="noopener nofollow ugc">RFC: Should matmuls use tf32 by default? · Issue #67384 · pytorch/pytorch · GitHub</a> .</p>
<p>I want to add: Thanks for bringing it up! It’s important that users write about issues they see for the PyTorch developers to get a better idea of how intuitive default behaviour like this is and where to strike the balance between “maximizing perfomance” and “avoiding surprises”.</p>
<p>Best regards</p>
<p>Thomas</p>