Matmul casting as float16 internally

username: conv8d
cooked: <p>I am using pytorch <code>1.8.0</code> and I cannot update it to newer versions due to other dependency issues.<br>
I am facing <a href="https://discuss.pytorch.org/t/bug-matmul-seems-to-cast-to-float16-internally/135248">this</a> issue (<code>C = A@B</code> where <code>A, B</code> are <code>torch.float32</code> but somehow <code>C</code> becomes <code>torch.float16</code>) but setting <code>torch.backends.cuda.matmul.allow_tf32 = False</code> does not solve it.</p>
<p>How can I fix this?</p>

username: khushi-411
cooked: <p>Hi!<br>
There was an issue relating to this in the previous version of PyTorch. Now that’s resolved. You’ll need to update your PyTorch version. Please see:</p>
<pre><code class="lang-python">In [2]: tensor1 = torch.randn(3, dtype=torch.float32)

In [3]: tensor2 = torch.randn(3, dtype=torch.float32)

In [4]: torch.matmul(tensor1, tensor2).dtype
Out[4]: torch.float32

In [5]: k = tensor1@tensor2
Out[5]: tensor(0.4118)

In [6]: k.dtype
Out[6]: torch.float32
</code></pre>
<p>Thanks!</p>
<p>EDIT: <a class="mention" href="/u/tom">@tom</a> shared this issue (ps: this is resolved) in the link you attached. Issue: <a href="https://github.com/pytorch/pytorch/issues/67384" class="inline-onebox" rel="noopener nofollow ugc">RFC: Should matmuls use tf32 by default? · Issue #67384 · pytorch/pytorch · GitHub</a>.</p>

username: conv8d
cooked: <p>Like I said, I cannot update the pytorch version from <code>1.8</code> because it breaks other dependencies in the virtual environment that I am using.<br>
Is there another solution?</p>

username: khushi-411
cooked: <p>Another option would be explicitly casting it to float32. In your case, it would be:</p>
<pre><code class="lang-python">C.type(torch.float32)
</code></pre>
<p>Thanks!</p>

username: conv8d
cooked: <p>But this would be after <code>C</code> has already been computed in <code>float16</code>, right?<br>
My issue is that <code>C</code> being in <code>float16</code> results in <code>nan</code>s in <code>C</code>.</p>

username: khushi-411
cooked: <blockquote>
<p>But this would be after <code>C</code> has already been computed in <code>float16</code> , right?</p>
</blockquote>
<p>Yep.</p>
<p>I don’t think you can do anything with internal problems in PyTorch. The only solution is updating the version.</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/conv8d">@conv8d</a>,</p>
<p>Have you tried setting the default dtype via <code>torch.set_default_dtype(torch.float32)</code>?</p>
<p>Also, can you set up a minimal reproducible script to show your error?</p>

username: ptrblck
cooked: <p>Note that TF32 is <em>not</em> <code>float16</code>, so I would like to clarify the issue a bit more.<br>
Are you seeing the <code>dtype</code> of a result tensor showing <code>float16</code> or are you running into a precision loss and speculate that <code>float16</code> might be used internally?</p>
<p>As <a class="mention" href="/u/khushi-411">@khushi-411</a> already pointed out: TF32 might be used (which has a reduced precision, but the same range as <code>float32</code>) and can be disabled via <code>torch.backends.cuda.matmul.allow_tf32 = False</code> also in <code>1.8.0</code> as seen <a href="https://github.com/pytorch/pytorch/blob/v1.8.0/torch/backends/cuda/__init__.py#L86-L97">here</a>. The default behavior was changed in later versions to disable TF32 by default for matmuls, but to keep it enabled for cuDNN calls.</p>
<p>Since this flag does not change the behavior, your issue seems to be unrelated to TF32.</p>
<aside class="quote no-group" data-username="conv8d" data-post="5" data-topic="169843">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/e5b9ba/48.png" class="avatar"> conv8d:</div>
<blockquote>
<p>My issue is that <code>C</code> being in <code>float16</code> results in <code>nan</code>s in <code>C</code>.</p>
</blockquote>
</aside>
<p>As already mentioned: TF32 will use the <code>float32</code> range and will not overflow for the <code>float16</code> range limitations, which also suggests your issue might be unrelated to TF32.</p>

username: Shashikant_Verma
cooked: <p>Hello,<br>
I am using torch version 1.13 and same error. Above answers suggest that issue is resolved in newer versions of torch.<br>
Was it not resolved till 1.13 too?</p>

username: ptrblck
cooked: <p>You can check the release notes about the change of the default TF32 behavior but I believe TF32 was disabled by default in the binaries in 1.13.</p>