Considerable absolute error in torch.matmul

username: Koramajin
cooked: <p>Hi, I’ve got high absolute error in torch.matmul. Here is my exmaple code.</p>
<pre><code class="lang-auto">import torch
from torch import tensor


def main():
    torch.cuda.manual_seed(42)
    mask = tensor([[0., 0., 0.], [1.0, 0., 0.]], device='cuda')
    matrix = torch.randn(3, 3, device='cuda')
    print(matrix)

    res = torch.matmul(mask, matrix)
    print(res)


if __name__ == "__main__":
    main()

</code></pre>
<p>And the result is</p>
<pre><code class="lang-auto">tensor([[ 0.1940,  2.1614, -0.1721],
        [ 0.8491, -1.9244,  0.6530],
        [-0.6494, -0.8175,  0.5280]], device='cuda:0')
tensor([[ 0.0000,  0.0000,  0.0000],
        [ 0.1940,  2.1621, -0.1720]], device='cuda:0')
</code></pre>
<p>where I got a high absolute error between 2.1614 and 2.1621. They are expected to be same in the result. But when I move all tensors to cpu, the result is correct. I wonder whether there is a problem in pytorch or cuda backend and how can I fix it.<br>
My Pytorch version is 1.10.1+cu113 and CUDA toolkit version is 11.5.</p>
<p>PS:<br>
As for my running environment, I first produced this issue in WSL2 and I can’t find out its actual cuda toolkit version. Because I just let it use the cuda library provided by the driver in Win. Installing a cuda toolkit of some specific version like 11.3 through apt will cause cuda become unavailable in Pytorch. So I can’t use nvcc to check the version of cuda toolkit in WSL2. Then I reproduced this problem in my server. The gpu of my server is A100 and I run the code in NGC Pytorch 21.11. The cuda toolkit version of my server is 11.5. But I still got the same problem.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/koramajin">@Koramajin</a> I am unable to reproduce the issue<br>
TORCH : 1.10.1<br>
CUDATOOLKIT : 11</p>
<pre><code class="lang-auto">tensor([[ 0.1940,  2.1614, -0.1721],
        [ 0.8491, -1.9244,  0.6530],
        [-0.6494, -0.8175,  0.5280]], device='cuda:0')
tensor([[ 0.0000,  0.0000,  0.0000],
        [ 0.1940,  2.1614, -0.1721]], device='cuda:0')

</code></pre>

username: KFrank
cooked: <p>Hi Koramajin!</p>
<aside class="quote no-group" data-username="Koramajin" data-post="1" data-topic="140947" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e5b9ba/48.png" class="avatar"> Koramajin:</div>
<blockquote>
<p>But when I move all tensors to cpu, the result is correct.</p>
</blockquote>
</aside>
<p>Could this be the <a href="https://pytorch.org/docs/1.10.1/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" rel="noopener nofollow ugc">TF32</a> “bug?”</p>
<p>What specific gpu are you using and does the issue go away if you set<br>
<code>torch.backends.cuda.matmul.allow_tf32 = True</code>?</p>
<p>Edit:</p>
<p>The above should read</p>
<pre data-code-wrap="python"><code class="lang-python">torch.backends.cuda.matmul.allow_tf32 = False
</code></pre>
<p><code>True</code> is the default for <code>allow_tf32</code>.  The suggestion is to turn <em>off</em> TF32<br>
which is done by setting <code>allow_tf32</code> to <code>False</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Koramajin
cooked: <p>Thanks for reply. But it doesn’t work, the result still has error.</p>

username: Koramajin
cooked: <p>It seems a problem associated with cudatoolkit. Can you tell a more specific version of your cuda toolkit?</p>

username: KFrank
cooked: <p>Hi Koramajin!</p>
<aside class="quote no-group" data-username="Koramajin" data-post="4" data-topic="140947" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e5b9ba/48.png" class="avatar"> Koramajin:</div>
<blockquote>
<p>Thanks for reply. But it doesn’t work, the result still has error.</p>
</blockquote>
</aside>
<p>I made a mistake in my earlier post – see the edit, above.</p>
<p>Please try specifically setting the TF32 flag to <code>False</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.backends.cuda.matmul.allow_tf32 = False
</code></pre>
<p>and see if that resolves the issue.</p>
<p>(Sorry for the earlier mistake.)</p>
<p>Please note:  My understanding is that your A100 gpu does support TF32,<br>
so this behavior would be expected.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Koramajin
cooked: <p>Thanks a lot. I didn’t realize that they had optimized matrix multiplication by using TensorFloat32 core in GPU.</p>

username: HaithemH
cooked: <p>Setting torch.backends.cuda.matmul.allow_tf32 = False wor well on colab</p>