Got Different Inference Conv2d Results on Different GPU Machine

Ardian_Umam: <p>Hi,</p>
<p>I trained a model, and test the model in different GPU machine: (i) NVIDIA GeForce RTX 3090 and (ii) GeForce RTX 2080 Ti. However, I got different accuracy results in both machine. Note that I use exactly same code base and same trained model, only evaluation mode is performed here. After some checking, I found that each Conv2d operation gives slightly different results using same input data, on the both machines. Here is the example of the Conv2d output:<br>
On machine (i):</p>
<pre><code class="lang-auto">tensor([-0.0041, -0.0202, -0.0055, -0.0015, -0.0053,  0.0510, -0.0171, -0.0144,
         0.0384,  0.0162], device='cuda:0', grad_fn=&lt;SliceBackward&gt;)
</code></pre>
<p>On machine (ii):</p>
<pre><code class="lang-auto">tensor([-0.0049, -0.0200, -0.0050, -0.0019, -0.0054,  0.0477, -0.0157, -0.0146,
         0.0360,  0.0177], device='cuda:0', grad_fn=&lt;SliceBackward&gt;)
</code></pre>
<p>I expect that the numbers must be the same in the both machines. Any idea how to solve this issue?</p>
<p>Additional info:</p>
<ul>
<li>Both machine uses same Torch version (1.9.0)</li>
<li>Machine (i) uses cuda 11.3, machine (ii) uses cuda 10.2</li>
<li>I print the Conv2d weight, and both machines give same numbers</li>
</ul>
<p>Thanks!</p>

KFrank: <p>Hi Ardian!</p>
<aside class="quote no-group" data-username="Ardian_Umam" data-post="1" data-topic="141021" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardian_umam/48/36724_2.png" class="avatar"> Ardian_Umam:</div>
<blockquote>
<p>I got different accuracy results in both machine.</p>
</blockquote>
</aside>
<p>Could this be TF32?  Note that your 3090 gpu has TF32, while your<br>
2080 gpu does not, so this is probably the issue.</p>
<p>(<a class="mention" href="/u/ptrblck">@ptrblck</a> – Could you guys consider turning off TF32 by default?  TF32<br>
is a nice feature, but recent evidence suggests that enabling it by default<br>
counts as a bug.)</p>
<p>Please see this recent post:</p>
<aside class="quote quote-modified" data-post="3" data-topic="140947">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/considerable-absolute-error-in-torch-matmul/140947/3">Considerable absolute error in torch.matmul</a> 
  </div>
  <blockquote>
    Hi Koramajin! 

Could this be the <a href="https://pytorch.org/docs/1.10.1/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" rel="noopener nofollow ugc">TF32</a> “bug?” 
What specific gpu are you using and does the issue go away if you set 
torch.backends.cuda.matmul.allow_tf32 = True? 
Edit: 
The above should read 
torch.backends.cuda.matmul.allow_tf32 = False

True is the default for allow_tf32.  The suggestion is to turn off TF32 
which is done by setting allow_tf32 to False. 
Best. 
K. Frank
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

ptrblck: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="141021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Could you guys consider turning off TF32 by default? TF32<br>
is a nice feature, but recent evidence suggests that enabling it by default<br>
counts as a bug.</p>
</blockquote>
</aside>
<p>Could you add your concerns to <a href="https://github.com/pytorch/pytorch/issues/67384">this RFE</a> so that we could track it, please?</p>