The same deep network, the same input and code, the results after the decimal point are inconsistent on different hardware graphics cards

Chen_Xie: <p>The same deep network, the same input and code, the results after the decimal point are inconsistent on different hardware graphics cards. first i thought maybe the random seed lead to this. however, i still get the inconsistent result with identical random seed. The difference is only after the decimal point on (2070 Vs 2080Ti or 3060)， is this normal?<br>
what should i do to make the results same?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0afdffda0683205a8fd0aff5f3d30ad23c73f897.png" data-download-href="https://discuss.pytorch.org/uploads/default/0afdffda0683205a8fd0aff5f3d30ad23c73f897" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0afdffda0683205a8fd0aff5f3d30ad23c73f897.png" alt="image" data-base62-sha1="1zeXT7HHGpUgYK1ckjVn4HSDAnJ" width="690" height="329" data-dominant-color="EFDBDB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1388×663 38.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It could be expected depending on the relative error and used operation.<br>
E.g. a change in the order of operation creates mismatches due to the limited floating point precision:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
print((s1 - s2).abs())
# tensor(3.0518e-05)
</code></pre>

Chen_Xie: <p>but i can get the same result use your code at different GPUS with identical random seed. This only means that after the random number is determined, the random results will be consistent. i dont think this is answer to my question. i still cannot figure out why my results are diffrent.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3218535018a5e0fba9a7ec75b21d518435e51b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/c3218535018a5e0fba9a7ec75b21d518435e51b2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c3218535018a5e0fba9a7ec75b21d518435e51b2.png" alt="image" data-base62-sha1="rQcVgPTArAxsXdbNGBq0YfG4iMG" width="690" height="66" data-dominant-color="231520"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1485×144 7.33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>No it doesn’t and would still depend on the used algorithm. I.e. if the <code>sum</code> kernel outputs deterministic results, your comparison is valid. If not, different results are expected.</p>

Chen_Xie: <p>i think i understand what your means, the diffrences do eixsit.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e467d8c9025451d3624af0c40869642abede974.png" data-download-href="https://discuss.pytorch.org/uploads/default/3e467d8c9025451d3624af0c40869642abede974" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e467d8c9025451d3624af0c40869642abede974_2_690x230.png" alt="image" data-base62-sha1="8SUCWPPW3QjsD8isuGWP5TFKxNi" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e467d8c9025451d3624af0c40869642abede974_2_690x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/e/3e467d8c9025451d3624af0c40869642abede974_2_1035x345.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e467d8c9025451d3624af0c40869642abede974.png 2x" data-dominant-color="1B171B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1367×456 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, this difference starts with three decimal places and cannot be seen if the value is too large. But when I set 10,000 random values, the difference can be observed. which is same situation as my results.</p>