Same model gets slightly different results on different platforms

ljh: <p>The model gives silightly different results on different platforms. And the model is trained on GPU and save as .cuda() format.<br>
Is there a way to make the result to be precisely the same?<br>
with torch 1.13.1+cu116, on Tesla A100<br>
torch 1.13.1+cu116, on GTX1060<br>
and torch 2.1.2 on MacOS mps</p>
<p>The test script goes like this</p>
<pre data-code-wrap="python"><code class="lang-python">    def test_model_consistency(self):
        if(torch.cuda.is_available()):
            device = torch.device("cuda")
        elif(torch.backends.mps.is_available()):
            device = torch.device("mps")
        else:
            device = torch.device("cpu")
        model_path = "best.pth"
        if(torch.cuda.is_available()):
            model = torch.load(model_path).to(device)
        else:
            model = torch.load(model_path, map_location=device).to(device)
        output = model(torch.zeros(1,128,6).to(device))
        print(output)
</code></pre>
<p>And I get<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6b9cc82594011b18de4dce3d8511b21cea468d.png" alt="image-20231229204928278" data-base62-sha1="dthrko7NkkE3w5NpfwRj94d14Tz" width="526" height="145"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/531e5d952bec28a0140302530fa840d0954a55c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/531e5d952bec28a0140302530fa840d0954a55c1" title="image-20231229205035341"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/531e5d952bec28a0140302530fa840d0954a55c1.png" alt="image-20231229205035341" data-base62-sha1="bRiGEuHH2c7y9VBAaEPHLYkqolr" width="690" height="183" data-dominant-color="2D2D2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image-20231229205035341</span><span class="informations">1108×295 21.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d" title="232f7e131c759d9a51ab89f0e3233d88"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d_2_690x167.jpeg" alt="232f7e131c759d9a51ab89f0e3233d88" data-base62-sha1="nOeYANGDDDNotSqJyIJfJqfla8J" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d_2_690x167.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a6dfe808f5d4b9cc4c14b603e26c094b34ed3d0d.jpeg 2x" data-dominant-color="33363B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">232f7e131c759d9a51ab89f0e3233d88</span><span class="informations">888×216 54.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There are slightly differences for the output, but I want them to be exactly the same. Is there any way to do that?</p>

ptrblck: <p>No, since different hardware and their corresponding libraries might use different algorithms causing the expected, small numerical mismatches caused due to the limited floating point precision.</p>