Torch.save numerical differences

Santosh_Manicka: <p>torch.save() on linux and torch.load() on OSX of the same data is causing discrepancies.</p>
<p>Example:<br>
On Linux (RH), run the following:</p>
<pre><code class="lang-auto">x = torch.randn(100)
y = torch.sigmoid(torch.exp(x))
d = {'x':x, 'y':y}
torch.save(d,'.test.dat')
</code></pre>
<p>Then, on OSX, run the following:</p>
<pre><code class="lang-auto">d = torch.load('.test.dat')
x,y = d['x'], d['y']
y2 = torch.sigmoid(torch.exp(x))
print((y == y2).all())
print((y - y2).sum())
</code></pre>
<p>Where I should be expecting True and zeros, I see False and non-zeros.</p>
<p>Python version: 3.7.11<br>
Pytorch version: 1.10.2</p>
<p>Has anyone else seen such a discrepancy?</p>

ptrblck: <p>Depending on the magnitude of the errors you are seeing, you might be running into the expected errors due to the limited floating point precision.</p>

Santosh_Manicka: <p>Shouldn’t the loss of precision occur in the same way across platforms (Linux, OSX, Windows, etc.)? Is it by any chance related to <a href="https://discuss.pytorch.org/t/numerical-differences-between-numpy-and-pytorch/89607/4">this discussion</a>?</p>

ptrblck: <p>Not necessarily, as different (vectorized) code paths could be used depending on the architecture, library support etc.<br>
Yes, I think the linked thread is related in case the abs. errors match the expectation, e.g. ~1e-5<code>for these</code>randn` values:</p>
<pre><code class="lang-python">x = torch.randn(100, 100)
x1 = x.sum()
x2 = x.sum(0).sum(0)
print((x1 - x2).abs().max())
# &gt; tensor(3.0518e-05)
</code></pre>

Santosh_Manicka: <p>The fact that computations cannot always be reproduced across platforms even without any stochasticity is pretty surprising, especially in this day and age when scientific reproducibility is a major concern.</p>
<p>In any case, thank you for the illuminating MWE above. Is there any reason to think that computations may not be reproduced even on the same platform? Is there an MWE that can illustrate it? I’m currently struggling with such an issue. I train a (custom) model and save its parameters on Linux, but if I load those parameters on the same platform and with the same model, I still see discrepancies in the model results sometimes. I’ve been trying to create an MWE that captures this phenomenon but I’ve been unable to.</p>
<p>My code has operations like torch.repeat_interleave that according to the <a href="https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html" rel="noopener nofollow ugc">documentation</a> may be non-deterministic, so I’ve tried creating MWEs that has such operations but still no luck…</p>
<p>Just to clarify, the above MWE by <a class="mention" href="/u/ptrblck">@ptrblck</a> has two “models”, one for creating x1 and the other for creating x2. My issue is that I have only one model in this sense and running it with the stored parameters that were generated and saved on the same platform (Linux and CPU only, not GPU) is still causing discrepancies. I would like to think that I’ve done enough investigation to rule out causes in the semantics of my code.</p>

ptrblck: <aside class="quote no-group" data-username="Santosh_Manicka" data-post="5" data-topic="144957">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_manicka/48/13705_2.png" class="avatar"> Santosh_Manicka:</div>
<blockquote>
<p>The fact that computations cannot always be reproduced across platforms even without any stochasticity is pretty surprising, especially in this day and age when scientific reproducibility is a major concern.</p>
</blockquote>
</aside>
<p>Reproducibility can be achieved for the same workloads and setup. However, since different algorithms can be used on different platforms, there won’t be a guarantee to be able to reproduce the same bitwise equal values.</p>
<aside class="quote no-group" data-username="Santosh_Manicka" data-post="5" data-topic="144957">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_manicka/48/13705_2.png" class="avatar"> Santosh_Manicka:</div>
<blockquote>
<p>Is there any reason to think that computations may not be reproduced even on the same platform?</p>
</blockquote>
</aside>
<p>If the same library setup is used (CPU, GPU, MKL, cuDNN, NCCL, cublas, CUDA, PyTorch, etc.) then you should be able to reproduce the same values if you stick to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>. In particular you would need seed the code and set <code>torch.use_deterministic_algorithms(True)</code>. If no deterministic algorithm can be found (even for a performance hit) you should get an error.</p>
<aside class="quote no-group" data-username="Santosh_Manicka" data-post="5" data-topic="144957">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/santosh_manicka/48/13705_2.png" class="avatar"> Santosh_Manicka:</div>
<blockquote>
<p>I’m currently struggling with such an issue. I train a (custom) model and save its parameters on Linux, but if I load those parameters on the same platform and with the same model, I still see discrepancies in the model results sometimes. I’ve been trying to create an MWE that captures this phenomenon but I’ve been unable to.</p>
</blockquote>
</aside>
<p>As described before, take a look at the docs and make sure you are sticking to the needed steps.</p>

Santosh_Manicka: <p>Turns out simply using double-precision (64-bit) tensors mitigated the issue to a great extent!</p>