Simple math problems showing different results for torch and numpy

username: Khabbab_Zakaria
cooked: <p>Hi,<br>
So I was working with a code when I realized this issue:<br>
Suppose,</p>
<pre><code class="lang-auto">import torch
import numpy as np

mu1 = torch.Tensor([0.51])
mu2 = torch.Tensor([0.52])
sigma1 = torch.Tensor([0.10])
sigma2 = torch.Tensor([0.11])
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5])

print(output)

mu1 = torch.Tensor([0.51])
mu2 = torch.Tensor([0.52])
sigma1 = torch.Tensor([0.10])
sigma2 = torch.Tensor([0.11])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)
</code></pre>
<p>The results are: tensor([0.0127])<br>
0.012665540341760373, pretty good!<br>
But if,</p>
<pre><code class="lang-auto">import torch
import numpy as np

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5])

print(output)

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)
</code></pre>
<p>gives results: tensor([5.9605e-08])<br>
2.3976444829543198e-08<br>
What is going on? Why such weird behaviour?</p>

username: eqy
cooked: <aside class="quote no-group" data-username="Khabbab_Zakaria" data-post="1" data-topic="142956">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/khabbab_zakaria/48/33485_2.png" class="avatar"> Khabbab_Zakaria:</div>
<blockquote>
<pre><code class="lang-auto">import torch
import numpy as np

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5])

print(output)

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)
</code></pre>
</blockquote>
</aside>
<p>I believe that this is because numpy will do operations in <code>float64</code> by default while in torch they are in <code>float32</code>. For example:</p>
<pre><code class="lang-auto">import torch
import numpy as np

mu1 = torch.Tensor([0.4651]).to(torch.double)
mu2 = torch.Tensor([0.4652]).to(torch.double)
sigma1 = torch.Tensor([0.7910]).to(torch.double)
sigma2 = torch.Tensor([0.7911]).to(torch.double)
output = torch.log(sigma2/sigma1) + (sigma1**2 + (mu1 - mu2)**2)/(2*sigma2**2) - torch.Tensor([0.5]).to(torch.double)

print(output)

mu1 = torch.Tensor([0.4651])
mu2 = torch.Tensor([0.4652])
sigma1 = torch.Tensor([0.7910])
sigma2 = torch.Tensor([0.7911])
output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5

print(output)
print(output.dtype)
</code></pre>
<p>yields the following output</p>
<pre><code class="lang-auto">tensor([2.3976e-08], dtype=torch.float64)
2.3976444829543198e-08
float64
</code></pre>
<p>on my machine.</p>

username: my3bikaht
cooked: <pre><code class="lang-auto">output = np.log(sigma2.item()/sigma1.item()) + (sigma1.item()**2 + (mu1.item() - mu2.item())**2)/(2*sigma2.item()**2) - 0.5
</code></pre>
<p>calling item() returns python’s standard float, which is double precision, thus np.log will keep processing floats as double precision<br>
If you want to keep precision intact you can replace item() with numpy(), this will convert all tensor.float32 to np.float32</p>
<pre><code class="lang-auto">output = np.log(sigma2.numpy()/sigma1.numpy()) + (sigma1.numpy()**2 + (mu1.numpy() - mu2.numpy())**2)/(2*sigma2.numpy()**2) - 0.5
&gt;&gt;&gt; output
array([5.9604645e-08], dtype=float32)
</code></pre>