What is the machine precision of pytorch with CPUs (or GPUs)?

username: Brando_Miranda
cooked: <p>What is the machine precision when working with pytorch? I know from (<a href="https://www.cfd-online.com/Wiki/Machine_precision" rel="nofollow noopener">https://www.cfd-online.com/Wiki/Machine_precision</a>) that for 32 bit computer its:</p>
<p>2^-52 ~ 10^-16 for double<br>
2^-23 ~ 10^-7 for single</p>
<p>Does that mean that for <code>torch.FloatTensor</code> we have <code>2^-23 ~ 10^-7</code>? and for 64 bit we get: <code>2^-104 ~ 10^ -32</code>?</p>
<p>I’m not sure how to actually check this but in numpy/python one can do as its done here <a href="https://stackoverflow.com/questions/19141432/python-numpy-machine-epsilon" rel="nofollow noopener">https://stackoverflow.com/questions/19141432/python-numpy-machine-epsilon</a></p>
<p>but if you throw in torch it doesn’t work:</p>
<pre><code>&gt;&gt;&gt; np.finfo(torch.FloatTensor)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/user/miniconda3/envs/hbf_env/lib/python3.6/site-packages/numpy/core/getlimits.py", line 392, in __new__
    raise ValueError("data type %r not inexact" % (dtype))
ValueError: data type &lt;class 'numpy.object_'&gt; not inexact</code></pre>

username: cbarrick
cooked: <p>AFAIK, the PyTorch dtypes have the same representation as the Numpy dtypes. They should all follow the <a href="https://en.wikipedia.org/wiki/IEEE_754" rel="nofollow noopener">IEEE 754</a> standard. So you’re right about the precision being <code>2^-23</code> for <code>FloatTensor</code>.</p>
<p>I’m not sure how to check it with code though…</p>

username: Brando_Miranda
cooked: <p>fyi if you want to see the datatype of your tensor you might need to access it’s datatype value (assuming pytorch versions don’t change etc):</p>
<pre><code class="lang-auto"># %%

import torch

x = torch.randn(3)

print(x)
print(x.dtype)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">tensor([-0.8643, -0.6282,  1.3406])
torch.float32
</code></pre>
<hr>
<p>fyi:</p>
<p>recall <a href="https://www.cfd-online.com/Wiki/Machine_precision" rel="noopener nofollow ugc">machine precision</a>:</p>
<blockquote>
<p>Machine precision is the smallest number ε such that the difference between 1 and 1 + ε is nonzero, ie., it is the smallest difference between two numbers that the computer recognizes. On a 32 bit computer, single precision is 2-23 (approximately 10-7) while double precision is 2-52 (approximately 10-16) .</p>
</blockquote>
<p>I am trying to figure out if what I have is enough to recognize my current error that is <code>2.00e-7</code></p>

username: Brando_Miranda
cooked: <p>I am running experiments on synthetic data (e.g. fitting a sine curve) and I get errors in pytorch that are really small. One if about <code>2.00e-7</code>. I was reading about machine precision and it seems really close to the machine precision. How do I know if this is going to cause problems (or if perhaps it already has e.g. I can’t differentiate between the different errors since they are “machine zero”).</p>
<p>errors:</p>
<pre><code class="lang-auto">p = np.array([2.3078539778125768e-07,
               1.9997889411762922e-07,
               2.729681222011256e-07,
               3.2532371115080884e-07])

m = np.array([3.309504692539563e-07,
                 4.1058904888091606e-06,
                 6.8326703386053605e-06,
                 7.4616147721799645e-06])
</code></pre>
<p>what confuses me is that I tried adding what I thought was to small of a number so that it returned no difference but it did return a difference (i.e. I tried to do <code>a+eps = a</code> using <code>eps = smaller than machine precision</code>):</p>
<pre><code class="lang-auto">import torch

x1 = torch.tensor(1e-6)
x2 = torch.tensor(1e-7)
x3 = torch.tensor(1e-8)
x4 = torch.tensor(1e-9)

eps = torch.tensor(1e-11)

print(x1.dtype)
print(x1)
print(x1+eps)

print(x2)
print(x2+eps)

print(x3)
print(x3+eps)

print(x4)
print(x4+eps)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">torch.float32
tensor(1.0000e-06)
tensor(1.0000e-06)
tensor(1.0000e-07)
tensor(1.0001e-07)
tensor(1.0000e-08)
tensor(1.0010e-08)
tensor(1.0000e-09)
tensor(1.0100e-09)
</code></pre>
<p>I expected everything to be zero but it wasn’t. Can someone explain to me what is going on? If I am getting losses close to <code>1e-7</code> should I use <code>double</code> rather than <code>float</code>? googling it seems that single is the precision for float afaik.</p>
<hr>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/63818676/what-is-the-machine-precision-in-pytorch-and-when-should-one-use-doubles">
  <header class="source">

      <a href="https://stackoverflow.com/questions/63818676/what-is-the-machine-precision-in-pytorch-and-when-should-one-use-doubles" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=256&amp;g=1" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/63818676/what-is-the-machine-precision-in-pytorch-and-when-should-one-use-doubles" target="_blank" rel="noopener nofollow ugc">What is the machine precision in pytorch and when should one use doubles?</a>
</h4>

<div class="tags">
  <strong>floating-point, pytorch, precision</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/63818676/what-is-the-machine-precision-in-pytorch-and-when-should-one-use-doubles" target="_blank" rel="noopener nofollow ugc">07:52PM - 09 Sep 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="quote quote-modified" data-post="1" data-topic="95715">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-does-one-start-using-double-without-unexpected-bugs/95715">How does one start using double without unexpected bugs?</a> 
  </div>
  <blockquote>
    On a related issue to <a href="https://discuss.pytorch.org/t/what-is-the-machine-precision-of-pytorch-with-cpus-or-gpus/9384">this</a>, I get errors in my regression that are close to machine precision (10^-7). As I tried things out machine precision things might be a lot more subtle than I expected, so I want to make sure. Is changing all the parameters to double in my model sufficient so that EVERYTHING is now computed using double to avoid errors? 
(btw read that post for more detail comments on issues if your curious) 
But what I want is: 

change code so everything uses double
hope that there are…
  </blockquote>
</aside>


username: Brando_Miranda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for tagging you. Can someone route me to the person that might know things about numerical issues? I am getting an error of <code>loss = 4.650724394150382e-16</code> but from what I read in the docs (<a href="https://pytorch.org/docs/stable/tensors.html" rel="nofollow noopener">https://pytorch.org/docs/stable/tensors.html</a>) it seems that pytorch is using single by default and not double…so why is my loss at a scale of <code>-16</code> if that’s much lower than what it should handle as far as I understand? In a different place I have <code>e-7</code>…so have I hit zero or not? would I be safer if the loss was at <code>-5</code> or higher and no confusion if it’s zero or not?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="5" data-topic="9384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>so why is my loss at a scale of <code>-16</code> if that’s much lower than what it should handle as far as I understand? In a different place I have <code>e-7</code> …so have I hit zero or not?</p>
</blockquote>
</aside>
<p><code>1e-6</code> is not the absolute minimal value before the value is rounded to zero as explained e.g. <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">here</a>.<br>
As you can see in the <code>Precision limitation on decimal values</code> section, the fixed interval between “small integer values” is approx. <code>1e-7</code>, which is why this can be used as the minimal step size between these values.<br>
The interpretation is of course depending on your use case and e.g. in the ML domain absolute errors of <code>~1e-6</code> are commonly accepted.</p>
<p>Here is a small example using the minimal denormal values as well as larger values with a fixed interval of 1:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.set_printoptions(precision=30)

a = torch.tensor(2**(-149)) # smalles denormal number
print(a)

b = torch.tensor(2**(-150)) # not representable
print(b)

x = torch.tensor(1.)
y = x + torch.tensor(2**(-23)) # addition representable
print(y)
print(x - y)

z = x + torch.tensor(2**(-24)) # not representable
print(z)

c = torch.tensor(2**23) # step size is 1
d = c + torch.tensor(0.5) # rounded
print(d - c)
e = c + torch.tensor(1.) # representable
print(e - c)
</code></pre>

username: Brando_Miranda
cooked: <p>I’ve read the article you sent sort of carefully plus this one (<a href="https://www.doc.ic.ac.uk/~eedwards/compsys/float" rel="nofollow noopener">https://www.doc.ic.ac.uk/~eedwards/compsys/float</a>).</p>
<p>What I am trying to understand is when is adding two numbers result in errors. My understanding from the article is that 7.33 decimal places is what is “remembered” in the mantissa. But it is also tracking the floating point location with the exponent.</p>
<p>So my current understanding is that if both numbers are in the same exponent range then adding them is fine (as long as it’s not smaller than 7.3 decimals…and if has parts in that range then the computer wouldn’t even be able to store that part anyway).</p>
<p>So what I am struggling to really articulate is when we get into issues. If we have both number be <code>(1.0 + 0.M)*2^-7</code> and <code>(1.0+M)*2^4</code> range…would we get into problems?</p>
<p>I guess what I’m trying to understand is when underflows and overflows occur (and hoping that is the only type of error).</p>
<p>My loss is really small (from my synthetic data set), so it’s not a standard scenario.</p>

username: Brando_Miranda
cooked: <p>Ok let’s see if this is a good summary of what I think is correct (modulo ignoring some details that I don’t fully understand right now of floats, like the bias).</p>
<p>But I’ve concluded that the best thing for me is to make sure my errors/numbers have two properties:</p>
<ul>
<li>
<ol>
<li>they are within 7decimals of each other (due to the mantissa being 24 bigs like you pointed out the log_10(2^24) = 7.225)</li>
</ol>
</li>
<li>
<ol>
<li>they are far enough from the edges. For this I take the mantissa to be 23 bits away from the lower edge (point position about -128+23) and the same for the largest edge but 127-23.</li>
</ol>
</li>
</ul>
<p>As long we satisfy that more or less we avoid adding two numbers that are too small for the machine to distinguish (condition 1) and avoid overflows/underflows (condition 2).</p>
<p>Perhaps there is a small detail I might be missing with the bias or some other float detail (like representing infinity, NaN). But I believe that is correct.</p>
<p>If anyone can correct the details, that would be fantastic.</p>
<hr>
<p>my current conclusion is because most learning rates are  <code>1e-1</code> ish I will have everything be 7 decimals from zero. That seems the safest. As lost as every number is in the range of 1.0 ± 7 decimals it should be fine. This really guarantees there won’t be weird machine precision if we stick to an absolute scale rather than a fixed one.</p>

username: liangbright
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="9384">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>in the ML domain absolute errors of <code>~1e-6</code> are commonly accepted.</p>
</blockquote>
</aside>
<p>“in the ML domain absolute errors of <code>~1e-6</code> are commonly accepted.” Could you provide a reference paper? so that I can put it in my paper to justify 1e-6. thank you.</p>

username: ptrblck
cooked: <p>No, I don’t have a reference paper at hand.</p>