Repository: numpy/numpy

Issue:
Output of np.finfo is misleading
It appears to me that either the documentation of `np.finfo()` is somewhat misleading or the functions returns a wrong result for the parameter `tiny`.

According to the doc of `np.finfo()` tiny is 

> the smallest positive usable number. 

According to the script below that is not true. 

I am not sure if this is really a bug. I suspect that tiny in `np.finfo()` actually refers to the smallest _normalized_ floating-point number. However, the IEEE 754 standard also includes _subnormal_ floating-point numbers. This is also implemented in np.float64. The smallest positive subnormal is `2**(-1022)*2**(-52)`. See [Wiki](https://en.wikipedia.org/wiki/Denormal_number).

It may make sense to distinguish between these two classes of floats since the latter does not have the full resolution of up to 16 decimals, but may have considerably less precision. However, from my point of view `np.float64` supports subnormals and they are usable numbers. Hence either the documentation should be adapted or a further attribute may be added to `np.finfo` such as `tiny_norm` vs. `tiny_subn`.

### Reproducing code example:

```python
import numpy as np

print('Output of np.finfo(np.float64)')
print(np.finfo(np.float64))

# value for tiny equals
tiny = 2**(-1022)

# define some smaller number
x = np.zeros(3)
x[0] = 2**(-1023)
x[1] = 2**(-1074)
x[2] = 2**(-1075) # underflow occurs

print('Print decimal representations of tiny and some smaller floats:')
print('2**(-1022) = Tiny = ', tiny)
print('2**(-1023) =', x[0])
print('2**(-1074) =', x[1])
print('2**(-1075) =', x[2])
```

### Output:

```
Output of np.finfo(np.float64)
Machine parameters for float64
---------------------------------------------------------------
precision =  15   resolution = 1.0000000000000001e-15
machep =    -52   eps =        2.2204460492503131e-16
negep =     -53   epsneg =     1.1102230246251565e-16
minexp =  -1022   tiny =       2.2250738585072014e-308
maxexp =   1024   max =        1.7976931348623157e+308
nexp =       11   min =        -max
---------------------------------------------------------------

Print decimal representations of tiny and some smaller floats:
2**(-1022) = Tiny =  2.2250738585072014e-308
2**(-1023) = 1.1125369292536007e-308
2**(-1074) = 5e-324
2**(-1075) = 0.0
```

<!-- If you are reporting a segfault please include a GDB traceback, which you
can generate by following
https://github.com/numpy/numpy/blob/master/doc/source/dev/development_environment.rst#debugging -->

<!-- Full error message, if any (starting from line Traceback: ...) -->

### Numpy/Python version information:
<!-- Output from 'import sys, numpy; print(numpy.__version__, sys.version)' -->
1.16.2 3.7.5 (default, Apr 19 2020, 20:18:17) 
[GCC 9.2.1 20191008]




Discussion:
@rk-mlu you are right, that is the smallest non-subnormal number, we should maybe add that as a comment in the docs. I suppose we could add a `tiny_subnormal`, not sure its typically relevant since you quickly start losing precision at that point.

Can you make a PR suggesting a change in the documentation? That would be great!
@seberg No problem, I will write up a comment for the documentation.

Resolving patch:
diff --git a/numpy/core/getlimits.py b/numpy/core/getlimits.py
--- a/numpy/core/getlimits.py
+++ b/numpy/core/getlimits.py
@@ -337,8 +337,8 @@ class finfo:
         The approximate decimal resolution of this type, i.e.,
         ``10**-precision``.
     tiny : float
-        The smallest positive usable number.  Type of `tiny` is an
-        appropriate floating point type.
+        The smallest positive floating point number with full precision
+        (see Notes).
 
     Parameters
     ----------
@@ -359,6 +359,18 @@ class finfo:
     impacts import times.  These objects are cached, so calling ``finfo()``
     repeatedly inside your functions is not a problem.
 
+    Note that ``tiny`` is not actually the smallest positive representable
+    value in a NumPy floating point type. As in the IEEE-754 standard [1]_,
+    NumPy floating point types make use of subnormal numbers to fill the
+    gap between 0 and ``tiny``. However, subnormal numbers may have
+    significantly reduced precision [2]_.
+    
+    References
+    ----------
+    .. [1] IEEE Standard for Floating-Point Arithmetic, IEEE Std 754-2008,
+           pp.1-70, 2008, http://www.doi.org/10.1109/IEEESTD.2008.4610935
+    .. [2] Wikipedia, "Denormal Numbers",
+           https://en.wikipedia.org/wiki/Denormal_number
     """
 
     _finfo_cache = {}
@@ -546,4 +558,3 @@ def __str__(self):
     def __repr__(self):
         return "%s(min=%s, max=%s, dtype=%s)" % (self.__class__.__name__,
                                     self.min, self.max, self.dtype)
-
diff --git a/numpy/core/machar.py b/numpy/core/machar.py
--- a/numpy/core/machar.py
+++ b/numpy/core/machar.py
@@ -40,8 +40,8 @@ class MachAr:
         Smallest (most negative) power of `ibeta` consistent with there
         being no leading zeros in the mantissa.
     xmin : float
-        Floating point number ``beta**minexp`` (the smallest [in
-        magnitude] usable floating value).
+        Floating-point number ``beta**minexp`` (the smallest [in
+        magnitude] positive floating point number with full precision).
     maxexp : int
         Smallest (positive) power of `ibeta` that causes overflow.
     xmax : float


Test patch: