Repository: numpy/numpy

Issue:
histogram failure due to loss of precision
`histogram` fails after estimating a negative count when all of the following conditions are met:
- The `low` edge of the `range` argument is a `float64`,
- The array for which the histogram is calculated is a `float32`,
- The value for `low` does not occur in the array,
- The value for `low` when converted to float32 _does_ occur in the array

The code below shows an example of how this happens:

```
In [142]: print(sys.version)
3.5.0 (default, Nov 20 2015, 16:20:41) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-16)]

In [143]: print(numpy.__version__)
1.11.2

In [155]: histogram(array([-24.35791367], dtype=float32), range=(-24.3579135, 0))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-155-57aa993972c6> in <module>()
----> 1 histogram(array([-24.35791367], dtype=float32), range=(-24.3579135, 0))

/dev/shm/gerrit/venv/stable-3.5/lib/python3.5/site-packages/numpy/lib/function_base.py in histogram(a, bins, range, normed, weights, density)
    609                 n.imag += np.bincount(indices, weights=tmp_w.imag, minlength=bins)
    610             else:
--> 611                 n += np.bincount(indices, weights=tmp_w, minlength=bins).astype(ntype)
    612 
    613         # Rename the bin edges for return.

ValueError: The first argument of bincount must be non-negative
```

BUG: fixes #8123 Histogram precision issue due to higher precision in internal mn, mx
When the input data hsa a lower precision than the specified ranges and error is raised in bincount.
This has been addressed in issues #8123 

The problem is caused by higher precision ````mn````, ````mx````. In this pull request, i cast ````mn```` and ````mx```` to the same type as the input data 

(messed up the previous pull request...could not reopen it)
Bug in histogram: incorrectly handles float32 inputs just below lower bin limit
I am passing a very benign array of `float32` values to `np.histogram(...)` which raises a `ValueError: The first argument of bincount must be non-negative`. The values contain no NaN or infinite values. I think this is a bug, and it is due to incorrect handling of single-precision values in the histogram function. Specifically, values that are _equal_ to the bin limits when compared as single-precision but are _outside them_ when compared as doubles.

First, a quick test code that works for me with 64-bit floats but raises the error with 32-bit floats on the last line:

```python
import numpy as np
a64=np.array([25552.234, 26000])

# Make a histogram from these two values. Put the lower bin limit just above the lower value.
bin_limits = (a64[0]+0.0005, 26200)
np.histogram(a64, 10, bin_limits)

# That worked. But repeat the exercise with the data converted to float32...
a32=a64.astype(np.float32)

# ... The following raises a ValueError:
np.histogram(a32, 10, bin_limits)
```

The full output in iPython looks like this:

```python
In [29]: a64=np.array([25552.234, 26000])

In [30]: bin_limits = (a64[0]+0.0005, 26200)

In [31]: np.histogram(a64, 10, bin_limits)
Out[31]: 
(array([0, 0, 0, 0, 0, 0, 1, 0, 0, 0]),
 array([ 25552.2345 ,  25617.01105,  25681.7876 ,  25746.56415,
         25811.3407 ,  25876.11725,  25940.8938 ,  26005.67035,
         26070.4469 ,  26135.22345,  26200.     ]))

In [32]: a32=a64.astype(np.float32)

In [33]: np.histogram(a32, 10, bin_limits)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-0a3f12cea374> in <module>()
----> 1 np.histogram(a32, 10, bin_limits)

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/function_base.pyc in histogram(a, bins, range, normed, weights, density)
    778             else:
    779                 n += np.bincount(indices, weights=tmp_w,
--> 780                                  minlength=bins).astype(ntype)
    781 
    782         # Rename the bin edges for return.

ValueError: The first argument of bincount must be non-negative

In [34]: np.version.version
Out[34]: '1.13.3'
```

Notice that `a32[0]` is equal to the lower bin limit or not, depending on whether we compare as float32 or as float:

```python
In [37]: a32==bin_limits[0]
Out[37]: array([ True, False], dtype=bool)

In [38]: a32[0]==bin_limits[0]
Out[38]: False

```

Looking into `function_base.py`, we can see the problem: 

```python
   740          for i in arange(0, len(a), BLOCK):
   741              tmp_a = a[i:i+BLOCK]
   742              if weights is None:
   743                  tmp_w = None
   744              else:
   745                  tmp_w = weights[i:i + BLOCK]
   746  
   747              # Only include values in the right range
   748              keep = (tmp_a >= mn)
   749              keep &= (tmp_a <= mx)
   750              if not np.logical_and.reduce(keep):
   751                  tmp_a = tmp_a[keep]
   752                  if tmp_w is not None:
   753                      tmp_w = tmp_w[keep]
   754              tmp_a_data = tmp_a.astype(float)
   755              tmp_a = tmp_a_data - mn
   756              tmp_a *= norm
   757  
   758              # Compute the bin indices, and for values that lie exactly on mx we
   759              # need to subtract one
   760              indices = tmp_a.astype(np.intp)
   761              indices[indices == bins] -= 1
   762  
   763              # The index computation is not guaranteed to give exactly
   764              # consistent results within ~1 ULP of the bin edges.
   765              decrement = tmp_a_data < bin_edges[indices]
   766              indices[decrement] -= 1
   767              # The last bin includes the right edge. The other bins do not.
   768              increment = ((tmp_a_data >= bin_edges[indices + 1])
   769                           & (indices != bins - 1))
   770              indices[increment] += 1
   771  
   772              # We now compute the histogram using bincount
   773              if ntype.kind == 'c':
   774                  n.real += np.bincount(indices, weights=tmp_w.real,
   775                                        minlength=bins)
   776                  n.imag += np.bincount(indices, weights=tmp_w.imag,
   777                                        minlength=bins)
   778              else:
   779                  n += np.bincount(indices, weights=tmp_w,
   780                                   minlength=bins).astype(ntype)
   781  
```

Up to line 753, we are discarding data that fall outside the given histogram limits. At lines 748 and 751, array `tmp_a` is still whatever type the user passes in (here, `float32), and the decision is made whether to keep data *based on the original float32 values*. But in line 755, the `tmp_a` is replaced with a (double-precision) float array. When the single-precision values are equal to the lowest bin edge *in single-precision comparisons* but then turn out to be just below the lowest edge in double-precision comparisons, then we get this error, erroneously.

I could update and make a PR, but I'm completely new to numpy and don't know what approach you'd like best. Some ideas:

1. You could just force values in the `indices` array to be in-range. This seems to be sort of what lines 760-770 do, so I'm not clear how it fails in the case I've raised. I think what we need is an analogue to line 761 but for the lower edge. Perhaps `indices[indices < 0] = 0` ? This behavior means that we bin float32 data with float32 comparisons, which might or might not be what we want.

1. Convert the input array to `float` before deciding which values to keep. This could be a performance problem in cases where most values will be outside the histogram limits, because you'd be converting all values. (The current approach only converts the ones that will be within limits, except that "within limits" is not quite correct for float32 values.)

1. Replace the `tmp_a.astype(float)` (line 754) so that we convert not to `float` but to the input type. I think this would solve the problem for all floating-point inputs but likely create some new problem for integer inputs. This might require that handling floats and ints differently with separate branches.

I'd say that we want to go with the first idea but am hoping an expert can weigh in.

I am using with numpy 1.13.1 on Mac OS 10.12.6 through the Macports system, with Python 2.7.14 and IPython 5.4.0.



Discussion:
Nice catch! Something like this:

``` diff
diff --git a/numpy/lib/function_base.py b/numpy/lib/function_base.py
index cb1a04f..1a93e01 100644
--- a/numpy/lib/function_base.py
+++ b/numpy/lib/function_base.py
@@ -733,7 +733,7 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
         # is 2x as fast) and it results in a memory footprint 3x lower in the
         # limit of large arrays.
         for i in arange(0, len(a), BLOCK):
-            tmp_a = a[i:i+BLOCK]
+            tmp_a = a[i:i+BLOCK].astype(float)
             if weights is None:
                 tmp_w = None
             else:
@@ -746,7 +746,7 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
                 tmp_a = tmp_a[keep]
                 if tmp_w is not None:
                     tmp_w = tmp_w[keep]
-            tmp_a_data = tmp_a.astype(float)
+            tmp_a_data = tmp_a
             tmp_a = tmp_a_data - mn
             tmp_a *= norm
```

should fix it.

this fix would fail if ````a```` is not castable to  ````float````.
This needs a test-case for the thing you're trying to fix. Commit message should start with `BUG: ` and include something like `fixes #8123` to make it auto-close the issue.

Also, you'll need to get the other tests to pass too ;). Thanks for solving the rebasing issues
ok, tnx. I'll do the changes. The tests are failing locally too, just wanted to push to have the fix on github.
i think i solved the problem. Finding a clean test case was a bit tricky since the data set where i encountered this problem was like 2 GB and had to bracket down the offending number. Turned out
the problem can be reproduced with just one input value.

I don't like that with this solution the result will depend on the `dtype` of the value even for a perfectly well-represented value like `1`: for `float32` it is counted as inside the bin, but for float64 and int you get 0. I think in your test case, the value should simply be counted as being out of range.

I think the real reason this goes wrong is that the array comparison with a numpy scalar is not properly done (which is actually a bigger bug...):
```python
# not good: float32 array with float64 array scalar (or regular scalar)
np.array([1.], dtype=np.float32)>=np.array(1.00000001, dtype=np.float64)
# array([ True], dtype=bool)

# not good float32 array with python float
np.array([1.], dtype=np.float32)>=1.00000001
# array([ True], dtype=bool)

# OK float32 array with float64 array
np.array([1.], dtype=np.float32)>=np.array([1.00000001], dtype=np.float64)
# array([False], dtype=bool)

# also OK: (array) scalar with (array) scalar
np.array(1., dtype=np.float32)>=np.array(1.00000001, dtype=np.float64)
# False
```
one solution here is to cast `mn`, `mx` always as float64 arrays with shape (1,). But perhaps we should fix the real problem... See gh-9194
p.s. I fear solving the real problem will take some discussion about what the desired behaviour for all ufuncs is, so perhaps we should go ahead and address the issue here. Which would mean avoiding array scalars. Or maybe more cleanly, do the inclusion of values in the right range *after* calculating the indices (i.e., remove indices that are too large or small).
actually I thought about that too. that 1 < 1 + tiny so the count should be zero.

maybe casting to the input data and the range (mn, mx) to the highest precision of both ? 

we should keep in mind that there is also numpy.float128

      type                   precision
      ------                 ------------
      float32                1.1920929e-07
      float64                2.2204460492503131e-16
      float128               1.084202172485504434e-19
Good point about `float128` -- my own feeling would be to have as little as possible of these complications in `histogram` itself. Indeed, I would suggest to remove those `+ 0.0`, which seem only necessary to allow an in-place subtraction/addition of 0.5 -- instead, just write those as explicit assignments: `mn = mn - 0.5`, etc.  And then check the indices instead of the array before sending off to bincount.
I've gone ahead and rebased this - some of the fixes were already applied in #10278 
@mherkazandjian, if you want to continue working on this, you should do

```
git fetch
git checkout histogram-precision
git reset --hard origin/histogram-precision
```
@mhvk: Your comments were addressed in #10278, but it seems that the extra patch here (or similar) is needed too.
I think the rebased PR looks good, modulo the small request for adding a comment on why the dtype case is done.
Actually, I think we want a variation on idea 1:

4. Acknowledge that the first selection (lines 747-753) is only _approximate_ for float32 data. Thus, do a second selection at line 770 that catches (and removes) errors made in the first, approximate selection. Something like:

```python
# The first "keep" is only approximately correct when the input data are single-precision float32.
# To prevent bincount from raising an error, select the data again.
keep = indices >= 0
keep &= indices < bins
indices = indices[keep]
if tmp_w is not None:
    tmp_w = tmp_w[keep]
```

For float data and (I think) for integer data, this new `keep` will be all-True, but it will catch slight numerical problems in single-precision inputs like the one raised in this issue. For this reason, perhaps it would be best to do this block only for floating but not full-precision floating inputs?
This is a duplicate of multiple issues, and has a candidate fix at #9189
I think the problem comes down to the fact that the bounds are `float64`, but the values are `float32`. Clearly we're sometimes down-casting the bounds, and sometimes upcasting the values, which is inconsistent.

The code in question has moved to here:

https://github.com/numpy/numpy/blob/0953c4d9543bd5e3c007f1013cf93779314ca327/numpy/lib/histograms.py#L594-L634
Thanks Eric, I agree with the assessment about down- vs up-casting values. Glad there is a fix in the works.
Hmm, this is rather alarming (#10322)

```python
>>> np.float32([1.0]) >= 1.00000001
array([ True])
>>> np.float32(1.0) >= 1.00000001
False
```
  

Resolving patch:
diff --git a/numpy/lib/histograms.py b/numpy/lib/histograms.py
--- a/numpy/lib/histograms.py
+++ b/numpy/lib/histograms.py
@@ -318,9 +318,17 @@ def _get_bin_edges(a, bins, range, weights):
         raise ValueError('`bins` must be 1d, when an array')
 
     if n_equal_bins is not None:
+        # gh-10322 means that type resolution rules are dependent on array
+        # shapes. To avoid this causing problems, we pick a type now and stick
+        # with it throughout.
+        bin_type = np.result_type(first_edge, last_edge, a)
+        if np.issubdtype(bin_type, np.integer):
+            bin_type = np.result_type(bin_type, float)
+
         # bin edges must be computed
         bin_edges = np.linspace(
-            first_edge, last_edge, n_equal_bins + 1, endpoint=True)
+            first_edge, last_edge, n_equal_bins + 1,
+            endpoint=True, dtype=bin_type)
         return bin_edges, (first_edge, last_edge, n_equal_bins)
     else:
         return bin_edges, None
@@ -605,21 +613,24 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
                 tmp_a = tmp_a[keep]
                 if tmp_w is not None:
                     tmp_w = tmp_w[keep]
-            tmp_a_data = tmp_a.astype(float)
-            tmp_a = tmp_a_data - first_edge
-            tmp_a *= norm
+
+            # This cast ensures no type promotions occur below, which gh-10322
+            # make unpredictable. Getting it wrong leads to precision errors
+            # like gh-8123.
+            tmp_a = tmp_a.astype(bin_edges.dtype, copy=False)
 
             # Compute the bin indices, and for values that lie exactly on
             # last_edge we need to subtract one
-            indices = tmp_a.astype(np.intp)
+            f_indices = (tmp_a - first_edge) * norm
+            indices = f_indices.astype(np.intp)
             indices[indices == n_equal_bins] -= 1
 
             # The index computation is not guaranteed to give exactly
             # consistent results within ~1 ULP of the bin edges.
-            decrement = tmp_a_data < bin_edges[indices]
+            decrement = tmp_a < bin_edges[indices]
             indices[decrement] -= 1
             # The last bin includes the right edge. The other bins do not.
-            increment = ((tmp_a_data >= bin_edges[indices + 1])
+            increment = ((tmp_a >= bin_edges[indices + 1])
                          & (indices != n_equal_bins - 1))
             indices[increment] += 1
 


Test patch:
diff --git a/numpy/lib/tests/test_histograms.py b/numpy/lib/tests/test_histograms.py
--- a/numpy/lib/tests/test_histograms.py
+++ b/numpy/lib/tests/test_histograms.py
@@ -299,6 +299,53 @@ def test_datetime(self):
         assert_equal(d_edge.dtype, dates.dtype)
         assert_equal(t_edge.dtype, td)
 
+    def do_precision_lower_bound(self, float_small, float_large):
+        eps = np.finfo(float_large).eps
+
+        arr = np.array([1.0], float_small)
+        range = np.array([1.0 + eps, 2.0], float_large)
+
+        # test is looking for behavior when the bounds change between dtypes
+        if range.astype(float_small)[0] != 1:
+            return
+
+        # previously crashed
+        count, x_loc = np.histogram(arr, bins=1, range=range)
+        assert_equal(count, [1])
+
+        # gh-10322 means that the type comes from arr - this may change
+        assert_equal(x_loc.dtype, float_small)
+
+    def do_precision_upper_bound(self, float_small, float_large):
+        eps = np.finfo(float_large).eps
+
+        arr = np.array([1.0], float_small)
+        range = np.array([0.0, 1.0 - eps], float_large)
+
+        # test is looking for behavior when the bounds change between dtypes
+        if range.astype(float_small)[-1] != 1:
+            return
+
+        # previously crashed
+        count, x_loc = np.histogram(arr, bins=1, range=range)
+        assert_equal(count, [1])
+
+        # gh-10322 means that the type comes from arr - this may change
+        assert_equal(x_loc.dtype, float_small)
+
+    def do_precision(self, float_small, float_large):
+        self.do_precision_lower_bound(float_small, float_large)
+        self.do_precision_upper_bound(float_small, float_large)
+
+    def test_precision(self):
+        # not looping results in a useful stack trace upon failure
+        self.do_precision(np.half, np.single)
+        self.do_precision(np.half, np.double)
+        self.do_precision(np.half, np.longdouble)
+        self.do_precision(np.single, np.double)
+        self.do_precision(np.single, np.longdouble)
+        self.do_precision(np.double, np.longdouble)
+
 
 class TestHistogramOptimBinNums(object):
     """