Repository: numpy/numpy

Issue:
np.histogram() inconsistent results
I think that #6100 may have introduced a bug in some edge cases. Here is a case found through fuzz testing:

```
|35> import numpy as np
|36> np.__version__
'1.10.4'
|37> arr = np.array([337, 404, 739, 806, 1007, 1811, 2012])
|39> hist, edges = np.histogram(arr, bins=8296, range=(2, 2280))
|40> mask = hist > 0
|41> left_edges = edges[:-1][mask]
|42> right_edges = edges[1:][mask]
|43> zip(arr, left_edges, right_edges)
[(337, 337.00000000000006, 337.27459016393448),
 (404, 404.00000000000006, 404.27459016393448),
 (739, 739.00000000000011, 739.27459016393448),
 (806, 806.00000000000011, 806.27459016393448),
 (1007, 1007.0000000000001, 1007.2745901639345),
 (1811, 1811.0000000000002, 1811.2745901639346),
 (2012, 2012.0000000000002, 2012.2745901639346)]
```

This was found through fuzz testing an internal accelerated histogram routine (which takes roughly the same strategy as #6100 but implemented in Cython) and comparing the results to `np.histogram()`. Comparing with numpy 1.9.2, everything worked identically. At least as of numpy 1.10.4, some of these edge cases popped up.

I believe the main difference between #6100 and our routine is that we do not precompute the histogram scaling factor `bins / (mx - mn)`: https://github.com/numpy/numpy/blob/master/numpy/lib/function_base.py#L638
We compute `((tmp_a - mn) * bins) / (mx - mn)` . The comments of our routine mention floating point problems as the reason for deliberately avoiding precomputation.



Discussion:
Pinging @astrofrog 

@rkern Want to make a PR?

Hmm, looks like it might be `linspace()` to blame, actually.

Well, no, we use `linspace()` too. Hmm.

Ah, I see. We are doing some extra work to correct the location in case it happens to be off by one in these edge cases. Not sure if there is a fast way to do this vectorized in Python.


Resolving patch:
diff --git a/numpy/lib/function_base.py b/numpy/lib/function_base.py
--- a/numpy/lib/function_base.py
+++ b/numpy/lib/function_base.py
@@ -639,6 +639,9 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
         # Pre-compute histogram scaling factor
         norm = bins / (mx - mn)
 
+        # Compute the bin edges for potential correction.
+        bin_edges = linspace(mn, mx, bins + 1, endpoint=True)
+
         # We iterate over blocks here for two reasons: the first is that for
         # large arrays, it is actually faster (for example for a 10^8 array it
         # is 2x as fast) and it results in a memory footprint 3x lower in the
@@ -657,14 +660,22 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
                 tmp_a = tmp_a[keep]
                 if tmp_w is not None:
                     tmp_w = tmp_w[keep]
-            tmp_a = tmp_a.astype(float)
-            tmp_a -= mn
+            tmp_a_data = tmp_a.astype(float)
+            tmp_a = tmp_a_data - mn
             tmp_a *= norm
 
             # Compute the bin indices, and for values that lie exactly on mx we
             # need to subtract one
             indices = tmp_a.astype(np.intp)
-            indices[indices == bins] -= 1
+            equals_endpoint = (indices == bins)
+            indices[equals_endpoint] -= 1
+
+            # The index computation is not guaranteed to give exactly
+            # consistent results within ~1 ULP of the bin edges.
+            decrement = tmp_a_data < bin_edges[indices]
+            indices[decrement] -= 1
+            increment = (tmp_a_data >= bin_edges[indices + 1]) & ~equals_endpoint
+            indices[increment] += 1
 
             # We now compute the histogram using bincount
             if ntype.kind == 'c':
@@ -673,8 +684,8 @@ def histogram(a, bins=10, range=None, normed=False, weights=None,
             else:
                 n += np.bincount(indices, weights=tmp_w, minlength=bins).astype(ntype)
 
-        # We now compute the bin edges since these are returned
-        bins = linspace(mn, mx, bins + 1, endpoint=True)
+        # Rename the bin edges for return.
+        bins = bin_edges
     else:
         bins = asarray(bins)
         if (np.diff(bins) < 0).any():


Test patch:
diff --git a/numpy/lib/tests/test_function_base.py b/numpy/lib/tests/test_function_base.py
--- a/numpy/lib/tests/test_function_base.py
+++ b/numpy/lib/tests/test_function_base.py
@@ -1407,6 +1407,17 @@ def test_finite_range(self):
         assert_raises(ValueError, histogram, vals, range=[np.nan,0.75])
         assert_raises(ValueError, histogram, vals, range=[0.25,np.inf])
 
+    def test_bin_edge_cases(self):
+        # Ensure that floating-point computations correctly place edge cases.
+        arr = np.array([337, 404, 739, 806, 1007, 1811, 2012])
+        hist, edges = np.histogram(arr, bins=8296, range=(2, 2280))
+        mask = hist > 0
+        left_edges = edges[:-1][mask]
+        right_edges = edges[1:][mask]
+        for x, left, right in zip(arr, left_edges, right_edges):
+            self.assertGreaterEqual(x, left)
+            self.assertLess(x, right)
+
 
 class TestHistogramOptimBinNums(TestCase):
     """