Repository: numpy/numpy

Issue:
numpy.digitize uses a linear search when it should be using binary search (Trac #2064)
_Original ticket http://projects.scipy.org/numpy/ticket/2064 on 2012-02-25 by trac user jpeel, assigned to unknown._

In numpy/lib/src/_compiled_base.c, numpy.digitize's code uses linear searches rather than binary searches as it should. The difference in speed is quite enormous when using very many bins. Quite frankly, it should probably use searchsorted after checking that the bins are monotonic.

This was opened as a result of a question on stackoverflow
[http://stackoverflow.com/questions/9444409/why-is-numpy-much-slower-than-matlab-on-a-digitize-example/9445500#9445500]
You can see an example of the speed issue there.



Discussion:
_trac user Sven Marnach wrote on 2012-02-25_

The function `searchsorted()` does basically the same thing as `digitize()`, but uses a binary search.

```
numpy.digitize(a, b)
```

is equivalent to

```
numpy.searchsorted(b, a, "right")
```

Maybe `digitize()` should be removed (or wrapped to call `searchsorted()`).

What's the status here? I have the same issue. I am rebinning data using `np.histogram2d(.., weights=data)/np.histogram2d(..)` and it is really slow.

Even though `searchsorted` is notably faster in 1.9 than it was before, it still cannot beat the linear search in `digitize` for the small number of bins that are the default for histogram-like functions. There are some graphs in the discussion of PR #4184 that illustrate this. I may try to revisit that once more, but I recall not finding an entirely satisfactory solution for all input sizes.

I see. Indeed my use case is around >200 bins. Replacing `Ncount[i] = digitize(sample[:, i], edges[i])` with `Ncount[i] = searchsorted(edges[i], sample[:, i], "right")` in `histogramdd` leads to a 2.5x speedup in my case (which is significant as I have several million points wanting to be rebinned). For now I live with having a customized `histogramdd` and `histogram2d` in my code base, but I really want to get rid of it.

I did some benchmarking with numpy 1.9.0 on that issue. You can look at the results here:
http://nbviewer.ipython.org/gist/skuschel/1599698bd271be63284f
So it turns out, that in the worst case (2 bins) digitize is about 2.5 times faster than searchsorted. But with increasing bin number searchsorted scales **significantly** better.

So the easiest solution is to use searchsorted always: Since you would generally tend to use more bins if you have more data points, i cannot think of an use case where you would have more than 1e6 samples, but want to use a few bins _only_. In this case you would loose some ms, who would even realize this?

But I can totally understand, that people dont want to make an algorithm worse, so to me there is one obvious solution: The histogramdd function chooses either digitize or searchsorted according to how many bins are requested. On my computer the cross over point is around 72 bins. But in any case, even a rough guess should be good enough. That should be an easy fix, shouldnt it? I can implement it, but I wanted to double check, since there was quite some discussion going on that topic.

Searchsorted could probably fall back on linear search once the search range fell below some cutoff. The numpy quicksort does that, switching over to insertion sort at about 20.


Resolving patch:
diff --git a/numpy/add_newdocs.py b/numpy/add_newdocs.py
--- a/numpy/add_newdocs.py
+++ b/numpy/add_newdocs.py
@@ -4837,14 +4837,15 @@ def luf(lamdaexpr, *args, **kwargs):
     Parameters
     ----------
     x : array_like
-        Input array to be binned. It has to be 1-dimensional.
+        Input array to be binned. Prior to Numpy 1.10.0, this array had to
+        be 1-dimensional, but can now have any shape.
     bins : array_like
         Array of bins. It has to be 1-dimensional and monotonic.
     right : bool, optional
         Indicating whether the intervals include the right or the left bin
         edge. Default behavior is (right==False) indicating that the interval
-        does not include the right edge. The left bin and is open in this
-        case. Ie., bins[i-1] <= x < bins[i] is the default behavior for
+        does not include the right edge. The left bin end is open in this
+        case, i.e., bins[i-1] <= x < bins[i] is the default behavior for
         monotonically increasing bins.
 
     Returns
@@ -4855,7 +4856,7 @@ def luf(lamdaexpr, *args, **kwargs):
     Raises
     ------
     ValueError
-        If the input is not 1-dimensional, or if `bins` is not monotonic.
+        If `bins` is not monotonic.
     TypeError
         If the type of the input is complex.
 
@@ -4869,6 +4870,13 @@ def luf(lamdaexpr, *args, **kwargs):
     attempting to index `bins` with the indices that `digitize` returns
     will result in an IndexError.
 
+    .. versionadded:: 1.10.0
+
+    `np.digitize` is  implemented in terms of `np.searchsorted`. This means
+    that a binary search is used to bin the values, which scales much better
+    for larger number of bins than the previous linear search. It also removes
+    the requirement for the input array to be 1-dimensional.
+
     Examples
     --------
     >>> x = np.array([0.2, 6.4, 3.0, 1.6])
@@ -4885,7 +4893,7 @@ def luf(lamdaexpr, *args, **kwargs):
     1.0 <= 1.6 < 2.5
 
     >>> x = np.array([1.2, 10.0, 12.4, 15.5, 20.])
-    >>> bins = np.array([0,5,10,15,20])
+    >>> bins = np.array([0, 5, 10, 15, 20])
     >>> np.digitize(x,bins,right=True)
     array([1, 2, 3, 4, 4])
     >>> np.digitize(x,bins,right=False)


Test patch: