Repository: numpy/numpy

Issue:
'auto' selection of histogram bins fails in some cases
This is an odd corner case, but there are times using auto to select the number of bins gives only a single bin.

    from pylab import *
    data = zeros(1000)
    data[:100] = 10
    np.histogram(data, bins='auto')

Output is a single bin

     (array([1000]), array([  0.,  10.]))

This must be the result of the sturges/fd estimators since (according to the documentation) auto just takes the max of these two.  It seems like this should fall back to the default number of bins (10 I think), or perhaps sqrt.


Discussion:
Looking at this some more it appears to be more than just undesirable behavior, documentation states that 'Auto'  is the "maximum of the 'Sturges' and 'FD' estimators."  Actual code returns the min, `min(_hist_bin_fd(x), _hist_bin_sturges(x))`.
Yes, its a bug. But the `min` is correct there since that is the width not the number of bins, the problem is that it returns an invalid result for `fd` (0), and then that 0 is turned into 1 as a fail safe. Practically, we probably want to use the other estimator when one fails. Don't remember who implemented this mostly, maybe we can ping them if they are happy to look at it.
Though the fix should be simple indeed. so if you feel like it, please do give it a shot!
@nayyarv or @madphysicist want to have a look fix this? Just thought I would ping since you guys worked on this part a lot.
Haha, there are always these unintended side effects I never even considered. The sturges backup was meant to catch situations like this, but i think it works better as max of bins than as min of bandwidth for this very reason.

I'll put something together this evening, but as 
Continued
@seberg said, this shouldn't be overly complicated for a bug fix.
@nayyarv I have an idea for a solution that I would like to run by you. I think that the user should be able to get a reasonable histogram here even if they select `'fd'`, so falling back on the other estimator is not necessarily the best option here. Instead, we could modify `_hist_bin_fd` to check if the IQR is zero because the data is shaped unfortunately vs actually having only one value. For example, if IQR is zero, use `np.unique` to see how many distinct values there are and use the min/max values instead.
Did this ever get resolved? / Will this get resolved? Just ran into this on a project I'm working on. 
@eileenjkim This will get fixed. I am currently a bit swamped with other projects, but it is close to the top of my list when I get some free time. I am doing a bit of research into this and other modifications to histograms right now actually. I can't promise a quick turnaround, but I will get to it...
Thanks @madphysicist ! Any ideas on what is causing this failure? I'm working with a pretty narrow distribution of data, but it seems to occur randomly with my project. Wondering if this is a reliable failure that I can plan for and avoid.

Resolving patch:
diff --git a/numpy/lib/histograms.py b/numpy/lib/histograms.py
--- a/numpy/lib/histograms.py
+++ b/numpy/lib/histograms.py
@@ -167,12 +167,22 @@ def _hist_bin_fd(x):
 def _hist_bin_auto(x):
     """
     Histogram bin estimator that uses the minimum width of the
-    Freedman-Diaconis and Sturges estimators.
+    Freedman-Diaconis and Sturges estimators if the FD bandwidth is non zero
+    and the Sturges estimator if the FD bandwidth is 0.
 
     The FD estimator is usually the most robust method, but its width
-    estimate tends to be too large for small `x`. The Sturges estimator
-    is quite good for small (<1000) datasets and is the default in the R
-    language. This method gives good off the shelf behaviour.
+    estimate tends to be too large for small `x` and bad for data with limited
+    variance. The Sturges estimator is quite good for small (<1000) datasets
+    and is the default in the R language. This method gives good off the shelf
+    behaviour.
+
+    .. versionchanged:: 1.15.0
+    If there is limited variance the IQR can be 0, which results in the
+    FD bin width being 0 too. This is not a valid bin width, so
+    ``np.histogram_bin_edges`` chooses 1 bin instead, which may not be optimal.
+    If the IQR is 0, it's unlikely any variance based estimators will be of
+    use, so we revert to the sturges estimator, which only uses the size of the
+    dataset in its calculation.
 
     Parameters
     ----------
@@ -188,10 +198,13 @@ def _hist_bin_auto(x):
     --------
     _hist_bin_fd, _hist_bin_sturges
     """
-    # There is no need to check for zero here. If ptp is, so is IQR and
-    # vice versa. Either both are zero or neither one is.
-    return min(_hist_bin_fd(x), _hist_bin_sturges(x))
-
+    fd_bw = _hist_bin_fd(x)
+    sturges_bw = _hist_bin_sturges(x)
+    if fd_bw:
+        return min(fd_bw, sturges_bw)
+    else:
+        # limited variance, so we return a len dependent bw estimator
+        return sturges_bw
 
 # Private dict initialized at module load time
 _hist_bin_selectors = {'auto': _hist_bin_auto,
@@ -440,7 +453,7 @@ def histogram_bin_edges(a, bins=10, range=None, weights=None):
     below, :math:`h` is the binwidth and :math:`n_h` is the number of
     bins. All estimators that compute bin counts are recast to bin width
     using the `ptp` of the data. The final bin count is obtained from
-    ``np.round(np.ceil(range / h))`.
+    ``np.round(np.ceil(range / h))``.
 
     'Auto' (maximum of the 'Sturges' and 'FD' estimators)
         A compromise to get a good value. For small datasets the Sturges


Test patch:
diff --git a/numpy/lib/tests/test_histograms.py b/numpy/lib/tests/test_histograms.py
--- a/numpy/lib/tests/test_histograms.py
+++ b/numpy/lib/tests/test_histograms.py
@@ -443,6 +443,24 @@ def test_novariance(self):
             assert_equal(len(a), numbins, err_msg="{0} estimator, "
                          "No Variance test".format(estimator))
 
+    def test_limited_variance(self):
+        """
+        Check when IQR is 0, but variance exists, we return the sturges value
+        and not the fd value.
+        """
+        lim_var_data = np.ones(1000)
+        lim_var_data[:3] = 0
+        lim_var_data[-4:] = 100
+
+        edges_auto = histogram_bin_edges(lim_var_data, 'auto')
+        assert_equal(edges_auto, np.linspace(0, 100, 12))
+
+        edges_fd = histogram_bin_edges(lim_var_data, 'fd')
+        assert_equal(edges_fd, np.array([0, 100]))
+
+        edges_sturges = histogram_bin_edges(lim_var_data, 'sturges')
+        assert_equal(edges_sturges, np.linspace(0, 100, 12))
+
     def test_outlier(self):
         """
         Check the FD, Scott and Doane with outliers.