Repository: scipy/scipy

Issue:
inaccurate std implementation in binned_statistic
stats/_binned_statistic_dd.py uses numerically unstable method for calculating standard deviation:
```
flatsum = np.bincount(binnumbers, values[vv])
flatsum2 = np.bincount(binnumbers, values[vv] ** 2)
result[vv, a] = np.sqrt(flatsum2[a] / flatcount[a] -
                        (flatsum[a] / flatcount[a]) ** 2)
```
If the input data contains large numbers, the squared numbers in the subtraction are very large. As the result precision can be lost - a well known problem in numerical computing.

The effect can be shown by increasing the input data values in the unit tests, for example:
```
cls.v = np.random.random(100) + 1e4
```
instead of the default
```
cls.v = np.random.random(100)
```
and all std-related unit tests will fail.

A better implementation would first calculate the means of the bins and then square (data-mean). Another safe option is just to use np.std but this could be slow.




Discussion:
I just looked at this on my dev box and not sure the example illustrates the issue:

```
(scipy-dev) Lucass-MacBook:stats rlucas$ python
Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy.stats as ss
>>> import numpy as np
>>> cls = np.random.random(100) + 1e4
>>> np.std(cls)
0.2854385260083998
>>> flatsum = np.sum(cls)
>>> flatsum
1000055.4102408785
>>> flatsum2 = np.sum(cls**2)
>>> flatsum
1000055.4102408785
>>> flatsum2
10001108243.668032
>>> flatcount=len(cls)
>>> np.sqrt(flatsum2 / flatcount - (flatsum / flatcount) ** 2)
0.285438502420829
>>> np.sqrt(flatsum2 / flatcount - (flatsum / flatcount) ** 2) - np.std(cls)
-2.3587570785466028e-08
```
The 2 approaches are accurate to 8 digits, I think the `1e4`  is not quite large enough and a larger value will exhibit the effect of violating cardinal sin of numeric computing.

```
>>> cls = np.random.random(100) + 1e8
>>> flatsum = np.sum(cls)
>>> flatsum2 = np.sum(cls**2)
>>> flatcount=len(cls)
>>> np.sqrt(flatsum2 / flatcount - (flatsum / flatcount) ** 2) - np.std(cls)
1.7091614844465095
>>> np.std(cls)
0.2908385155534905
>>> np.sqrt(flatsum2 / flatcount - (flatsum / flatcount) ** 2)
2.0
```

The standard approach to this is [Welford's algorithm](https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Welford's_online_algorithm). I can look into implementing that in this case but I'm not sure that it is necessary, `np.std` should be fast and that would be an easy fix (I'm happy to implement either solution but will try a PR with the `np.std` approach even though Welford would be super fun opportunity to implement.

Would like to get some input from others in `stats` though as I'm not as familiar with this method or what type of broadcasting is necessary to support. If broadcasting is necessary that might require something more than Welford. 

@chrisb83 @pvanmulbregt @WarrenWeckesser  any thoughts or preferences on which way to proceed here? 
looks like `numpy.std` has something like the Welform algorithm (accounting for large numbers) built in:
https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/calculation.c#L360-L513

I'll look a little closer at the supported uses for the `binned_statistic` method and see if the numpy method can solve the problem. 



Welford's algorithm appears to be an online algorithm used when computing the variance in a single pass.  Here the data is being traversed twice already, so the simplest approach may just be
to first subtract the mean before squaring.  Continuing the example from https://github.com/scipy/scipy/issues/10126#issuecomment-489867040

```
flatcount = len(cls)
flatsum = np.sum(cls)
flatsum2 = np.sum((cls-flatsum/flatcount)**2)
std = np.sqrt(flatsum2a/flatcount)
```


@pvanmulbregt  that is a good idea on the way to proceed, I will write the unit test with the example above and try the 2 pass approach.  What precisely constitutes Welford and what constitutes something else is fuzzy to me. 

Presumably this approach works so I expect a PR on this by weekend. 

Resolving patch:
diff --git a/scipy/stats/_binned_statistic.py b/scipy/stats/_binned_statistic.py
--- a/scipy/stats/_binned_statistic.py
+++ b/scipy/stats/_binned_statistic.py
@@ -562,11 +562,10 @@ def binned_statistic_dd(sample, values, statistic='mean',
         result.fill(0)
         flatcount = np.bincount(binnumbers, None)
         a = flatcount.nonzero()
-        for vv in xrange(Vdim):
-            flatsum = np.bincount(binnumbers, values[vv])
-            flatsum2 = np.bincount(binnumbers, values[vv] ** 2)
-            result[vv, a] = np.sqrt(flatsum2[a] / flatcount[a] -
-                                    (flatsum[a] / flatcount[a]) ** 2)
+        for i in np.unique(binnumbers):
+            for vv in xrange(Vdim):
+                #NOTE: take std dev by bin, np.std() is 2-pass and stable
+                result[vv, i] = np.std(values[vv, binnumbers == i])
     elif statistic == 'count':
         result.fill(0)
         flatcount = np.bincount(binnumbers, None)


Test patch:
diff --git a/scipy/stats/tests/test_binned_statistic.py b/scipy/stats/tests/test_binned_statistic.py
--- a/scipy/stats/tests/test_binned_statistic.py
+++ b/scipy/stats/tests/test_binned_statistic.py
@@ -19,6 +19,7 @@ def setup_class(cls):
         cls.v = np.random.random(100)
         cls.X = np.random.random((100, 3))
         cls.w = np.random.random(100)
+        cls.u = np.random.random(100) + 1e6
 
     def test_1d_count(self):
         x = self.x
@@ -39,6 +40,17 @@ def test_gh5927(self):
         for statistic in statistics:
             res = binned_statistic(x, v, statistic, bins=10)
 
+    def test_big_number_std(self):
+        # tests for numerical stability of std calculation
+        # see issue gh-10126 for more
+        x = self.x
+        u = self.u
+
+        stat1, edges1, bc = binned_statistic(x, u, 'std', bins=10)
+        stat2, edges2, bc = binned_statistic(x, u, np.std, bins=10)
+
+        assert_allclose(stat1, stat2)
+
     def test_1d_result_attributes(self):
         x = self.x
         v = self.v