Repository: pandas-dev/pandas

Issue:
Problem with pandas.Series.std()
Hi,

I ran the following commands on "pandas.Series" object named "S":

> > > S.std()
> > >  39192.660667875185
> > > S.values.std()
> > >  17570.025893232512
> > > numpy.std(S)
> > >  39192.500822771792
> > > numpy.std(S.values)
> > >  17570.025893232512

Correct standard deviation is 17570.02589
I should have got the same value in all 4 cases.

This seems to be a bug in pandas.Series
Pickle file for "S" object is here:
https://gist.github.com/yashoteja/4971970

Thank you,
Yashoteja



Discussion:
Hi,

I am new to github as a whole.
If there are any issues with the format of my previous post,
please inform me so that I would rectify them next time onwards.

Thanks,
Yashoteja

I believe this is the usual 1/sqrt(N) vs. 1/sqrt(N-1) issue.
numpy and pandas apperently use different defaults for the 
`ddof` parameter. pandas default to 1/sqrt(N-1)

``` python
a=pd.Series(np.array(range(10)))

In [6]: a.std()
Out[6]: 3.0276503540974917

In [7]: a.values.std(ddof=1)
Out[7]: 3.0276503540974917

In [8]: a.values.std()
Out[8]: 2.8722813232690143

In [9]: a.std(ddof=0)
Out[9]: 2.8722813232690143
```

hey actually I loaded his dataset, it's apparently an `int64` overflow and/or roundoff issue somewhere, because it doesn't happen when you cast his Series to `float64` (it's a big series with a lot of large `int64` values)...so this is legitimately a bug.

``` python
In [5]: S.astype('float64').std()
Out[5]: 17570.097551905212

In [6]: S.astype('int64').std()
Out[6]: 39192.660667875185
```

still figuring out exactly where the overflow/roundoff is happening.

got it, in "nanops.py" with some print statements added:

``` python
    X = _ensure_numeric(values.sum(axis))
    XX = _ensure_numeric((values ** 2).sum(axis))
    print type(X)
    print X
    print X ** 2
```

yields

```
<type 'numpy.int64'>
4063418664
-1935372834766006720
```

this would work if `X` were a builtin `int` because they overflow to `long` automatically, but numpy `int64`s don't...

i can do a patch, but I'm not sure if the correct solution is to cast `values` to `float64` first here before doing anything, or to cast `int64` to `int`/`long` after the fact in case it's an `int64`? the two would yield slightly different results, the latter would be faster (since it's casting up a scalar rather than casting up an array), but the latter would be a bit uglier-looking of a hack too (IMHO). also I'm not 100% sure it would cover all cases because I don't know the overflow rules within `ndarray.__pow__` and `ndarray.sum()` for `int64` arrays..

anyone have any thoughts?

actually...I think the only safe thing to do is to cast everything to `float64` upfront, because you can overflow within `ndarray.__pow__` and `ndarray.sum()` too:

``` python
In [6]: np.asarray([4063418664], dtype='int64')
Out[6]: array([4063418664], dtype=int64)

In [7]: np.asarray([4063418664], dtype='int64') ** 2
Out[7]: array([-1935372834766006720], dtype=int64)

In [8]: np.asarray([2**63-1,1], dtype='int64')
Out[8]: array([9223372036854775807,                   1], dtype=int64)

In [9]: np.asarray([2**63-1,1], dtype='int64').sum()
Out[9]: -9223372036854775808
```

(or we can just avoid calling `std()` in nanops.py in case of integers, since they won't have NaNs to being with...)

Upcast to float is fine because stdev should always yield a floating point number. 


Resolving patch:
diff --git a/pandas/core/nanops.py b/pandas/core/nanops.py
--- a/pandas/core/nanops.py
+++ b/pandas/core/nanops.py
@@ -138,6 +138,9 @@ def get_median(x):
 
 
 def _nanvar(values, axis=None, skipna=True, ddof=1):
+    if not isinstance(values.dtype.type, np.floating):
+        values = values.astype('f8')
+
     mask = isnull(values)
 
     if axis is not None:


Test patch:
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -1401,7 +1401,8 @@ def testit():
 
             # check the result is correct
             nona = self.series.dropna()
-            assert_almost_equal(f(nona), alternate(nona))
+            assert_almost_equal(f(nona), alternate(nona.values))
+            assert_almost_equal(f(self.series), alternate(nona.values))
 
             allna = self.series * nan
             self.assert_(np.isnan(f(allna)))
@@ -1410,6 +1411,12 @@ def testit():
             s = Series([1, 2, 3, None, 5])
             f(s)
 
+            # 2888
+            l = [0]
+            l.extend(list(range(2**40,2**40+1000)))
+            s = Series(l, dtype='int64')
+            assert_almost_equal(float(f(s)), float(alternate(s.values)))
+
             # check date range
             if check_objects:
                 s = Series(bdate_range('1/1/2000', periods=10))