Repository: pandas-dev/pandas

Issue:
ENH: Standard Error of the Mean (sem) aggregation method
A very common operation when trying to work with data is to find out the error range for the data.  In scientific research, including error ranges is required.

There are two main ways to do this: standard deviation and standard error of the mean.  Pandas has an optimized std aggregation method for both dataframe and groupby.  However, it does not have an optimized standard error method, meaning users who want to compute error ranges have to rely on the unoptimized scipy method.  

Since computing error ranges is such a common operation, I think it would be very useful if there was an optimized `sem` method like there is for `std`.



Discussion:
Does statsmodels do this?
On Apr 17, 2014 2:27 AM, "toddrjen" notifications@github.com wrote:

> A very common operation when trying to work with data is to find out the
> error range for the data. In scientific research, including error ranges is
> required.
> 
> There are two main ways to do this: standard deviation and standard error
> of the mean. Pandas has an optimized std aggregation method for both
> dataframe and groupby. However, it does not have an optimized standard
> error method, meaning users who want to compute error ranges have to rely
> on the unoptimized scipy method.
> 
> Since computing error ranges is such a common operation, I think it would
> be very useful if there was an optimized sem method like there is for std.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/6897
> .

Not as far as I can find.  And I don't think it really belongs in statsmodels.  In my opinion it is a pretty basic data wrangling task, like getting a mean or standard deviation, not the more advanced statistical modeling provided by statsmodel.

can u point to the scipy method?

http://docs.scipy.org/doc/scipy-0.13.0/reference/generated/scipy.stats.sem.html

@toddrjen What do you mean with an optimized method? `std` is optimized, so you don't have to rely on an 'unoptimized' scipy.stats method, you can just do: `df.std()/len(df)` 

And by the way, scipy.stats.sem is not that 'unoptimized'. In fact, it is even faster, as this does not do eg the extra nan-checking as pandas does: 

```
In [2]: s = pd.Series(np.random.randn(1000))

In [7]: from scipy import stats

In [8]: stats.sem(s.values)
Out[8]: 0.031635197968083853

In [9]: s.std() / np.sqrt(len(s))
Out[9]: 0.031635197968083832

In [11]: %timeit stats.sem(s.values)
10000 loops, best of 3: 46.2 µs per loop

In [12]: %timeit s.std() / np.sqrt(len(s))
10000 loops, best of 3: 85.7 µs per loop

In [12]: %timeit s.std() / np.sqrt(len(s))
```

But of course, the question still remains, do we provide a shortcut to this functionality in the form of a `sem` method, or do we just expect out users to divide the std themselves.

would be code-bloat IMHO, closing

thanks for the suggestion.

if you disagree, pls comment.

@jreback i don't think this is code bloat relative to the alternative:

You can't _really_ use `scipy.stats.sem` because it doesn't handle `nan`s:

```
In [19]: from scipy.stats import sem

In [20]: df = DataFrame(np.random.randn(10, 3), columns=['a', 'b', 'c'])

In [21]: df
Out[21]:
        a       b       c
0  1.1658  0.2184 -2.0823
1  0.5625 -0.5034  0.7028
2 -0.8424  0.1333 -1.1065
3  0.9335 -0.6088  1.4308
4 -0.1027 -0.1888 -0.5816
5 -0.5202  0.3210 -0.9942
6 -0.8666  0.8711 -0.5691
7 -0.7701 -2.1855 -0.4302
8  1.0664 -1.2672  0.7117
9 -0.7530 -0.8466  0.0194

[10 rows x 3 columns]

In [22]: sem(df[df > 0])
Out[22]: array([ nan,  nan,  nan])
```

Okay, so let's try it with `scipy.stats.mstats.sem`:

```
In [26]: from scipy.stats.mstats import sem as sem

In [27]: sem(df[df > 0])
Out[27]:
masked_array(data = [-- -- --],
             mask = [ True  True  True],
       fill_value = 1e+20)
```

That's hardly what I would expect here, and masked arrays are almost as fun as recarrays. I'm +1 on reopening this.

Here's what it would take to get the desired result from `scipy`:

```
In [32]: Series(sem(np.ma.masked_invalid(df[df > 0])),index=df.columns)
Out[32]:
a    0.1321
b    0.1662
c    0.2881
dtype: float64

In [33]: df[df > 0].std() / sqrt(df[df > 0].count())
Out[33]:
a    0.1321
b    0.1662
c    0.2881
dtype: float64
```

no, but isn't this just

``s.std()/np.sqrt(len(s))` and even that's 'arbitrary' in my book

not an issue with the code-bloat per se, but the definition

agreed. that's really simple. i was just making a point about the nan handling, you can't just do `len` because that counts nans. not a huge deal

not averse to this, but it just seems so simple that a user should do this (as I might want a different definition); that said if this is pretty 'standard' then would be ok

every science institution i've ever worked in (just 3 really so not a _whole_ lot of weight there) has used `sem` at some point (even if just to get a rough idea of error ranges). i see your point about different definitions, maybe other folks want to chime in

ok...will reopen for consideration in 0.15 then

I  have also been at three different institutions, and they also all used SEM.  And I have seen it on hundreds of papers, presentations, and posters.

@toddrjen 

ok...that's fine then, pls submit a PR! (needs to go in `core/nanops.py`) with some updating in `core/ops.py`

Pull request submitted: #7133 


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3794,7 +3794,8 @@ def mad(self,  axis=None, skipna=None, level=None, **kwargs):
 
         @Substitution(outname='variance',
                       desc="Return unbiased variance over requested "
-                           "axis\nNormalized by N-1")
+                           "axis.\n\nNormalized by N-1 by default. "
+                           "This can be changed using the ddof argument")
         @Appender(_num_doc)
         def var(self, axis=None, skipna=None, level=None, ddof=1, **kwargs):
             if skipna is None:
@@ -3811,7 +3812,8 @@ def var(self, axis=None, skipna=None, level=None, ddof=1, **kwargs):
 
         @Substitution(outname='stdev',
                       desc="Return unbiased standard deviation over requested "
-                           "axis\nNormalized by N-1")
+                           "axis.\n\nNormalized by N-1 by default. "
+                           "This can be changed using the ddof argument")
         @Appender(_num_doc)
         def std(self, axis=None, skipna=None, level=None, ddof=1, **kwargs):
             if skipna is None:
@@ -3827,6 +3829,24 @@ def std(self, axis=None, skipna=None, level=None, ddof=1, **kwargs):
             return np.sqrt(result)
         cls.std = std
 
+        @Substitution(outname='standarderror',
+                      desc="Return unbiased standard error of the mean over "
+                           "requested axis.\n\nNormalized by N-1 by default. "
+                           "This can be changed using the ddof argument")
+        @Appender(_num_doc)
+        def sem(self, axis=None, skipna=None, level=None, ddof=1, **kwargs):
+            if skipna is None:
+                skipna = True
+            if axis is None:
+                axis = self._stat_axis_number
+            if level is not None:
+                return self._agg_by_level('sem', axis=axis, level=level,
+                                          skipna=skipna, ddof=ddof)
+
+            return self._reduce(nanops.nansem, axis=axis, skipna=skipna,
+                                ddof=ddof)
+        cls.sem = sem
+
         @Substitution(outname='compounded',
                       desc="Return the compound percentage of the values for "
                            "the requested axis")
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -486,7 +486,7 @@ def __getattr__(self, attr):
                              (type(self).__name__, attr))
 
     def __getitem__(self, key):
-        raise NotImplementedError
+        raise NotImplementedError('Not implemented: %s' % key)
 
     def _make_wrapper(self, name):
         if name not in self._apply_whitelist:
@@ -695,12 +695,7 @@ def std(self, ddof=1):
         For multiple groupings, the result index will be a MultiIndex
         """
         # todo, implement at cython level?
-        if ddof == 1:
-            return self._cython_agg_general('std')
-        else:
-            self._set_selection_from_grouper()
-            f = lambda x: x.std(ddof=ddof)
-            return self._python_agg_general(f)
+        return np.sqrt(self.var(ddof=ddof))
 
     def var(self, ddof=1):
         """
@@ -715,6 +710,14 @@ def var(self, ddof=1):
             f = lambda x: x.var(ddof=ddof)
             return self._python_agg_general(f)
 
+    def sem(self, ddof=1):
+        """
+        Compute standard error of the mean of groups, excluding missing values
+
+        For multiple groupings, the result index will be a MultiIndex
+        """
+        return self.std(ddof=ddof)/np.sqrt(self.count())
+
     def size(self):
         """
         Compute group sizes
@@ -1332,7 +1335,6 @@ def get_group_levels(self):
             'name': 'group_median'
         },
         'var': 'group_var',
-        'std': 'group_var',
         'first': {
             'name': 'group_nth',
             'f': lambda func, a, b, c, d: func(a, b, c, d, 1)
@@ -1341,10 +1343,6 @@ def get_group_levels(self):
         'count': 'group_count',
     }
 
-    _cython_transforms = {
-        'std': np.sqrt,
-    }
-
     _cython_arity = {
         'ohlc': 4,  # OHLC
     }
@@ -1455,7 +1453,6 @@ def aggregate(self, values, how, axis=0):
 
     def _aggregate(self, result, counts, values, how, is_numeric):
         agg_func, dtype = self._get_aggregate_function(how, values)
-        trans_func = self._cython_transforms.get(how, lambda x: x)
 
         comp_ids, _, ngroups = self.group_info
         if values.ndim > 3:
@@ -1469,7 +1466,7 @@ def _aggregate(self, result, counts, values, how, is_numeric):
         else:
             agg_func(result, counts, values, comp_ids)
 
-        return trans_func(result)
+        return result
 
     def agg_series(self, obj, func):
         try:
@@ -1669,7 +1666,6 @@ def names(self):
         'min': 'group_min_bin',
         'max': 'group_max_bin',
         'var': 'group_var_bin',
-        'std': 'group_var_bin',
         'ohlc': 'group_ohlc',
         'first': {
             'name': 'group_nth_bin',
@@ -1688,7 +1684,6 @@ def names(self):
     def _aggregate(self, result, counts, values, how, is_numeric=True):
 
         agg_func, dtype = self._get_aggregate_function(how, values)
-        trans_func = self._cython_transforms.get(how, lambda x: x)
 
         if values.ndim > 3:
             # punting for now
@@ -1699,7 +1694,7 @@ def _aggregate(self, result, counts, values, how, is_numeric=True):
         else:
             agg_func(result, counts, values, self.bins)
 
-        return trans_func(result)
+        return result
 
     def agg_series(self, obj, func):
         dummy = obj[:0]
diff --git a/pandas/core/nanops.py b/pandas/core/nanops.py
--- a/pandas/core/nanops.py
+++ b/pandas/core/nanops.py
@@ -308,6 +308,24 @@ def get_median(x):
     return _wrap_results(get_median(values), dtype) if notempty else np.nan
 
 
+def _get_counts_nanvar(mask, axis, ddof):
+    count = _get_counts(mask, axis)
+
+    d = count-ddof
+
+    # always return NaN, never inf
+    if np.isscalar(count):
+        if count <= ddof:
+            count = np.nan
+            d = np.nan
+    else:
+        mask2 = count <= ddof
+        if mask2.any():
+            np.putmask(d, mask2, np.nan)
+            np.putmask(count, mask2, np.nan)
+    return count, d
+
+
 @disallow('M8')
 @bottleneck_switch(ddof=1)
 def nanvar(values, axis=None, skipna=True, ddof=1):
@@ -316,31 +334,28 @@ def nanvar(values, axis=None, skipna=True, ddof=1):
 
     mask = isnull(values)
 
-    if axis is not None:
-        count = (values.shape[axis] - mask.sum(axis)).astype(float)
-    else:
-        count = float(values.size - mask.sum())
+    count, d = _get_counts_nanvar(mask, axis, ddof)
 
-    d = count-ddof
     if skipna:
         values = values.copy()
         np.putmask(values, mask, 0)
 
-    # always return NaN, never inf
-    if np.isscalar(count):
-        if count <= ddof:
-            count = np.nan
-            d = np.nan
-    else:
-        mask = count <= ddof
-        if mask.any():
-            np.putmask(d, mask, np.nan)
-            np.putmask(count, mask, np.nan)
-
     X = _ensure_numeric(values.sum(axis))
     XX = _ensure_numeric((values ** 2).sum(axis))
     return np.fabs((XX - X ** 2 / count) / d)
 
+
+def nansem(values, axis=None, skipna=True, ddof=1):
+    var = nanvar(values, axis, skipna, ddof=ddof)
+
+    if not isinstance(values.dtype.type, np.floating):
+        values = values.astype('f8')
+    mask = isnull(values)
+    count, _ = _get_counts_nanvar(mask, axis, ddof)
+
+    return np.sqrt(var)/np.sqrt(count)
+
+
 @bottleneck_switch()
 def nanmin(values, axis=None, skipna=True):
     values, mask, dtype, dtype_max = _get_values(values, skipna, fill_value_typ='+inf')


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -10864,6 +10864,23 @@ def test_var_std(self):
             self.assertFalse((result < 0).any())
             nanops._USE_BOTTLENECK = True
 
+    def test_sem(self):
+        alt = lambda x: np.std(x, ddof=1)/np.sqrt(len(x))
+        self._check_stat_op('sem', alt)
+
+        result = self.tsframe.sem(ddof=4)
+        expected = self.tsframe.apply(lambda x: x.std(ddof=4)/np.sqrt(len(x)))
+        assert_almost_equal(result, expected)
+
+        arr = np.repeat(np.random.random((1, 1000)), 1000, 0)
+        result = nanops.nansem(arr, axis=0)
+        self.assertFalse((result < 0).any())
+        if nanops._USE_BOTTLENECK:
+            nanops._USE_BOTTLENECK = False
+            result = nanops.nansem(arr, axis=0)
+            self.assertFalse((result < 0).any())
+            nanops._USE_BOTTLENECK = True
+
     def test_skew(self):
         _skip_if_no_scipy()
         from scipy.stats import skew
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -681,11 +681,14 @@ def _check_results(grouped):
             assert_frame_equal(result, expected)
 
             # group frame by function dict
-            result = grouped.agg(
-                OrderedDict([['A', 'var'], ['B', 'std'], ['C', 'mean']]))
+            result = grouped.agg(OrderedDict([['A', 'var'],
+                                              ['B', 'std'],
+                                              ['C', 'mean'],
+                                              ['D', 'sem']]))
             expected = DataFrame(OrderedDict([['A', grouped['A'].var()],
                                               ['B', grouped['B'].std()],
-                                              ['C', grouped['C'].mean()]]))
+                                              ['C', grouped['C'].mean()],
+                                              ['D', grouped['D'].sem()]]))
             assert_frame_equal(result, expected)
 
         by_weekday = self.tsframe.groupby(lambda x: x.weekday())
@@ -1633,8 +1636,13 @@ def _testit(op):
             result = op(grouped)['C']
             assert_series_equal(result, exp)
 
+        _testit(lambda x: x.count())
         _testit(lambda x: x.sum())
+        _testit(lambda x: x.std())
+        _testit(lambda x: x.var())
+        _testit(lambda x: x.sem())
         _testit(lambda x: x.mean())
+        _testit(lambda x: x.median())
         _testit(lambda x: x.prod())
         _testit(lambda x: x.min())
         _testit(lambda x: x.max())
@@ -4166,8 +4174,8 @@ def test_tab_completion(self):
             'agg','aggregate','apply','boxplot','filter','first','get_group',
             'groups','hist','indices','last','max','mean','median',
             'min','name','ngroups','nth','ohlc','plot', 'prod',
-            'size','std','sum','transform','var', 'count', 'head', 'describe',
-            'cummax', 'quantile', 'rank', 'cumprod', 'tail',
+            'size', 'std', 'sum', 'transform', 'var', 'sem', 'count', 'head',
+            'describe', 'cummax', 'quantile', 'rank', 'cumprod', 'tail',
             'resample', 'cummin', 'fillna', 'cumsum', 'cumcount',
             'all', 'shift', 'skew', 'bfill', 'irow', 'ffill',
             'take', 'tshift', 'pct_change', 'any', 'mad', 'corr', 'corrwith',
@@ -4306,6 +4314,61 @@ def __eq__(self, other):
                                                            name='grp'))
         tm.assert_frame_equal(result, expected)
 
+    def test__cython_agg_general(self):
+        ops = [('mean', np.mean),
+               ('median', np.median),
+               ('var', np.var),
+               ('add', np.sum),
+               ('prod', np.prod),
+               ('min', np.min),
+               ('max', np.max),
+               ('first', lambda x: x.iloc[0]),
+               ('last', lambda x: x.iloc[-1]),
+               ('count', np.size),
+               ]
+        df = DataFrame(np.random.randn(1000))
+        labels = np.random.randint(0, 50, size=1000).astype(float)
+
+        for op, targop in ops:
+            result = df.groupby(labels)._cython_agg_general(op)
+            expected = df.groupby(labels).agg(targop)
+            try:
+                tm.assert_frame_equal(result, expected)
+            except BaseException as exc:
+                exc.args += ('operation: %s' % op,)
+                raise
+
+    def test_ops_general(self):
+        ops = [('mean', np.mean),
+               ('median', np.median),
+               ('std', np.std),
+               ('var', np.var),
+               ('sum', np.sum),
+               ('prod', np.prod),
+               ('min', np.min),
+               ('max', np.max),
+               ('first', lambda x: x.iloc[0]),
+               ('last', lambda x: x.iloc[-1]),
+               ('count', np.size),
+               ]
+        try:
+            from scipy.stats import sem
+        except ImportError:
+            pass
+        else:
+            ops.append(('sem', sem))
+        df = DataFrame(np.random.randn(1000))
+        labels = np.random.randint(0, 50, size=1000).astype(float)
+
+        for op, targop in ops:
+            result = getattr(df.groupby(labels), op)().astype(float)
+            expected = df.groupby(labels).agg(targop)
+            try:
+                tm.assert_frame_equal(result, expected)
+            except BaseException as exc:
+                exc.args += ('operation: %s' % op,)
+                raise
+
 
 def assert_fp_equal(a, b):
     assert (np.abs(a - b) < 1e-12).all()
diff --git a/pandas/tests/test_multilevel.py b/pandas/tests/test_multilevel.py
--- a/pandas/tests/test_multilevel.py
+++ b/pandas/tests/test_multilevel.py
@@ -1378,7 +1378,7 @@ def test_count(self):
         self.assertRaises(KeyError, frame.count, level='x')
 
     AGG_FUNCTIONS = ['sum', 'prod', 'min', 'max', 'median', 'mean', 'skew',
-                     'mad', 'std', 'var']
+                     'mad', 'std', 'var', 'sem']
 
     def test_series_group_min_max(self):
         for op, level, skipna in cart_product(self.AGG_FUNCTIONS,
diff --git a/pandas/tests/test_panel.py b/pandas/tests/test_panel.py
--- a/pandas/tests/test_panel.py
+++ b/pandas/tests/test_panel.py
@@ -127,6 +127,13 @@ def alt(x):
             return np.std(x, ddof=1)
         self._check_stat_op('std', alt)
 
+    def test_sem(self):
+        def alt(x):
+            if len(x) < 2:
+                return np.nan
+            return np.std(x, ddof=1)/np.sqrt(len(x))
+        self._check_stat_op('sem', alt)
+
     # def test_skew(self):
     #     from scipy.stats import skew
 
diff --git a/pandas/tests/test_panel4d.py b/pandas/tests/test_panel4d.py
--- a/pandas/tests/test_panel4d.py
+++ b/pandas/tests/test_panel4d.py
@@ -98,6 +98,13 @@ def alt(x):
             return np.std(x, ddof=1)
         self._check_stat_op('std', alt)
 
+    def test_sem(self):
+        def alt(x):
+            if len(x) < 2:
+                return np.nan
+            return np.std(x, ddof=1)/np.sqrt(len(x))
+        self._check_stat_op('sem', alt)
+
     # def test_skew(self):
     #     from scipy.stats import skew
 
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -1980,6 +1980,19 @@ def test_var_std(self):
         result = s.std(ddof=1)
         self.assertTrue(isnull(result))
 
+    def test_sem(self):
+        alt = lambda x: np.std(x, ddof=1)/np.sqrt(len(x))
+        self._check_stat_op('sem', alt)
+
+        result = self.ts.sem(ddof=4)
+        expected = np.std(self.ts.values, ddof=4)/np.sqrt(len(self.ts.values))
+        assert_almost_equal(result, expected)
+
+        # 1 - element series with ddof=1
+        s = self.ts.iloc[[0]]
+        result = s.sem(ddof=1)
+        self.assert_(isnull(result))
+
     def test_skew(self):
         _skip_if_no_scipy()
 
diff --git a/pandas/tseries/tests/test_resample.py b/pandas/tseries/tests/test_resample.py
--- a/pandas/tseries/tests/test_resample.py
+++ b/pandas/tseries/tests/test_resample.py
@@ -5,7 +5,8 @@
 from pandas.compat import range, lrange, zip, product
 import numpy as np
 
-from pandas import Series, TimeSeries, DataFrame, Panel, isnull, notnull, Timestamp
+from pandas import (Series, TimeSeries, DataFrame, Panel, Index,
+                    isnull, notnull, Timestamp)
 
 from pandas.tseries.index import date_range
 from pandas.tseries.offsets import Minute, BDay
@@ -104,6 +105,48 @@ def test_resample_basic(self):
         expect = s.groupby(grouper).agg(lambda x: x[-1])
         assert_series_equal(result, expect)
 
+    def test_resample_how(self):
+        rng = date_range('1/1/2000 00:00:00', '1/1/2000 00:13:00',
+                            freq='min', name='index')
+        s = Series(np.random.randn(14), index=rng)
+        grouplist = np.ones_like(s)
+        grouplist[0] = 0
+        grouplist[1:6] = 1
+        grouplist[6:11] = 2
+        grouplist[11:] = 3
+        args = ['sum', 'mean', 'std', 'sem', 'max', 'min',
+                'median', 'first', 'last', 'ohlc']
+
+        def _ohlc(group):
+            if isnull(group).all():
+                return np.repeat(np.nan, 4)
+            return [group[0], group.max(), group.min(), group[-1]]
+        inds = date_range('1/1/2000', periods=4, freq='5min')
+
+        for arg in args:
+            if arg == 'ohlc':
+                func = _ohlc
+            else:
+                func = arg
+            try:
+                result = s.resample('5min', how=arg,
+                                    closed='right', label='right')
+
+                expected = s.groupby(grouplist).agg(func)
+                self.assertEqual(result.index.name, 'index')
+                if arg == 'ohlc':
+                    expected = DataFrame(expected.values.tolist())
+                    expected.columns = ['open', 'high', 'low', 'close']
+                    expected.index = Index(inds, name='index')
+                    assert_frame_equal(result, expected)
+                else:
+                    expected.index = inds
+                    assert_series_equal(result, expected)
+            except BaseException as exc:
+
+                exc.args += ('how=%s' % arg,)
+                raise
+
     def test_resample_basic_from_daily(self):
         # from daily
         dti = DatetimeIndex(