ENH: Implement "standard error of the mean"
State: closed

closes #6897

As discussed in issue #6897, here is a pull request implemented "standard error of the mean" (sem) calculations in pandas.  This is an extremely common but very simple statistical test.  It is used pretty universally in scientific data analysis.

sem is implemented in nanops, generic, and groupby.  Unit tests are included and the docs have been updated.

I have also included an improved .gitignore, based on matplotlib's but including everything from the old .gitignore (hopefully better organized now, though).


Labels:
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@toddrjen will take a look at this for 0.14.1

I may cherry-pick your `.gitignore` now though...thanks

picked this: https://github.com/pydata/pandas/commit/592a5379dea49f6e88e736a651bf82a00ea4fcc8

thanks!

@cpcloud I have implemented your suggestsions, I think:
1. in groupby, there is now a separate _count_sqrt method, while the sem method just calls the std and _count_sqrt methods.  This leaves the decision on whether to optimize up to the std and _count_sqrt methods (which have different rules).
2. I have also implemented a general test in test_groupby that tests _cython_agg_general for all optimized operations and compares them to an expected result.  It just uses a simple numeric test dataframe, it doesn't test non-numeric data or data with nan values, but to make sure basic operations are working it should be sufficient.

I have implemented all the requested changes:

`nanvar` and `nansem` now both use a common private function to get the element count.  I have also implemented unit tests for all the `nan*` functions in `nanops`, including `nanvar` and `nansem`, in a new `test_nanops.py` unit test file.

I have removed `count_sqrt`.  `grouper.sem` is now implemented using `grouper.std` and `grouper.count` directly.  Also, as suggested, I removed `std` from `cython_agg` and implemented `grouper.std` as `sqrt(grouper.var)`.  This also allowed a little simplification of `cython_agg`.  It is private API so I don't know if the change needs to be documented.

I have also updated the docs to refer to 0.14.1 rather than 0.14.0.

All unit tests are passing now.

The new unittests not related to `sem` are in their own commits (f5bbd07 and 5771791).  Since they are just additional unittests and don't change any behavior, they could be cherry-picked for 0.14.0.  The `grouper.std` changes are also in their own commit (1b7e239), but that change is probably too invasive for 0.14.0.

for a quick skim looks ok. this will be considered for 0.14.1. nothing except critical bugs will be included in 0.14.0 at this point.

ok, you can put release notes in v0.14.1 whatsnew. when you are ready for review, pls ping.

Okay, it seems to be good now.  Please take a look.

I fixed up the nanops unit tests compared to what I had before.  This identified a bunch of small bugs, which I fixed, as well as places where support for pandas objects in nanops could be fairly easily improved.  Support for pandas objects is not 100%, but it is much, much better than it was.

create a new issue for  all of the 'small bugs'. This should be done on top of this PR (or completely independently as possible). Mark and show all cases in the PR, and annotate the code and tests as much as possible.

It really is better to have small independent PR's rather than large multi-issue PR's. When things are interrelated its ok, but the bigger they get the harder to review, comment and fix. 

Going to need a perf test on this. As well as a squash to a small number of commits, ideally have separate issues in separate commits (this is equivalent to PR's on top of each other).

@toddrjen this PR is doing way too much. you need to break this down into much simpler parts. remove all things associated with object dtype conversions, and Panel and higher dim implementations. They can be done separately and on top of each other. You are introducing way too much logic in nanops that is ONLY for ndarray/scalar types. checking for `apply` is simply nonsense here. This makes it impossible to follow. and their is no separation between functions at all. This is making bugs in the future MUCH MORE LIKELY.

So in order to accept this you need to split it into parts:
- 1) sem implm
- 2) changes for any `inf` handing object dtypes
- 3) changes for ndim > 2 handing. Need justification for why you are doing this.

pls create separate issues for 2 and 3.

I have reduced this to the stuff related to nanops, plus the previously-suggested simplification of groupby's std.

@toddrjen ok that looks good (just need the fix on the release notes ; move to v0.14.1.txt). that and a perf check and i think its ok to go in.

Then can spin off the bug fixes to another PR (they are wanted, but need some explict tests for them that validate)

I have already split the fixes into another branch and removed the pandas-related stuff from nanops there, but I will worry about that after this is merged.  The documentation changes you requested have also been implemented.

just need perf check to make sure nothing changed, see here: https://github.com/pydata/pandas/wiki/Performance-Testing

What, specifically, do you need in terms of results from the performance
test?  I get a pretty big table, should I post that here?  Or do you just
want to know if there is any change outside of a specific range? If so,
what range?

post anything > ratio of say 1.2 (post the table format, but just those entries); if something is < 0.8 you can post as well (meaning it sped up)

Here are the results with values outside the range you specified:

| Test name | head `ms` | base `ms` | ratio |
| --- | --- | --- | --- |
| panel_shift_minor | 0.1006 | 0.4030 | 0.2497 |
| dataframe_reindex | 1.2967 | 1.6870 | 0.7686 |
| stat_ops_series_std | 0.7694 | 0.3263 | 2.3578 |

ok, can you repeat the `stat_ops_series_std` a couple of times to see if it changes (time is < 1ms), so sometimes suspect. and if not, see if you can figure out the reason for the increase?

I ran it again with 20 reps and 10 burnin and now it is at .9868, so the previous result seems to just be noise.

I just rebased on the most recent master and unit tests are passing, so I think this is ready to merge.

Looks ok, pls squash down to a smaller number of commits and i;ll merge; ping when ready

I have squashed the commits.

thanks!