skipna added to groupby numeric ops
State: closed

 - [x] closes #15675 
 - [x] tests added / passed
 - [ ] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Added the skipna in cython routines. Tried to support the skipna for `numeric_only=False` scenario specially  for datetimes but it breaks other things most of the time. 
Pushed a bare minimum code. Feeding such a dataframe fails.

```
import pandas as pd
import numpy as np
 df = pd.DataFrame(
            {'group': [1, 1, 2, 2],
             'int': [1, np.nan, 3, 1],
             'float': [4., 5., 6., 4],
             'category_int': [7, 8, np.nan, 3],
             'datetime': [pd.Timestamp('2013-01-01 05:00:00'),
                          pd.NaT,
                          pd.Timestamp('2013-01-03 00:00:00.555444'),
                          pd.Timestamp('2013-01-04 12:00:00.458795')],
             'datetimetz': [
                 pd.Timestamp('2013-01-01 12:00:00', tz='US/Eastern'),
                 np.nan,
                 pd.Timestamp('2013-01-01 12:00:00', tz='US/Eastern'),
                 pd.Timestamp('2013-01-03 00:00:00', tz='US/Eastern')],
             'timedelta': [pd.Timedelta('1.0s'),
                           pd.Timedelta('3s'),
                           pd.Timedelta('1s'),
                           np.nan]},
            columns=['group', 'int', 'float', 'category_int', 'datetime',
                     'datetimetz','timedelta'])
result = df.groupby('group')
result = result.sum(numeric_only=False, skipna=False)
print(result)
```
It fails for timedelta. However notice the bypass I had put on Implementation Error in `_cython_agg_blocks` else it was failing for DatetimeTZ also.
hmm, might be related to this bug: https://github.com/pandas-dev/pandas/issues/15884#issuecomment-291301611

(this falls thru the cython_agg, where it *should* work, but doesn't becauase of a bug), then the ``.apply`` is *also* buggy.
@mayukh18 if you have it working for most cases, then split the input frame up in 2 tests, working and non-working (and you can even xfail the non-working ones).
@jreback I didn't understand your last comment fully. Are you suggesting on pushing the code as it is like this and just manage with 2 separate tests?
so separate out everything that is not working into a separate test

then we can iiterate on fixing it

ultimately we want 1 test but sometimes it's helpful to make the basic case work w/o the distraction of other cases

another (maybe better) are to parameteize the cases so hey r treated as separate test (again might be easier to debug that way)
@jreback So the situation now is like everything is done. In ideal case, i.e. if that bug isn't there, skipna will work on the datetimes too. But now, due to the bug, the ops all break down when the frame has datetimes. Now this (new code + bug) is leading a small few other tests to fail which can't be passed without correcting the bug. The test for skipna however runs ok without the datetimes.

So should I push the code after adding another skipna test having datetimes? what do you suggest?
Sorry if I misled you but I hadn't actually pushed the code. This was the older push. I'll go though these and see if I have covered all these.
can you rebase and update?
I had completed it more or less quite sometime back but my cython development environment broke down. I work on Windows and just can't fix it even after trying everything. Strange but I just don't know what to do in this situation.
@mayukh18 have you seen: http://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-windows-development-environment

using 3.6 on windows is quite easy, simply install 2015 (free download). create a new conda env and it should just work.
The problem I am facing is with setting up C compilers while building cython. Will a new conda env set those up on it's own?
you simply install VS 2015
Does it have to be 2015? Because I installed 2017 and it didn't solve it. Thanks for the help.
2017 in theory should work, but simply use 2015 (you can install many of these).
note 2015 means you can build using 3.5 & 3.6. TO do something else is much much harder (e.g. 2.7 is tricky).
I guess that may be the problem. I used a virtualenv of 2.7. That time I had a installation of vs 2008 I guess. Then it broke somehow and things went apart. I'll give 2015 and 3.6 a fresh try.
yeah 2.7 is *hard*, just use 3.5 or 3.6 with VS2015. should be a breeze (that in fact is the point of the changes in VS), it *is* backward compatible for building (but that starts at 3.5)
Thanks a lot for the help.
can you rebase / update
closing as stale, if you'd like to continue, please ping.
Hi @jreback , is this something that will still be useful? I'll take this up again then.
@mayukh18 as I just ran into this very issue, it would be much appreciated if you could do so.
Cool. Started on this.
@mayukh18 sorry for the very late reply, but do you have an updated version of this branch / started working on it again? 
It's certainly still something that would be very useful
@jorisvandenbossche have finally got to it. Opened up a new PR #41399 since I could not revive the branch of this PR.