ENH: Allow aggregate numeric operations on timedelta64.
State: closed

closes #5724

Currently, `timedelta64` columns can't be used in aggregate group operations. They are quietly dropped or, if no other columns are present, an exception is raised:

```
In [2]: df
Out[2]: 
     A        E
0  foo 00:00:01
1  bar 00:00:02
2  foo 00:00:03
3  bar 00:00:04


In [3]: df.groupby('A').mean()
Out[3]: 
(...)
DataError: No numeric types to aggregate
```

But any operation that works on numerical data is also well defined on timedeltas. This PR relaxes the restriction on those methods designated "numeric only" to accept numeric dtypes _and_ `timedelta64`. I include a unit test with one simple case each for `DataFrameGroupBy` and `SeriesGroupBy`.

```
In [2]: df.groupby('A').mean()
Out[2]: 

           E
A           
bar 00:00:03
foo 00:00:02
```


Labels:
name: Groupby
name: API Design
name: Timedelta
description: Timedelta data type

Comments:
FYI their is an issue for adding timedelta support to groupby - see if u can find it

also this needs a fair amount of tests as trit are many paths thru groupby

basically need to mimic most of the datetime64 type tests

Have we established some pattern for skipping timedelta tests for numpy < 1.7?

Also, while I'm here, I'll fix the English in this error....

   ValueError: to_timedelta is not support for numpy < 1.7

numpy timedelta is thoroughly broken < 1.7

See for example: https://github.com/pydata/pandas/blob/master/pandas/tseries/tests/test_timedeltas.py#L21

@danielballan hows' this coming?

A quick, perhaps obvious, thought: this is not exactly analogous to what was done for datetime. For example, summing datetimes is (correctly) not defined but summing timedeltas should be.

But many of the tests will be analogous. I have to put this aside until next week, but I can pick it up by the beginning of May. If someone wants to tap in before then, feel free to speak up.

gr8 thanks

would like like to get some minimal set at least in for 0.14

as I agree it's a bit nontrivial 
so test away!

@danielballan can you rebase, add a couple more tests, and release note / v0.14.0.txt ?

I am working on finishing this patch, and I'm not convinced this is the right approach. What about making timedeltas considered a numeric type? Otherwise we're going to have to update all locations where there are is_numeric checks.

I suppose this opens the question about what "is_numeric" means.

In particular, the current Pandas master does

```
def _cython_agg_blocks(self, how, numeric_only=True):
    ...
    if numeric_only:
        data = data.get_numeric_data(copy=False)
```

I just started using Pandas today, so it's possible the original patch is the right approach.

@jreback

I considered lumping time deltas in with is_numeric, but I decided it was
clearer, more readable this way. Still up for debate though.

On Tuesday, April 29, 2014, Keir Mierle notifications@github.com wrote:

> @jreback https://github.com/jreback
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/6884#issuecomment-41642641
> .

Rebased.

I can add tests this week, but I also welcome any contributions that @keir has to offer. FYI, @keir, as I revisit this, I find one place where we check `is_numeric` in a way that excludes timedeltas: above the call to `ensure_float`. This is why I made them separate.

@danielballan, I am operating off trunk, which seems to have changed one of the functions (_cython_agg_blocks). I ported the code tried my approach to make timedelta is_numeric, but I am now not convinced of the approach. Do you want to update to master?

OK. I updated to master this morning. See the latest version of the commit. (N.B. The tests are broken; this is a work-in-progress commit.)

@danielballan how's this coming?

@jreback If I can push through this this weekend, still time for 0.14? Actually maybe better to wait for 0.14.1 so we can get #5986 also.

well I want to get a RC1 out, but if you are ready after, then ok

FYI, I think #5986 for quantile has been fixed by @TomAugspurger very recently

I don't know if I tested timedeltas at all in `quantile`, just timestamps. 

I don't suggest merging this patch. I have a new, smaller, different patch that changes timedelta64 to be a numeric type. I will try to upload it later today but currently I have some production issues.

@keir Does that approach create problems when `com.ensure_float` is called on timedelta64 data? I am eager to see it and build out tests. Maybe we are too late for 0.14, which is too bad, but I am eager to have it available on master regardless.

@danielballan 0.14.0 released (ann coming later today), but this could be on master/0.14.1 in any event

@danielballan can you rebase this?

Rebased.

gr8, need a release note as well (v0.14.1.txt is the new place), under API changes, I think a small example would be nice too. also do you have tests for multiple dtypes when aggregating? tests for resampling?

Sure. I do have such a test, related to #5922, but it is failing and stumping me for now. If @keir does not sound off again I will stick with my approach and get these tests in order this weekend.

is that the correct reference issue? it goes to an odd one (the 5922)

@jreback I should have said #5592. See https://github.com/danielballan/pandas/blob/e3a2928caa483fa7b6951ee8f3e74d88a036aa82/pandas/tests/test_groupby.py#L458

ok gr8...that was a nasty test

@danielballan ?