Refactored Resample API breaking change
State: closed

~~on top of #11603~~

closes #11732
closes #12072
closes #9052 
closes #12140 

ToDo:
- [x] rewrite/expand main docs
- [x] add aggregate section

New API

```
In [4]: np.random.seed(1234)

In [5]: df = pd.DataFrame(np.random.rand(10,4),
                     columns=list('ABCD'),
                     index=pd.date_range('2010-01-01 09:00:00', periods=10, freq='s'))

In [6]: df
Out[6]: 
                            A         B         C         D
2010-01-01 09:00:00  0.191519  0.622109  0.437728  0.785359
2010-01-01 09:00:01  0.779976  0.272593  0.276464  0.801872
2010-01-01 09:00:02  0.958139  0.875933  0.357817  0.500995
2010-01-01 09:00:03  0.683463  0.712702  0.370251  0.561196
2010-01-01 09:00:04  0.503083  0.013768  0.772827  0.882641
2010-01-01 09:00:05  0.364886  0.615396  0.075381  0.368824
2010-01-01 09:00:06  0.933140  0.651378  0.397203  0.788730
2010-01-01 09:00:07  0.316836  0.568099  0.869127  0.436173
2010-01-01 09:00:08  0.802148  0.143767  0.704261  0.704581
2010-01-01 09:00:09  0.218792  0.924868  0.442141  0.909316

In [7]: df.resample('2s')
Out[7]: DatetimeIndexResampler [freq=<2 * Seconds>,axis=0,closed=left,label=left,convention=start,base=0]

In [8]: r = df.resample('2s')

In [9]: r.sum()
Out[9]: 
                            A         B         C         D
2010-01-01 09:00:00  0.971495  0.894701  0.714192  1.587231
2010-01-01 09:00:02  1.641602  1.588635  0.728068  1.062191
2010-01-01 09:00:04  0.867969  0.629165  0.848208  1.251465
2010-01-01 09:00:06  1.249976  1.219477  1.266330  1.224904
2010-01-01 09:00:08  1.020940  1.068634  1.146402  1.613897

In [10]: r[['A','B']].agg(['mean','sum'])
Out[10]: 
                            A                   B          
                         mean       sum      mean       sum
2010-01-01 09:00:00  0.485748  0.971495  0.447351  0.894701
2010-01-01 09:00:02  0.820801  1.641602  0.794317  1.588635
2010-01-01 09:00:04  0.433985  0.867969  0.314582  0.629165
2010-01-01 09:00:06  0.624988  1.249976  0.609738  1.219477
2010-01-01 09:00:08  0.510470  1.020940  0.534317  1.068634
```

Upsampling

```
In [11]: s = Series(np.arange(5,dtype='int64'),
   ....:               index=date_range('2010-01-01', periods=5, freq='Q'))

In [12]: s
Out[12]: 
2010-03-31    0
2010-06-30    1
2010-09-30    2
2010-12-31    3
2011-03-31    4
Freq: Q-DEC, dtype: int64

In [13]: s.resample('M').ffill()
Out[13]: 
2010-03-31    0
2010-04-30    0
2010-05-31    0
2010-06-30    1
2010-07-31    1
2010-08-31    1
2010-09-30    2
2010-10-31    2
2010-11-30    2
2010-12-31    3
2011-01-31    3
2011-02-28    3
2011-03-31    4
Freq: M, dtype: int64

In [14]: s.resample('M').asfreq()
Out[14]: 
2010-03-31     0
2010-04-30   NaN
2010-05-31   NaN
2010-06-30     1
2010-07-31   NaN
2010-08-31   NaN
2010-09-30     2
2010-10-31   NaN
2010-11-30   NaN
2010-12-31     3
2011-01-31   NaN
2011-02-28   NaN
2011-03-31     4
Freq: M, dtype: float64
```


Labels:
name: API Design
name: Resample
description: resample method

Comments:
Although I really like the fact of a more consistent API, I think this is so backwards incompatible that it really is problematic to just put it in a release like this.

I also kind of like the simplicity of a basic resample to explain in tutorials (although once it is not a basic resample anymore, this interface is nicer). A two-step method is more complicated than a 1-step..

Regarding to the back compat issue, there are maybe other ways to solve this? Eg, I think of using another name, some keyword indicating the behaviour (but this will get ugly), .... 

> Although I really like the fact of a more consistent API, I think this is so backwards incompatible that it really is problematic to just put it in a release like this.

pls explain. The point of breaking compat is to change future behavior. breaking changes always have short term pain, but shying away from actual long time needed inconsistencies is MUCH MUCH worse.

yes we could add a `.resample2` method which is the new impl, and preserve the original with a deprecation warning. Then of course we'd have to maintain `.resample2` for a bit.

One thing we could do to help preserve backwards compat is to keep around `how` for now as a deprecated argument. If `how` is set, raise a deprecation warning and preserve the original behavior. This would at least keep cases like `s.resample('24H', how='max')` working, though it would indeed break `s.resample('24H')`.

I already do the deprecation warning (and just return the result) if `how` is actually specified.

The only case this is actually breaking is if NO `how` is specified at all (which is prob very common)

```
In [1]: s = Series(range(5))

In [2]: s = Series(range(5),index=date_range('20130101',periods=5,freq='s'))

In [3]: s.resample('D',how='min')
/Users/jreback/miniconda/bin/ipython:1: FutureWarning: how is .resample()
the new syntax is .resample(...).min()
  #!/bin/bash /Users/jreback/miniconda/bin/python.app
Out[3]: 
2013-01-01    0
Freq: D, dtype: int64

In [4]: s.resample('D')
Out[4]: DatetimeIndexResampler [freq-><Day>,axis->0,closed->left,label->left,convention->start,base->0]
```

This might be too magical and/or tricky to pull off, but we might add fallback methods to `Resampler` corresponding to the `Series`/`DataFrame` API that issue a deprecation warning, and then call `.mean()` followed by the desired operation.

hacking  `Resampler.__getattr__`  gets almost all the way there, see [here](https://github.com/pydata/pandas/commit/86b47f2c4cd2a6f95900d47ffaea517b4d6feff9)

```
In [1]: s = Series(range(5),index=date_range('20130101',periods=5,freq='s'))

In [2]: s.resample('D').ix[0]
pandas/tseries/resample.py:73: FutureWarning: .resample() is now a deferred operation
use .resample(...).mean() instead of .resample(...)
  return getattr(self._deprecate_api(),attr)
Out[2]: 2

In [2]: s.resample('D').sort_values()
pandas/tseries/resample.py:73: FutureWarning: .resample() is now a deferred operation
use .resample(...).mean() instead of .resample(...)
  return getattr(self._deprecate_api(),attr)
Out[2]: 
2013-01-01    2
dtype: int64
```

@jreback nice. One thing we'll have to catch explicitly is `__setitem__` and other assignment or in-place operations. We need to catch those operations so they don't silently fail.

so defined all of the arithmetic ops / comparison ops and instance checking
seems that we _can_ masquerade as a Series (or DataFrame) pretty much, or at least
to work and show the deprecation warning.

```
In [1]: s = Series(range(5),index=date_range('20130101',periods=5,freq='s'))

In [2]: r = s.resample('H')

In [3]: r>2
pandas/tseries/resample.py:80: FutureWarning: .resample() is now a deferred operation
use .resample(...).mean() instead of .resample(...)
  result = self._deprecated()
Out[3]: 
2013-01-01    False
Freq: H, dtype: bool

In [3]: r*2
pandas/tseries/resample.py:80: FutureWarning: .resample() is now a deferred operation
use .resample(...).mean() instead of .resample(...)
  result = self._deprecated()
Out[3]: 
2013-01-01    4
Freq: H, dtype: int64
```

so its actually quite a tricky problem to catch this. imagine you had this code originally

```
r = series.resample('H')
r.iloc[0] = 5
```

The warning will show up, but the setting is actually on a copy. I prob should just raise.

> The warning will show up, but the setting is actually on a copy. I prob should just raise.

Indeed, this needs to raise. I would say this raising is more important than adding the Series/DataFrame facade. If we don't have a facade, things will fail loudly, which is unfortunate, but if we have a broken facade, things will fail silently!

@jorisvandenbossche how are we removing deprecation warnings on older whatsnew files?
I can just change the code I guess

@jreback See https://github.com/pydata/pandas/issues/6856#issuecomment-41648827 for general discussion on this. 
I am personally in favor of converting the older whatsnew files to static code-blocks (and acknowledging that it are 'historic' pages that do not necessarily reflect the current state of the package).

For now, adding `:okwarning:` to the ipython blocks will suppress the warnings during doc building.

> > Although I really like the fact of a more consistent API, I think this is so backwards incompatible that it really is problematic to just put it in a release like this.
> 
> pls explain. The point of breaking compat is to change future behavior. breaking changes always have short term pain, but shying away from actual long time needed inconsistencies is MUCH MUCH worse.

To answer this: of course we have to keep a right balance between avoiding breaking changes and improving pandas. But that is not always very clear what the right balance is, and I could also imagine other parts of pandas that could use some breaking changes, but that we are reluctant about to do because of its impact (not to mention the indexing api ... :-)) 
And most API changes we do are not breaking changes (but deprecations), and are most of the time more corner cases (but OK, I work with time series and use resample a lot, so maybe I am a bit biased)

But, as you now updated to include a deprecation warning or fail with an informative error, it sounds already a lot better to me! (at least there will be no silent failure, that's probably the most important)

Maybe I will ping the mailing list about this, as it is a rather large change?

@jorisvandenbossche you e-mail to the list is good. thanks.

I agree about most changes, but some things just need to be more consistent, this and the window functions are in the groupby pattern (which is quite in-grained in pandas). So to me this is a no-brainer, EVEN if we had to break back-compat (in this case was able to provide a nice easy upgrade path so that makes this pretty easy to swallow IMHO).

if you want to review, this is also ready to go.

any more comments?

@shoyer @jorisvandenbossche @wesm 

final comments

@jreback will try to review this tomorrow

From the doc build log on travis, there seem to be some occurences of resample in cookbook.rst and timedeltas.rst that have to be adapted

@jorisvandenbossche cookbook / timedelta.rst fixed (next push)

I'll take a look. I approve of the more `groupby`-like API. Users would find deprecation warnings very annoying so I'm not sure what's the right call there. 

First round usage feedback:

I am using this series:

```
rng = pd.date_range('1/1/2012', periods=100, freq='S')
ts = pd.Series(np.arange(len(rng)), index=rng)
```
- If you are downsampling, but call `upsample`, I think you get the original values at those timestamps? (so actually some kind of regular sampling)
  
  ```
  In [27]: ts.resample('20s').upsample()
  Out[27]:
  2012-01-01 00:00:00     0
  2012-01-01 00:00:20    20
  2012-01-01 00:00:40    40
  2012-01-01 00:01:00    60
  2012-01-01 00:01:20    80
  Freq: 20S, dtype: int32
  ```
  
  Is that the intended behaviour? (If so, maybe should be more explicit in the docs) Or should we disallow upsample if we are downsampling? (not needed I think)
- The fillling methods (ffill, pad, bfill, backfill):
  - Should there also be a generic `fillna` where these methods can be specified as a keyword argument? (which would actually be like `ts.resample(..).upsample().fillna(..)`, but a bit shorter)
  - I am not really a fan of the duplicate method names .. (I know it is like this for series, but this is new API)
- repr of the DatetimeIndexResampler: same comment as for the Rolling objects, and I think you changed it there? So would make it consistent.
- DatetimeIndexResampler has no docstring
- `transform` does not work with resample (you get a `AbstractMethodError`)
- More complex aggregations:
  - Doing `r[['A', 'B']].agg({'result1' : np.sum,'result2' : np.mean})` gives a blank AssertionError, which is not very helpful
  - Doing `r.agg({'A' : ['sum','std'], 'B' : ['mean','std'] })` works as expected, but `r[['A', 'B']].agg({'A' : ['sum','std'], 'B' : ['mean','std'] })` gives _Exception: Must produce aggregated value_ ?
- `apply` also gives _Exception: Must produce aggregated value_ if you pass it a function that does not return a aggregated value. For groupby this works, would be nice to have this consistent.

Having the API breakage on `ts.resample(freq)` is regrettable, but on the bright side it would produce an immediate error in a lot of cases. It's likely to make some people pretty grumpy -- maybe in the interest of progress we could have a `resample_old` method that emulates the original behavior (with `how='mean'` as the default)? 

Looked through the rest of the code and things look fine to me. 

If you are downsampling, but call upsample, I think you get the original values at those timestamps? (so actually some kind of regular sampling)

@jorisvandenbossche 

you put this example up

```
In [27]: ts.resample('20s').upsample()
Out[27]:
2012-01-01 00:00:00     0
2012-01-01 00:00:20    20
2012-01-01 00:00:40    40
2012-01-01 00:01:00    60
2012-01-01 00:01:20    80
Freq: 20S, dtype: int32
```

> Is that the intended behaviour? (If so, maybe should be more explicit in the docs) Or should we disallow upsample if we are downsampling? (not needed I think)

This is exactly right. This is the 'purpose' of `.upsample()` its basically a `.reindex()`. In fact maybe I should call call it `.reindex()`? you are picking out values at the new freq.

could also call it `.asfreq()`

I have finally cleaned up the aggregation API, preserving everything except for a single case  of a nested dict that has renaming (see the whatnew, where I raise a `SpecificationError` ). This was a bear to get it 2 work across groupby/resample/window functions. They use mostly the same code, though there are a number of cases. (this will close #9052) now

If we were to drop the renaming logic this would be a lot simpler, so maybe should consider that.

e.g. 
`grouped[['A', 'B']].agg({'result1' : 'mean', 'result2' : 'sum'})`

 which is convenient, but then gets complicated once you allow 

`grouped[['A', 'B']].agg({'A' : {'result1' : 'mean'}, 'B' : {'result2' : 'sum'}})`

Certainly someone can take a crack at simplifying this code as I added all of the tests cases.

It is unfortunate that this is now tied up in the resample change, but that's how it worked out.

also this PEP's `tseries/resample.py` and `core/groupby.py`, the remaining ones.

cc @jorisvandenbossche @shoyer @wesm @TomAugspurger 

Looking good to me here. I'll leave it to @jorisvandenbossche to give the thumbs up to merge

@jreback I will do a last review tomorrow!

@jorisvandenbossche thanks! hopefully will be the last :)

I experimented a bit with the branch, some smaller comments:
- the repr: can we follow here PEP8 as well? :-) (I mean spaces after the comma's, I think this would make it a bit more readable, maybe also quotes around the strings)
- `upsample` has been removed? (but is still in the documentation) Or is this now `asfreq`?
- `apply` gives `Exception: Must produce aggregated value` (for a series) if you pass it a function that does not return a aggregated value. For groupby this works, would be nice to have this consistent. 
  When applying it on a resampled dataframe, you get the cryptic `ValueError: cannot copy sequence with size 3 to array axis with dimension 180` message. Typical example is just a `.apply(lambda x: x)`

Further, there is still a lot of documentation missing. Some examples:
- `Resampler.fillna` has no explanation
- `DatetimeIndexResampler` has no docstring yet
- `apply`, `agg`, `aggregate` have no docstring, `transform` a very brief

In theory it would be best if this were addressed in this PR, but given the PEP8 changes waiting on this PR, it's OK for me to leave this for a follow-up PR

@jorisvandenbossche 

thanks, let me see what I can address for your comments and I'll make a follow up issue.

`.upsample()` -> `.asfreq()` (I may not have pushed that doc part of the change)