API: provide Rolling/Expanding/EWM objects for deferred rolling type calculations #10702
State: closed

closes #10702 
closes #9052
xref #4950, removal of depr

So this basically takes all of the `pd.rolling_*`,`pd.expanding_*`,`pd.ewma_*` routines and allows an object oriented interface, similar to groupby.

Some benefits:
- nice tab completions on the Rolling/Expanding/EWM objects
- much cleaner code internally
- complete back compat, e.g. everything just works like it did
- added a `.agg/aggregate` function, similar to groupby, where you can do multiple aggregations at once
- added `__getitem__` accessing, e.g. `df.rolling(....)['A','B'].sum()` for a nicer API
- allows for much of #8659  to be done very easily
- fix for coercing Timedeltas properly
- handling nuiscance (string) columns

Other:
- along with window doc rewrite, fixed doc-strings for groupby/window to provide back-refs

ToDO:
- [x] I think that all of the doc-strings are correct, but need check
- [x] implement `.agg`
- [x] update API.rst, what's new
- [x] deprecate the `pd.expanding_*`,`pd.rolling_*`,`pd.ewma_*` interface as this is polluting the top-level namespace quite a bit
- [x] change the docs to use the new API

```
In [4]: df = DataFrame({'A' : range(5), 'B' : pd.timedelta_range('1 day',periods=5), 'C' : 'foo'})

In [5]: df.rolling(window=2).sum()
Out[5]: 
    A      B    C
0 NaN    NaT  foo
1   1 3 days  foo
2   3 5 days  foo
3   5 7 days  foo
4   7 9 days  foo

In [6]: df.rolling(window=2)['A','C'].sum()
Out[6]: 
    A    C
0 NaN  foo
1   1  foo
2   3  foo
3   5  foo
4   7  foo

In [2]: r = df.rolling(window=3)

In [3]: r.
r.A         r.C         r.corr      r.cov       r.max       r.median    r.name      r.skew      r.sum       
r.B         r.apply     r.count     r.kurt      r.mean      r.min       r.quantile  r.std       r.var       
```

do rolling/expanding/ewma ops

```
In [1]: s = Series(range(5))

In [2]: r = s.rolling(2)

I# pd.rolling_sum
In [3]: r.sum()
Out[3]: 
0   NaN
1     1
2     3
3     5
4     7
dtype: float64

# nicer repr
In [4]: r
Out[4]: Rolling [window->2,center->False,axis->0]

In [5]: e = s.expanding(min_periods=2)

# pd.expanding_sum
In [6]: e.sum()
Out[6]: 
0   NaN
1     1
2     3
3     6
4    10
dtype: float64

In [7]: em = s.ewm(com=10)

# pd.ewma
In [8]: em.mean()
Out[8]: 
0    0.000000
1    0.523810
2    1.063444
3    1.618832
4    2.189874
dtype: float64
```

and allow the various aggregation type of ops (similar to groupby)

```
In [1]: df = DataFrame({'A' : range(5), 'B' : pd.timedelta_range('1 day',periods=5), 'C' : 'foo'})

In [2]: r = df.rolling(2,min_periods=1)

In [3]: r.agg([np.sum,np.mean])
Out[3]: 
    A           B                    C     
  sum mean    sum            mean  sum mean
0   0  0.0 1 days 1 days 00:00:00  foo  foo
1   1  0.5 3 days 1 days 12:00:00  foo  foo
2   3  1.5 5 days 2 days 12:00:00  foo  foo
3   5  2.5 7 days 3 days 12:00:00  foo  foo
4   7  3.5 9 days 4 days 12:00:00  foo  foo

In [4]: r.agg({'A' : 'sum', 'B' : 'mean'})
Out[4]: 
   A               B
0  0 1 days 00:00:00
1  1 1 days 12:00:00
2  3 2 days 12:00:00
3  5 3 days 12:00:00
4  7 4 days 12:00:00
```


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Cc @jhamman who has been working on this for xray.

pushing to 0.18.0, I think `__getitem__` will be a really nice add here. might as well do all of this at once.

ok, this is ready, MUCH bigger rabbit hole that I thought.

note on the doc-strings. 

Since now we have much more like a groupby interface, e.g. 

`s.rolling(....).sum()`, the doc-strings for `Rolling.sum` are minimal but have a See also back to the Series/DataFrame.rolling (We don't have the notion of a `RollingSeries`,`RolldingDataFrame` class so this would be quite tricky).

Further I did the same with groupby doc-strings (again don't have the class distinction on the See Also).

@jorisvandenbossche @shoyer @sinhrks @TomAugspurger @cpcloud 

One thing I've wanted to add but haven't, which may be easier using this framework, at least interface-wise, is rolling exponentially weighted functions -- i.e. add a window to all the ewm*() parameters. Obviously this would need to be implemented in Cython for performance, but perhaps interface-wise it would be simpler using the scheme proposed here.

yep that would be quite straightforward to do interface wise but yes would need to be added to the cython functions (but not too hard there)

any comments?

@shoyer @jorisvandenbossche 

ok so you are both in favor of completely deprecating both `freq/how`.

an alternative is add them a keywords like we have 'how' now, e.g. in the actual functions themselves, rather than as an attribute of a `Rolling` type of object.

I think they are there to prevent common errors. E.g. say you have 1-min data and want daily rolling stats. Then in order to do this we would have:

`s.resample('D',how='max').rolling(window=3).max()`

rather than (not that the `how` is currently defaulted, but shown for clarity
`s.rolling(window=3).max(freq='D',how='max')`
or
`s.rolling(window=3,freq='D').max(how='max')`

Further now that I look at it, the window above is a function of the data (by definition), and IIRC, this is a source of confusion. I wonder if `window='3D'` would make this more clear.

e.g.
`s.rolling(window='3D').max()`

> Further now that I look at it, the window above is a function of the data (by definition), and IIRC, this is a source of confusion. I wonder if window='3D' would make this more clear.

I like the idea of defining rolling window size in terms of index frequency, but I think this particular API would be confusing. In particular, it starts to make a distinction between `1H` and `60Min` which resample does not -- the magnitude and unit for the frequency are now independently meaningful. It will be surprising when these start to give different results.

ok that was just a wild thought

so what about making freq/how optional args to each function then ? 

or still think we ought to deprecate ?

I'm still in favor of deprecating freq/how. If we can't come up with an unambiguous place to put the arguments that's a good sign that they aren't a good fit.

@shoyer @jorisvandenbossche ok, deprecated `freq`, `how`.

On another note, might this be a good time to fix `.resample`, so

`s.resample('D',how='max')` -> `s.resample('D').max()`.

Note that this WOULD be a breaking change, meaning the `s.resample('D')` would raise an exception rather than 'just work', because the `how` arg is defaulted to `'mean'`. So it wouldn't go unnoticed.

thoughts?

@jreback I don't think a deprecation is needed? It is fully new API, so I wouldn't add things we will remove shortly. 
It just means that not all `pd.rolling_mean(..)` translates exactly to a `df.rolling(..).mean(..)`, but I don't think that is a problem.

In some way I also like that `resample` would look more like `groupby` and now `rolling`, but this is indeed more controversial (more breaking). But maybe let's leave that discussion for a separate issue? (it is already complicated enough here :-))

@jorisvandenbossche I left in the deprecations in `freq`/`how` so that existing code would run and just throw deprecation warnings, rather than breaking things straightaway.

Implementation wise: the old `pd.rolling_..` functions are using the new implementation under the hood?

Hmm, which is indeed the case I think. In that case it is indeed more difficult to leave out the freq/how keywords .. It is not easier to just leave the old implementation as is?

@jorisvandenbossche the old API is wrapping the new API. This cleans up a massive amount of code, which was a secondary goal here. 

I _can_ do this to allow the existing `pd.rolling_*` to accept `freq`/`how` but not the new ones I think. 
But not really sure this is that big of a deal. In 0.19.0 we just drop these new deprecations (along with `pd.rolling_*`

Just catching up on this thread.   I have extensive code where the
optimization is to pass a naked np.ndarray to pd.rolling_median.

It is a looped  median smoothing across 42M separate time series.By keeping
it as ndarrays I can use numba acceleration on some of the other code , e.g
 the stopping criteria.

An early post says
- complete back compat, e.g. everything just works like it did

But is that true in this case.

On Tue, Dec 1, 2015 at 6:22 AM, Jeff Reback notifications@github.com
wrote:

> @jorisvandenbossche https://github.com/jorisvandenbossche the old API
> is wrapping the new API. This cleans up a massive amount of code, which was
> a secondary goal here.
> 
> I _can_ do this to allow the existing pd.rolling_\* to accept freq/how but
> not the new ones I think.
> But not really sure this is that big of a deal. In 0.19.0 we just drop
> these new deprecations (along with pd.rolling_*
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11603#issuecomment-160952343.

of course - u will get a deprecation warning in any event

it is quite trivial to wrap a Series around it so this shouldn't be a problem anyhow

why would u loop around a rolling_median? 

@jonblunt if you're using pure numpy arrays, you might just want to use [bottleneck](https://github.com/kwgoodman/bottleneck)'s `move_median` instead.

The algorithm is a median smoothing that requires repeated iteration until
stopping condition i reached (mapd of the smoothed n with smoothed n-1)

In addition between each pass the smoothed series is compared to the
original data to identify any outliers.  There are also specific rules
about how to interpolate / fill across Nans.. This basically rules out
bottleneck which does not allow nans in the series.

The fact that each series is different, has different masked days and will
stop after a varying number of passes makes it difficult to  run at a more
aggregate level than the single time series.

In the optimized code running this for 45000 series  each of length 160
takes 7 min It starts with  a single series and multiindex and in two
nested groupbys runs the smoothing algorithm on a single series and then
returns a single dataframe of the same shape and index as the original.

Of the 7 min, that smooth_a_series inner loop takes 4 min of which the
rolling moment time is  70 seconds.

If I go back to using series , and masking values in the series , saving
the results to series and then looking for outlier values in the series and
a mapd calculation across the two series  this adds another 2-3 min to the
run time;  or since this is 0.25 % of the total or about  15hrs cumulative
run time.  By staying in ndarrays and saving values in place I displace
gazillion object creations.

But let me say the skiplist moving median algorithm for moving median is
why we are using pandas for this project.There is a reason why Oracle does
not have a moving median   algorithm either inits database or,last time I
checked, in its native functions in Oracle Enterprise R. naive sorting
algorthms really don't scale for this .

On Tue, Dec 1, 2015 at 5:58 PM, Stephan Hoyer notifications@github.com
wrote:

> @jonblunt https://github.com/jonblunt if you're using pure numpy
> arrays, you might just want to use bottleneck
> https://github.com/kwgoodman/bottleneck's move_median instead.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11603#issuecomment-161136118.

@jonblunt well, not really sure what exactly you are doing, and not really sure where your bottlenecks are. n any event, this is trivial to wrap/unwrap these using the new API; and this is costless.

```
In [5]: arr = np.arange(5)

In [6]: Series(arr).rolling(window=2).median().values
Out[6]: array([ nan,  0.5,  1.5,  2.5,  3.5])
```

BTW you should NEVER use nested groupby, simply use a multi-groupby.

I can see the  overhead of nested groupbys.  Its on my list, but it is a
big refactor..

In any case is there any way to maintain access to this great algorithm
without it being a method of an object. if it is not in the main name
space, could there  a utility module that  exposes the bare algorithm which
is going t

On Tue, Dec 1, 2015 at 9:18 PM, Jeff Reback notifications@github.com
wrote:

> BTW you should NEVER use nested groupby, simply use a multi-groupby.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11603#issuecomment-161167097.

@jonblunt 
you mean something like this?

```
def rolling_median(arr, window):
    return Series(arr).rolling(window=window).median().values
```

That's what I don't want to do, but think I will have to

I just tested that approach with these results

smoothed = rolling_median(array, params)

where both smoothed and  array are numpy arrays.  Actually they are slices
of the same array.

Using profiler, all the calls to rolling moments are made from one
function whose total run time was

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

300171    1.462    0.000  130.972    0.000
smooth_item_at_loc_very_fast.py:174(smooth_once)

 I then replaced each call to ther rolling functions with this wrapper that
just puts the array in a series and  then returns the raw values

def alg_wrap(alg, *_params):
    def _alg(array, *_kargs):
        return alg(pd.Series(array), *_kargs).values
    return partial(_alg, *_params)

The new results were

300171    1.716    0.000  204.076    0.001
smooth_item_at_loc_almost_very_fast.py:174(smooth_once)

 Using the 'numpy array rolling moments took  57s cumulative run time. so
the increase is more than the basic time to run the algorithm.

70s x 400  `= 8hrs cumulative run time across the full workload.

At this point in the code all the arrays are the same length  , one row per
day no missing rows, so I don't need any alignment or reindexing support.
I only put the data back into a pandas DataFrame at the end to return from
the group by at the next step.

I am thinking of some

On Tue, Dec 1, 2015 at 10:51 PM, Ka Wo Chen notifications@github.com
wrote:

> @jonblunt https://github.com/jonblunt
> you mean something like this?
> 
> def rolling_median(arr, window):
>     return Series(arr).rolling(window=window).median().values
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11603#issuecomment-161180237.

Incomplete thought in last post.

 I was thinking of defining  the rolling median algorithm outside the
pandas object and then adding it as a method to the pandas objects.

then provide a decorator to add the method to a lightweight object

class null_series(object):
def wrap_moment( array, params)

def rolling_median(arr, window):
    return Series(arr).rolling(window=window).median().values

On Wed, Dec 2, 2015 at 12:54 PM, Jon Blunt jonblunt99@gmail.com wrote:

> That's what I don't want to do, but think I will have to
> 
> I just tested that approach with these results
> 
> smoothed = rolling_median(array, params)
> 
> where both smoothed and  array are numpy arrays.  Actually they are slices
> of the same array.
> 
> Using profiler, all the calls to rolling moments are made from one function whose total run time was
> 
> ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> 
> 300171    1.462    0.000  130.972    0.000 smooth_item_at_loc_very_fast.py:174(smooth_once)
> 
>  I then replaced each call to ther rolling functions with this wrapper
> that just puts the array in a series and  then returns the raw values
> 
> def alg_wrap(alg, *_params):
>     def _alg(array, *_kargs):
>         return alg(pd.Series(array), *_kargs).values
>     return partial(_alg, *_params)
> 
> The new results were
> 
> 300171    1.716    0.000  204.076    0.001 smooth_item_at_loc_almost_very_fast.py:174(smooth_once)
> 
>  Using the 'numpy array rolling moments took  57s cumulative run time. so
> the increase is more than the basic time to run the algorithm.
> 
> 70s x 400  `= 8hrs cumulative run time across the full workload.
> 
> At this point in the code all the arrays are the same length  , one row
> per day no missing rows, so I don't need any alignment or reindexing
> support.  I only put the data back into a pandas DataFrame at the end to
> return from the group by at the next step.
> 
> I am thinking of some
> 
> On Tue, Dec 1, 2015 at 10:51 PM, Ka Wo Chen notifications@github.com
> wrote:
> 
> > @jonblunt https://github.com/jonblunt
> > you mean something like this?
> > 
> > def rolling_median(arr, window):
> >     return Series(arr).rolling(window=window).median().values
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > https://github.com/pydata/pandas/pull/11603#issuecomment-161180237.

This is the basis of the scheme I was thinking of but I am not a great meta
programmer.

Null series needs all the instance/class attributes that the Series
rolling moment method uses.

This looks wierd because the existing rolling median is not a method

iimport pandas as pd
import numpy as np

class Null_Series(object):
    def **init**(self, kargs):
        self.params  = kargs
    def **getattr**(self, attr):
            return self.params[attr]
    def **call**(self, value, *_kargs):
        kkargs  = self.params.copy()
        kkargs.update(kargs)
        self.value =value
        return self.moment(self, *_kkargs )

```
def wraped_moment( moment, **params):
obj = Null_Series(params)
obj.moment = moment
return obj
```

def rolling_median( self, *_kargs):
    return pd.rolling_median( self.value, *_kargs )

array =np.asarray([ 1.0, 2.0, 3.0, 4.0, np.nan, 6.0 ,np.nan, 8.0, 9.0 ,
10.0 ])

rolling_median_obj  = wraped_moment(rolling_median, window  = 3)

rolling_median_obj(array, min_periods = 2)

array([ nan,  1.5,  2. ,  3. ,  3.5,  5. ,  nan,  7. ,  8.5,  9. ])

On Wed, Dec 2, 2015 at 1:16 PM, Jon Blunt jonblunt99@gmail.com wrote:

> Incomplete thought in last post.
> 
>  I was thinking of defining  the rolling median algorithm outside the
> pandas object and then adding it as a method to the pandas objects.
> 
> then provide a decorator to add the method to a lightweight object
> 
> class null_series(object):
> def wrap_moment( array, params)
> 
> def rolling_median(arr, window):
>     return Series(arr).rolling(window=window).median().values
> 
> On Wed, Dec 2, 2015 at 12:54 PM, Jon Blunt jonblunt99@gmail.com wrote:
> 
> > That's what I don't want to do, but think I will have to
> > 
> > I just tested that approach with these results
> > 
> > smoothed = rolling_median(array, params)
> > 
> > where both smoothed and  array are numpy arrays.  Actually they are
> > slices of the same array.
> > 
> > Using profiler, all the calls to rolling moments are made from one function whose total run time was
> > 
> > ncalls  tottime  percall  cumtime  percall filename:lineno(function)
> > 
> > 300171    1.462    0.000  130.972    0.000 smooth_item_at_loc_very_fast.py:174(smooth_once)
> > 
> >  I then replaced each call to ther rolling functions with this wrapper
> > that just puts the array in a series and  then returns the raw values
> > 
> > def alg_wrap(alg, *_params):
> >     def _alg(array, *_kargs):
> >         return alg(pd.Series(array), *_kargs).values
> >     return partial(_alg, *_params)
> > 
> > The new results were
> > 
> > 300171    1.716    0.000  204.076    0.001 smooth_item_at_loc_almost_very_fast.py:174(smooth_once)
> > 
> >  Using the 'numpy array rolling moments took  57s cumulative run time. so
> > the increase is more than the basic time to run the algorithm.
> > 
> > 70s x 400  `= 8hrs cumulative run time across the full workload.
> > 
> > At this point in the code all the arrays are the same length  , one row
> > per day no missing rows, so I don't need any alignment or reindexing
> > support.  I only put the data back into a pandas DataFrame at the end to
> > return from the group by at the next step.
> > 
> > I am thinking of some
> > 
> > On Tue, Dec 1, 2015 at 10:51 PM, Ka Wo Chen notifications@github.com
> > wrote:
> > 
> > > @jonblunt https://github.com/jonblunt
> > > you mean something like this?
> > > 
> > > def rolling_median(arr, window):
> > >     return Series(arr).rolling(window=window).median().values
> > > 
> > > —
> > > Reply to this email directly or view it on GitHub
> > > https://github.com/pydata/pandas/pull/11603#issuecomment-161180237.

I did the minimal impact change to my code to use a Series in Rolling
Median .. instead of an array.

Using profiler all the calls to rolling moments are made from one
function whose total run time was

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

300171    1.462    0.000  130.972    0.000
smooth_item_at_loc_very_fast.py:174(smooth_once)

 I then replaced each call to ther rolling functions with this wrapper that
just puts the array in a series and  then returns the raw values

def alg_wrap(alg, *_params):
    def _alg(array, *_kargs):
        return alg(pd.Series(array), *_kargs).values
    return partial(_alg, *_params)

The new results were

300171    1.716    0.000  204.076    0.001
smooth_item_at_loc_almost_very_fast.py:174(smooth_once)

 Using the 'numpy array rolling moments took  57s cumulative run time. so
the increase is more than the basic time to run the algorithm.

70s x 400  `= 8hrs cumulative run time for the process across the full
workload.

My belief is that most of that cost is the good stuff that pandas brings in
terms of indexing, alignment, dtypes, unified missing data etc.

I think creating a lightweight object that sends a passed in ndarray to the
routine  and returns the raw data  would have much lower overhead.

Could use  an extra keyword parameter in the method  raw = False ( defualt
) return as series of the same shape and index, If True return the
underlying ndarray.


@jonblunt you are almost certainly better off using cython  (or numba) and using skiplist directly. if you keep wrapping/unwrapping an array (that is pretty short). The overhead of that will certainly be more than the actual computation time. 

any other comments

@jorisvandenbossche @shoyer @sinhrks @TomAugspurger