ENH: allow .rolling / .expanding as groupby methods
State: closed

closes #12738
closes #12486 
closes #12363
- [x] more tests (other methods)

~~\- [ ] doc section in groupby~~ will do later

```
In [3]: pd.options.display.max_rows=10

In [4]:    df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,
                      'B': np.arange(40)})

In [5]: df
Out[5]: 
    A   B
0   1   0
1   1   1
2   1   2
3   1   3
4   1   4
.. ..  ..
35  3  35
36  3  36
37  3  37
38  3  38
39  3  39

[40 rows x 2 columns]

In [6]:    df.groupby('A').apply(lambda x: x.rolling(4).B.mean())
Out[6]: 
A    
1  0      NaN
   1      NaN
   2      NaN
   3      1.5
   4      2.5
         ... 
3  35    33.5
   36    34.5
   37    35.5
   38    36.5
   39    37.5
Name: B, dtype: float64

In [7]:    df.groupby('A').rolling(4).B.mean()
Out[7]: 
A    
1  0      NaN
   1      NaN
   2      NaN
   3      1.5
   4      2.5
         ... 
3  35    33.5
   36    34.5
   37    35.5
   38    36.5
   39    37.5
Name: B, dtype: float64
In [9]:    df.index = pd.date_range('20130101',freq='s',periods=40)

In [10]:    df
Out[10]: 
                     A   B
2013-01-01 00:00:00  1   0
2013-01-01 00:00:01  1   1
2013-01-01 00:00:02  1   2
2013-01-01 00:00:03  1   3
2013-01-01 00:00:04  1   4
...                 ..  ..
2013-01-01 00:00:35  3  35
2013-01-01 00:00:36  3  36
2013-01-01 00:00:37  3  37
2013-01-01 00:00:38  3  38
2013-01-01 00:00:39  3  39

[40 rows x 2 columns]

In [11]:    df.groupby('A').apply(lambda x: x.resample('4s').mean())
Out[11]: 
                         A     B
A                               
1 2013-01-01 00:00:00  1.0   1.5
  2013-01-01 00:00:04  1.0   5.5
  2013-01-01 00:00:08  1.0   9.5
  2013-01-01 00:00:12  1.0  13.5
  2013-01-01 00:00:16  1.0  17.5
2 2013-01-01 00:00:20  2.0  21.5
  2013-01-01 00:00:24  2.0  25.5
  2013-01-01 00:00:28  2.0  29.5
3 2013-01-01 00:00:32  3.0  33.5
  2013-01-01 00:00:36  3.0  37.5

In [12]:    df.groupby('A').resample('4s').mean()
Out[12]: 
                         A     B
A                               
1 2013-01-01 00:00:00  1.0   1.5
  2013-01-01 00:00:04  1.0   5.5
  2013-01-01 00:00:08  1.0   9.5
  2013-01-01 00:00:12  1.0  13.5
  2013-01-01 00:00:16  1.0  17.5
2 2013-01-01 00:00:20  2.0  21.5
  2013-01-01 00:00:24  2.0  25.5
  2013-01-01 00:00:28  2.0  29.5
3 2013-01-01 00:00:32  3.0  33.5
  2013-01-01 00:00:36  3.0  37.5
```


Labels:
name: Enhancement
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
cc @lminer

ok, this is ready. comments.

@shoyer @jorisvandenbossche @TomAugspurger @sinhrks 

Is the example at the top still up to date? 
I am wondering a bit if the grouper should be in the result or not?

Also, how does the result looks like if you have a non-sorted grouper? 
Currently with apply it sorts the groups (but also includes the grouper as the index)

@jorisvandenbossche all updated. Had to work thru some issues. But much more fully tested now. This basically replicates what `.apply` is doing with the new syntax by actually _using_ apply. In theory we could have a more efficient impl, but will leave that for later.

Also this PR cleans up a bunch of cases where the name is not returned for groupbys. So much more consistency now.

@jorisvandenbossche if you'd have a look

Will take a look later today!

Still trying to wrap my head around these changes, but already notices two things (using the timeseries df from the whatsnew docs):

```
df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,'B': np.arange(40)})
df.index = pd.date_range('20130101',freq='s',periods=40)
```
- Backwards compatibility for resample on groupby. With this PR this now gives a ResamplerGroupby:
  
  ```
  In [33]: df.groupby('A').resample('4s', 'mean')
  Out[33]: DatetimeIndexResamplerGroupby [freq=<4 * Seconds>, axis=0, closed=left,
  label=left, convention=e, base=0]
  
  ```
  
  While with 0.18.0 this still gives you a deprecation warning and the same result as in 0.17:
  
  ```
  In [16]:  df.groupby('A').resample('4s', 'mean')
  C:\Anaconda\envs\pandas018\Scripts\ipython-script.py:1: FutureWarning: how in .r
  esample() is deprecated
  the new syntax is .resample(...).mean()
  if __name__ == '__main__':
  Out[16]:
                     A     B
  A
  1 2013-01-01 00:00:00  1   1.5
  2013-01-01 00:00:04  1   5.5
  2013-01-01 00:00:08  1   9.5
  2013-01-01 00:00:12  1  13.5
  2013-01-01 00:00:16  1  17.5
  2 2013-01-01 00:00:20  2  21.5
  2013-01-01 00:00:24  2  25.5
  2013-01-01 00:00:28  2  29.5
  3 2013-01-01 00:00:32  3  33.5
  2013-01-01 00:00:36  3  37.5
  ```
- There seems to be an inconsistency between returning a MI between resample and rolling (for dataframes):
  
  ```
  In [30]: df.groupby('A').resample('3s').mean()
  Out[30]:
                     A     B
  A
  1 2013-01-01 00:00:00  1   1.0
  2013-01-01 00:00:03  1   4.0
  2013-01-01 00:00:06  1   7.0
  2013-01-01 00:00:09  1  10.0
  2013-01-01 00:00:12  1  13.0
  2013-01-01 00:00:15  1  16.0
  2013-01-01 00:00:18  1  18.5
  2 2013-01-01 00:00:18  2  20.0
  2013-01-01 00:00:21  2  22.0
  2013-01-01 00:00:24  2  25.0
  2013-01-01 00:00:27  2  28.0
  2013-01-01 00:00:30  2  30.5
  3 2013-01-01 00:00:30  3  32.0
  2013-01-01 00:00:33  3  34.0
  2013-01-01 00:00:36  3  37.0
  2013-01-01 00:00:39  3  39.0
  
  In [31]: df.groupby('A').rolling(3).mean()
  Out[31]:
                     A     B
  2013-01-01 00:00:00  NaN   NaN
  2013-01-01 00:00:01  NaN   NaN
  2013-01-01 00:00:02  1.0   1.0
  2013-01-01 00:00:03  1.0   2.0
  2013-01-01 00:00:04  1.0   3.0
  2013-01-01 00:00:05  1.0   4.0
  2013-01-01 00:00:06  1.0   5.0
  2013-01-01 00:00:07  1.0   6.0
  2013-01-01 00:00:08  1.0   7.0
  2013-01-01 00:00:09  1.0   8.0
  2013-01-01 00:00:10  1.0   9.0
  2013-01-01 00:00:11  1.0  10.0
  2013-01-01 00:00:12  1.0  11.0
  2013-01-01 00:00:13  1.0  12.0
  2013-01-01 00:00:14  1.0  13.0
  ...
  ```
  
  (in case of selecting a series (`df.groupby('A').B.rolling(3).mean()` and `df.groupby('A').B.resample('3s').mean()` both return a MI)

on your last, can you show a complete example. this works for me

```
In [5]: df = DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,
                                'B': np.arange(40)},
                               index=date_range('1/1/2000',
                                                freq='s',
                                                periods=40))

In [6]: df
Out[6]: 
                     A   B
2000-01-01 00:00:00  1   0
2000-01-01 00:00:01  1   1
2000-01-01 00:00:02  1   2
2000-01-01 00:00:03  1   3
2000-01-01 00:00:04  1   4
...                 ..  ..
2000-01-01 00:00:35  3  35
2000-01-01 00:00:36  3  36
2000-01-01 00:00:37  3  37
2000-01-01 00:00:38  3  38
2000-01-01 00:00:39  3  39

[40 rows x 2 columns]

In [7]: df.groupby('A').resample('3s').mean()
Out[7]: 
                         A     B
A                               
1 2000-01-01 00:00:00  1.0   1.0
  2000-01-01 00:00:03  1.0   4.0
  2000-01-01 00:00:06  1.0   7.0
  2000-01-01 00:00:09  1.0  10.0
  2000-01-01 00:00:12  1.0  13.0
...                    ...   ...
2 2000-01-01 00:00:30  2.0  30.5
3 2000-01-01 00:00:30  3.0  32.0
  2000-01-01 00:00:33  3.0  34.0
  2000-01-01 00:00:36  3.0  37.0
  2000-01-01 00:00:39  3.0  39.0

[16 rows x 2 columns]

In [8]: df.groupby('A').rolling(3).mean()
Out[8]: 
                         A     B
A                               
1 2000-01-01 00:00:00  NaN   NaN
  2000-01-01 00:00:01  NaN   NaN
  2000-01-01 00:00:02  1.0   1.0
  2000-01-01 00:00:03  1.0   2.0
  2000-01-01 00:00:04  1.0   3.0
...                    ...   ...
3 2000-01-01 00:00:35  3.0  34.0
  2000-01-01 00:00:36  3.0  35.0
  2000-01-01 00:00:37  3.0  36.0
  2000-01-01 00:00:38  3.0  37.0
  2000-01-01 00:00:39  3.0  38.0

[40 rows x 2 columns]
```

@jorisvandenbossche all fixed up (was not doing the deprecation step).

Just fetched your PR again, and I am still seeing this difference:

```
In [1]: df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,'B': np.arange(40)})

In [2]: df.index = pd.date_range('20130101',freq='s',periods=40)

In [3]: df.groupby('A').resample('3s').mean()
Out[3]:
                       A     B
A
1 2013-01-01 00:00:00  1   1.0
  2013-01-01 00:00:03  1   4.0
  2013-01-01 00:00:06  1   7.0
  2013-01-01 00:00:09  1  10.0
  2013-01-01 00:00:12  1  13.0
  2013-01-01 00:00:15  1  16.0
  2013-01-01 00:00:18  1  18.5
2 2013-01-01 00:00:18  2  20.0
  2013-01-01 00:00:21  2  22.0
  2013-01-01 00:00:24  2  25.0
  2013-01-01 00:00:27  2  28.0
  2013-01-01 00:00:30  2  30.5
3 2013-01-01 00:00:30  3  32.0
  2013-01-01 00:00:33  3  34.0
  2013-01-01 00:00:36  3  37.0
  2013-01-01 00:00:39  3  39.0

In [4]: df.groupby('A').rolling(3).mean()
Out[4]:
                       A     B
2013-01-01 00:00:00  NaN   NaN
2013-01-01 00:00:01  NaN   NaN
2013-01-01 00:00:02  1.0   1.0
2013-01-01 00:00:03  1.0   2.0
2013-01-01 00:00:04  1.0   3.0
2013-01-01 00:00:05  1.0   4.0
2013-01-01 00:00:06  1.0   5.0
2013-01-01 00:00:07  1.0   6.0
2013-01-01 00:00:08  1.0   7.0
2013-01-01 00:00:09  1.0   8.0
2013-01-01 00:00:10  1.0   9.0
2013-01-01 00:00:11  1.0  10.0
2013-01-01 00:00:12  1.0  11.0
2013-01-01 00:00:13  1.0  12.0
2013-01-01 00:00:14  1.0  13.0
2013-01-01 00:00:15  1.0  14.0
2013-01-01 00:00:16  1.0  15.0
2013-01-01 00:00:17  1.0  16.0
2013-01-01 00:00:18  1.0  17.0
2013-01-01 00:00:19  1.0  18.0
2013-01-01 00:00:20  NaN   NaN
2013-01-01 00:00:21  NaN   NaN
2013-01-01 00:00:22  2.0  21.0
2013-01-01 00:00:23  2.0  22.0
2013-01-01 00:00:24  2.0  23.0
2013-01-01 00:00:25  2.0  24.0
2013-01-01 00:00:26  2.0  25.0
2013-01-01 00:00:27  2.0  26.0
2013-01-01 00:00:28  2.0  27.0
2013-01-01 00:00:29  2.0  28.0
2013-01-01 00:00:30  2.0  29.0
2013-01-01 00:00:31  2.0  30.0
2013-01-01 00:00:32  NaN   NaN
2013-01-01 00:00:33  NaN   NaN
2013-01-01 00:00:34  3.0  33.0
2013-01-01 00:00:35  3.0  34.0
2013-01-01 00:00:36  3.0  35.0
2013-01-01 00:00:37  3.0  36.0
2013-01-01 00:00:38  3.0  37.0
2013-01-01 00:00:39  3.0  38.0

In [5]: pd.__version__
Out[5]: '0.18.0+109.g54e22be'
```

@jorisvandenbossche you updated? (I am slightly ahead as merged a couple of things), but this was fixed a while ago

```
In [1]: df = pd.DataFrame({'A': [1] * 20 + [2] * 12 + [3] * 8,'B': np.arange(40)})

In [2]: In [2]: df.index = pd.date_range('20130101',freq='s',periods=40)

In [3]: In [3]: df.groupby('A').resample('3s').mean()
Out[3]: 
                         A     B
A                               
1 2013-01-01 00:00:00  1.0   1.0
  2013-01-01 00:00:03  1.0   4.0
  2013-01-01 00:00:06  1.0   7.0
  2013-01-01 00:00:09  1.0  10.0
  2013-01-01 00:00:12  1.0  13.0
  2013-01-01 00:00:15  1.0  16.0
  2013-01-01 00:00:18  1.0  18.5
2 2013-01-01 00:00:18  2.0  20.0
  2013-01-01 00:00:21  2.0  22.0
  2013-01-01 00:00:24  2.0  25.0
  2013-01-01 00:00:27  2.0  28.0
  2013-01-01 00:00:30  2.0  30.5
3 2013-01-01 00:00:30  3.0  32.0
  2013-01-01 00:00:33  3.0  34.0
  2013-01-01 00:00:36  3.0  37.0
  2013-01-01 00:00:39  3.0  39.0
In [4]: df.groupby('A').rolling(3).mean()
Out[4]: 
                         A     B
A                               
1 2013-01-01 00:00:00  NaN   NaN
  2013-01-01 00:00:01  NaN   NaN
  2013-01-01 00:00:02  1.0   1.0
  2013-01-01 00:00:03  1.0   2.0
  2013-01-01 00:00:04  1.0   3.0
  2013-01-01 00:00:05  1.0   4.0
  2013-01-01 00:00:06  1.0   5.0
  2013-01-01 00:00:07  1.0   6.0
  2013-01-01 00:00:08  1.0   7.0
  2013-01-01 00:00:09  1.0   8.0
  2013-01-01 00:00:10  1.0   9.0
  2013-01-01 00:00:11  1.0  10.0
  2013-01-01 00:00:12  1.0  11.0
  2013-01-01 00:00:13  1.0  12.0
  2013-01-01 00:00:14  1.0  13.0
  2013-01-01 00:00:15  1.0  14.0
  2013-01-01 00:00:16  1.0  15.0
  2013-01-01 00:00:17  1.0  16.0
  2013-01-01 00:00:18  1.0  17.0
  2013-01-01 00:00:19  1.0  18.0
2 2013-01-01 00:00:20  NaN   NaN
  2013-01-01 00:00:21  NaN   NaN
  2013-01-01 00:00:22  2.0  21.0
  2013-01-01 00:00:23  2.0  22.0
  2013-01-01 00:00:24  2.0  23.0
  2013-01-01 00:00:25  2.0  24.0
  2013-01-01 00:00:26  2.0  25.0
  2013-01-01 00:00:27  2.0  26.0
  2013-01-01 00:00:28  2.0  27.0
  2013-01-01 00:00:29  2.0  28.0
  2013-01-01 00:00:30  2.0  29.0
  2013-01-01 00:00:31  2.0  30.0
3 2013-01-01 00:00:32  NaN   NaN
  2013-01-01 00:00:33  NaN   NaN
  2013-01-01 00:00:34  3.0  33.0
  2013-01-01 00:00:35  3.0  34.0
  2013-01-01 00:00:36  3.0  35.0
  2013-01-01 00:00:37  3.0  36.0
  2013-01-01 00:00:38  3.0  37.0
  2013-01-01 00:00:39  3.0  38.0

In [5]: pd.__version__
Out[5]: '0.18.0+120.g094c425'
```

pushed again just in case

@jreback one more lost index name, when you group a Series by another (using the original dataframe)

``` python
In [27]: df['B'].groupby(df.A).rolling(4).mean()
Out[27]:
A  idx
1  0       NaN
   1       NaN
   2       NaN
   3       1.5
   4       2.5
          ...
3  35     33.5
   36     34.5
   37     35.5
   38     36.5
   39     37.5
dtype: float64
```

Let me know if you want to fix that here, otherwise I'll open an issue.

And I'm (correctly) seeing a MultiIndex for the `df.groupby('A').resample('3s').mean()`

Will try again (I just fetched the head of this PR, didn't rebase myself as I thought you already did that)

@TomAugspurger It's with the rolling case I do not see a MultiIndex (`df.groupby('A').rolling(3).mean()`), `resample` did work for me as well.

@jreback Tested it again (fetched this PR, rebased on latest master myself, rebuild pandas to be certain), and still getting the same result as I showed in https://github.com/pydata/pandas/pull/12743#issuecomment-210904108 (using python 2.7, numpy 1.10.1, Windows)

@jorisvandenbossche hmm, ok I did on 2.7/1.10.4 and it doesn't look right. odd. (on windows)

ok, this has been failing on windows on my branch : https://ci.appveyor.com/project/jreback/pandas/build/1.0.2077/job/qfrsv7p46f6ns6xg

ok, this is not a 2.7/numpy issue, but something specific to windows.

ok pushed a commit to fix. This comes back to some of the 'figuring what the shape of the return' is logic. IOW its a heuristic depending on whether something is mutated (this is the standard case). But when we are doing a chained window/resample we need to force this.

pushed a new version, had some flake/formatting issues on windows.

@TomAugspurger I picked up that last name fix (was a bug in the concat step).

@TomAugspurger @jorisvandenbossche any more comments.

Nothing else from me 👍 

ok will merge shortly unless @jorisvandenbossche has any further comments.

Tested on master, and now indeed the inconsistency is solved! Thanks

@jreback, thanks.

I want to do grouby then rolling then corr on two columns, code below takes 18s+ on 1M rows:

```
df.groupby('name', as_index=False, sort=False, group_keys=False).apply(
        lambda x: x['_a'].rolling(d).corr(other=x['_b'], pairwise=True))
```

**Is there a faster way to do this**? i tried code but it does not work (got exceptions):

```
_g = df.groupby('name', as_index=False, sort=False, group_keys=False)
_g['_a'].rolling(5).corr(other=_g['_b'], pairwise=False)
```

Another question: **how to ignore the group key (just keep the original index)** for below code:

```
_g['_a'].rolling(d).min()
```