Repository: pandas-dev/pandas

Issue:
API: Add pipe method to GroupBy objects
Extend the new "pipe" protocol to GroupBy objects to allow for piping of a wider class of functions.  Currently, one can only create pipes that chain together objects inheriting from NDFrame.  But the concept of piping is general and could be extended to other pandas objects, specifically anything inheriting from GroupBy.

The use case is to write pipe that allow one to freely transform back-and-forth between NDFrames and GroupBy objects.  Example:

```
df = DataFrame({A: [...], B: [...]})

def f(dfgb):
    return dfgb['B'].value_counts()

def g(srs):
    return srs * 2

grouped = df.groupby('A')

grouped.pipe(f).pipe(g)
```

Note that these transformations are transformations are
- GroupBy -> Series
- Series -> Series
  and the chain seamlessly switches from a GroupBy.pipe to a NDFrame.pipe

There are a few ways to implement this.  A simple way is to break out the core functionality of "pipe" into a pure function and then to call that function in any method implementation of pipe.  Another way is to think of piping as a mix-in trait, put it as a method in a base class, and then mix that base class into any class that wants to implement pipe-ability.  I have no strong preference between these options, and I'm open to other implementations that may be more inline with Pandas' design goals or the long-term vision of the "pipe" concept.

A strawman implementation of the first implementation suggestion can be found here:
https://github.com/pydata/pandas/compare/master...ghl3:groupby-pipe

CC
@TomAugspurger 
@shoyer 



Discussion:
Yep, this looks like a good idea to me.

I don't have strong feelings about how its implemented, though I suspect the mixin approach is the way to go -- that could make it easier to add documentation specific to each class without a lot of duplicated text. (On the other hand, I suspect few people look at the docstrings on groupby methods.) Either way it's pretty straightforward.

How is this different than the already existing `df.groupby(..).apply(..)`? (apart from the ability to pass a tuple in pipe)

It's similar.  df.groupby(..).apply(..) applies the function to the underlying DataFrame in each group, so the function should take a DataFrame.  This pipe implementation would act on the groupby itself, so you pass it functions whose argument is a DataFrameGroupBy.

So, with this, you could do:

```
def f(dfgb):
    return dfgb.get_group('A')

dfgb.pipe(f)
```

But if you tried with apply, it would fail:

```
dfgb.apply(f)
```

Ah sorry, misread the fact it would act on the whole GroupBy object instead on the DataFrames/groups.

Do you have an example of a real use case for this? (apart from the dummy example above, just curious)

Sure.  One thing I find myself doing a lot with pandas is working on classification problems (as in machine-learning like problems), and in particular using pandas plotting as a means of exploring and diagnosing classification problems.  I've found that a nice way to build helper functions for transforming and plotting data in this domain is to work with a DataFrameGroupBy, where the grouped variable is the class associated with the classification.  It's simply a convenient interface to build functions around, as the class is implicit in the grouped column, and it supports multiple classes or nested classes, etc.  I have many such functions whose first argument is a DataFrameGroupBy.

So, a common pattern is to start with an initial dataframe, do a lot of transformations on it, and then feed it into a function that takes a DataFrameGroupBy for the purpose of plotting or reporting.  This means that the last function call in my chain of transformations is one that takes a dataframe group by.  With the new piping functions, it would be nice to do something like:

`df = df.pipe(f).pipe(g).pipe(h).groupby('group).pipe(generate_report)`

Of course, all this is possible without adding a pipe function, one can always create a temporary variable or do this in a number of other ways, but since we already have the pipe function on the DataFrame, I think adding it to the GroupBy creates a nice symmetry.

If I read the [R data rangling cheatsheet](http://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf) right, then in R-land `df %>% group_by(...) %>% mutate(...)` (or `... %>% summarise(…)`is basically `df.groupby(...).apply(...)` (i.e. apply the function to each group). Not sure if then `pipe` should map to different semantic (e.g. pipe in the complete gb object).


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -664,3 +664,38 @@ def _get_distinct_objs(objs):
             ids.add(id(obj))
             res.append(obj)
     return res
+
+
+def _pipe(obj, func, *args, **kwargs):
+    """
+    Apply a function ``func`` to object ``obj`` either by passing obj as the
+    first argument to the function or, in the case that the func is a tuple,
+    interpret the first element of the tuple as a function and pass the obj to
+    that function as a keyword argument whose key is the value of the second
+    element of the tuple.
+
+    Parameters
+    ----------
+    func : callable or tuple of (callable, string)
+        Function to apply to this object or, alternatively, a
+        ``(callable, data_keyword)`` tuple where ``data_keyword`` is a
+        string indicating the keyword of `callable`` that expects the
+        object.
+    args : iterable, optional
+        positional arguments passed into ``func``.
+    kwargs : dict, optional
+        a dictionary of keyword arguments passed into ``func``.
+
+    Returns
+    -------
+    object : the return type of ``func``.
+    """
+    if isinstance(func, tuple):
+        func, target = func
+        if target in kwargs:
+            msg = '%s is both the pipe target and a keyword argument' % target
+            raise ValueError(msg)
+        kwargs[target] = obj
+        return func(*args, **kwargs)
+    else:
+        return func(obj, *args, **kwargs)
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -3497,8 +3497,10 @@ def sample(self, n=None, frac=None, replace=False, weights=None,
             Alternatively a ``(callable, data_keyword)`` tuple where
             ``data_keyword`` is a string indicating the keyword of
             ``callable`` that expects the %(klass)s.
-        args : positional arguments passed into ``func``.
-        kwargs : a dictionary of keyword arguments passed into ``func``.
+        args : iterable, optional
+            positional arguments passed into ``func``.
+        kwargs : mapping, optional
+            a dictionary of keyword arguments passed into ``func``.
 
         Returns
         -------
@@ -3508,7 +3510,7 @@ def sample(self, n=None, frac=None, replace=False, weights=None,
         -----
 
         Use ``.pipe`` when chaining together functions that expect
-        on Series or DataFrames. Instead of writing
+        Series, DataFrames or GroupBy objects. Instead of writing
 
         >>> f(g(h(df), arg1=a), arg2=b, arg3=c)
 
@@ -3537,15 +3539,7 @@ def sample(self, n=None, frac=None, replace=False, weights=None,
 
     @Appender(_shared_docs['pipe'] % _shared_doc_kwargs)
     def pipe(self, func, *args, **kwargs):
-        if isinstance(func, tuple):
-            func, target = func
-            if target in kwargs:
-                raise ValueError('%s is both the pipe target and a keyword '
-                                 'argument' % target)
-            kwargs[target] = self
-            return func(*args, **kwargs)
-        else:
-            return func(self, *args, **kwargs)
+        return com._pipe(self, func, *args, **kwargs)
 
     _shared_docs['aggregate'] = ("""
     Aggregate using callable, string, dict, or list of string/callables
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -40,7 +40,7 @@
 
 from pandas.core.common import (_values_from_object, AbstractMethodError,
                                 _default_index, _not_none, _get_callable_name,
-                                _asarray_tuplesafe)
+                                _asarray_tuplesafe, _pipe)
 
 from pandas.core.base import (PandasObject, SelectionMixin, GroupByError,
                               DataError, SpecificationError)
@@ -656,9 +656,10 @@ def __iter__(self):
     @Substitution(name='groupby')
     def apply(self, func, *args, **kwargs):
         """
-        Apply function and combine results together in an intelligent way. The
-        split-apply-combine combination rules attempt to be as common sense
-        based as possible. For example:
+        Apply function and combine results together in an intelligent way.
+
+        The split-apply-combine combination rules attempt to be as common
+        sense based as possible. For example:
 
         case 1:
         group DataFrame
@@ -692,7 +693,10 @@ def apply(self, func, *args, **kwargs):
 
         See also
         --------
-        aggregate, transform"""
+        pipe : Apply function to the full GroupBy object instead of to each
+            group.
+        aggregate, transform
+        """
 
         func = self._is_builtin_func(func)
 
@@ -1691,6 +1695,54 @@ def tail(self, n=5):
         mask = self._cumcount_array(ascending=False) < n
         return self._selected_obj[mask]
 
+    def pipe(self, func, *args, **kwargs):
+        """ Apply a function with arguments to this GroupBy object,
+
+        .. versionadded:: 0.21.0
+
+        Parameters
+        ----------
+        func : callable or tuple of (callable, string)
+            Function to apply to this GroupBy object or, alternatively, a
+            ``(callable, data_keyword)`` tuple where ``data_keyword`` is a
+            string indicating the keyword of ``callable`` that expects the
+            GroupBy object.
+        args : iterable, optional
+               positional arguments passed into ``func``.
+        kwargs : dict, optional
+                 a dictionary of keyword arguments passed into ``func``.
+
+        Returns
+        -------
+        object : the return type of ``func``.
+
+        Notes
+        -----
+        Use ``.pipe`` when chaining together functions that expect
+        Series, DataFrames or GroupBy objects. Instead of writing
+
+        >>> f(g(h(df.groupby('group')), arg1=a), arg2=b, arg3=c)
+
+        You can write
+
+        >>> (df
+        ...    .groupby('group')
+        ...    .pipe(f, arg1)
+        ...    .pipe(g, arg2)
+        ...    .pipe(h, arg3))
+
+        See more `here
+        <http://pandas.pydata.org/pandas-docs/stable/groupby.html#pipe>`_
+
+        See Also
+        --------
+        pandas.Series.pipe : Apply a function with arguments to a series
+        pandas.DataFrame.pipe: Apply a function with arguments to a dataframe
+        apply : Apply function to each group instead of to the
+            full GroupBy object.
+        """
+        return _pipe(self, func, *args, **kwargs)
+
 
 GroupBy._add_numeric_operations()
 


Test patch:
diff --git a/pandas/tests/groupby/test_groupby.py b/pandas/tests/groupby/test_groupby.py
--- a/pandas/tests/groupby/test_groupby.py
+++ b/pandas/tests/groupby/test_groupby.py
@@ -3762,6 +3762,75 @@ def test_gb_key_len_equal_axis_len(self):
             assert df.loc[('foo', 'bar', 'B')] == 2
             assert df.loc[('foo', 'baz', 'C')] == 1
 
+    def test_pipe(self):
+        # Test the pipe method of DataFrameGroupBy.
+        # Issue #17871
+
+        random_state = np.random.RandomState(1234567890)
+
+        df = DataFrame({'A': ['foo', 'bar', 'foo', 'bar',
+                              'foo', 'bar', 'foo', 'foo'],
+                        'B': random_state.randn(8),
+                        'C': random_state.randn(8)})
+
+        def f(dfgb):
+            return dfgb.B.max() - dfgb.C.min().min()
+
+        def square(srs):
+            return srs ** 2
+
+        # Note that the transformations are
+        # GroupBy -> Series
+        # Series -> Series
+        # This then chains the GroupBy.pipe and the
+        # NDFrame.pipe methods
+        result = df.groupby('A').pipe(f).pipe(square)
+
+        index = Index([u'bar', u'foo'], dtype='object', name=u'A')
+        expected = pd.Series([8.99110003361, 8.17516964785], name='B',
+                             index=index)
+
+        assert_series_equal(expected, result)
+
+    def test_pipe_args(self):
+        # Test passing args to the pipe method of DataFrameGroupBy.
+        # Issue #17871
+
+        df = pd.DataFrame({'group': ['A', 'A', 'B', 'B', 'C'],
+                           'x': [1.0, 2.0, 3.0, 2.0, 5.0],
+                           'y': [10.0, 100.0, 1000.0, -100.0, -1000.0]})
+
+        def f(dfgb, arg1):
+            return (dfgb.filter(lambda grp: grp.y.mean() > arg1, dropna=False)
+                        .groupby(dfgb.grouper))
+
+        def g(dfgb, arg2):
+            return dfgb.sum() / dfgb.sum().sum() + arg2
+
+        def h(df, arg3):
+            return df.x + df.y - arg3
+
+        result = (df
+                  .groupby('group')
+                  .pipe(f, 0)
+                  .pipe(g, 10)
+                  .pipe(h, 100))
+
+        # Assert the results here
+        index = pd.Index(['A', 'B', 'C'], name='group')
+        expected = pd.Series([-79.5160891089, -78.4839108911, None],
+                             index=index)
+
+        assert_series_equal(expected, result)
+
+        # test SeriesGroupby.pipe
+        ser = pd.Series([1, 1, 2, 2, 3, 3])
+        result = ser.groupby(ser).pipe(lambda grp: grp.sum() * grp.count())
+
+        expected = pd.Series([4, 8, 12], index=pd.Int64Index([1, 2, 3]))
+
+        assert_series_equal(result, expected)
+
 
 def _check_groupby(df, result, keys, field, f=lambda x: x.sum()):
     tups = lmap(tuple, df[keys].values)
diff --git a/pandas/tests/groupby/test_whitelist.py b/pandas/tests/groupby/test_whitelist.py
--- a/pandas/tests/groupby/test_whitelist.py
+++ b/pandas/tests/groupby/test_whitelist.py
@@ -239,17 +239,17 @@ def test_groupby_blacklist(df_letters):
 def test_tab_completion(mframe):
     grp = mframe.groupby(level='second')
     results = set([v for v in dir(grp) if not v.startswith('_')])
-    expected = set(
-        ['A', 'B', 'C', 'agg', 'aggregate', 'apply', 'boxplot', 'filter',
-         'first', 'get_group', 'groups', 'hist', 'indices', 'last', 'max',
-         'mean', 'median', 'min', 'ngroups', 'nth', 'ohlc', 'plot',
-         'prod', 'size', 'std', 'sum', 'transform', 'var', 'sem', 'count',
-         'nunique', 'head', 'describe', 'cummax', 'quantile',
-         'rank', 'cumprod', 'tail', 'resample', 'cummin', 'fillna',
-         'cumsum', 'cumcount', 'ngroup', 'all', 'shift', 'skew',
-         'take', 'tshift', 'pct_change', 'any', 'mad', 'corr', 'corrwith',
-         'cov', 'dtypes', 'ndim', 'diff', 'idxmax', 'idxmin',
-         'ffill', 'bfill', 'pad', 'backfill', 'rolling', 'expanding'])
+    expected = {
+        'A', 'B', 'C', 'agg', 'aggregate', 'apply', 'boxplot', 'filter',
+        'first', 'get_group', 'groups', 'hist', 'indices', 'last', 'max',
+        'mean', 'median', 'min', 'ngroups', 'nth', 'ohlc', 'plot',
+        'prod', 'size', 'std', 'sum', 'transform', 'var', 'sem', 'count',
+        'nunique', 'head', 'describe', 'cummax', 'quantile',
+        'rank', 'cumprod', 'tail', 'resample', 'cummin', 'fillna',
+        'cumsum', 'cumcount', 'ngroup', 'all', 'shift', 'skew',
+        'take', 'tshift', 'pct_change', 'any', 'mad', 'corr', 'corrwith',
+        'cov', 'dtypes', 'ndim', 'diff', 'idxmax', 'idxmin',
+        'ffill', 'bfill', 'pad', 'backfill', 'rolling', 'expanding', 'pipe'}
     assert results == expected