ENH: Add pipe method
State: closed

Closes https://github.com/pydata/pandas/issues/10129

In the dev meeting, we settled on the following:
- `.pipe` will **not** include a check for `__pipe_func__` on the function passed in.
- To avoid messiness with lambdas when a function takes the DataFrame other than in the first position, users can pass in a `(callable, data_keyword)`  argument to `.pipe` (thanks @mwaskom)

``` python
   import statsmodels.formula.api as sm

   bb = pd.read_csv('data/baseball.csv', index_col='id')

   (bb.query('h > 0')
      .assign(ln_h = lambda df: np.log(df.h))
      # sm.possion expects `formula, data`
      .pipe((sm.poisson, 'data'), 'hr ~ ln_h + year + g + C(lg)')
      .fit()
      .summary()
   )
## -- End pasted text --
Optimization terminated successfully.
         Current function value: 2.116284
         Iterations 24
Out[1]:
<class 'statsmodels.iolib.summary.Summary'>
"""
                          Poisson Regression Results
==============================================================================
Dep. Variable:                     hr   No. Observations:                   68
Model:                        Poisson   Df Residuals:                       63
Method:                           MLE   Df Model:                            4
Date:                Tue, 02 Jun 2015   Pseudo R-squ.:                  0.6878
Time:                        20:57:27   Log-Likelihood:                -143.91
converged:                       True   LL-Null:                       -460.91
                                        LLR p-value:                6.774e-136
===============================================================================
                  coef    std err          z      P>|z|      [95.0% Conf. Int.]
-------------------------------------------------------------------------------
Intercept   -1267.3636    457.867     -2.768      0.006     -2164.767  -369.960
C(lg)[T.NL]    -0.2057      0.101     -2.044      0.041        -0.403    -0.008
ln_h            0.9280      0.191      4.866      0.000         0.554     1.302
year            0.6301      0.228      2.762      0.006         0.183     1.077
g               0.0099      0.004      2.754      0.006         0.003     0.017
===============================================================================
"""
```

Thanks everyone for the input in the issue.

---

This is mostly ready. What's a good name for the argument to `pipe`? I have `func` right now, @shoyer you had `target` in the issue thread. I've been using `target` as the keyword expecting the data, i.e. where the DataFrame should be pipe to.

The tests are extremely minimal... but so is the implementation. Am I missing any obvious edge-cases?

We'll see how this goes. I don't think I push `.pipe` as a protocol at all in the documentation, though we can change that in the future. We should be forwards-compatible if we do ever go down the `__pipe_func__` route.


Labels:
name: Enhancement
name: API Design

Comments:
A couple of other things that would be nice to highlight in the docs:
1. Let's mention how this was inspired by popular pipe operator `%>%` from R's magrittr package, but the implementation here is explicit and Pythonic. I would encourage reading the source code -- we might even include it in the docs.
2. We should encourage just using `pipe` instead of [monkey patching](http://pandas.pydata.org/pandas-docs/stable/faq.html#ref-monkey-patching). I would consider removing the mention of monkey patching at all -- this is a far better way to go.

> I would consider removing the mention of monkey patching at all

Any objections to removing this section from the docs? I forgot it even existed.

> Let's mention how this was inspired by popular pipe operator %>% from R's magrittr package, but the implementation here is explicit and Pythonic. I would encourage reading the source code -- we might even include it in the docs.

The pipe method is inspired by unix pipes and more recently dplyr_ and magrittr_, which
have introduced the popular `(%>%)` (read pipe) operator for R_.
The implementation of `pipe` here is quite clean and feels right at home in python.
We encourage you to view the source code (`pd.DataFrame.pipe??` in IPyhton).

---

Ok, addressed all the comments I think (thanks). I removed the section on monkey patching.

The outstanding issue I see is @shoyer's comment about maybe checking whether we're about to clobber a kwarg when the tuple-style is used.

``` python
if target in kwargs:
    raise ValueError('%s is both the pipe target and a keyword argument' % target)
```

to catch a case of `df.add(1).pipe((f, 'data'), x=x, y=y, data=df)`. Right now we (silently) replace `kwargs['data']`, The closest parallel I see is writing `f(a=1, a=2)`, which Python catches.

I put the kwarg clobbering `(df.pipe((f, 'data'), x=1, data=2))` in a second commit: https://github.com/pydata/pandas/commit/968d0aeb217e111692ce904df2c5595ff6140de6

EDIT: I also removed the monkey patching docs FYI, can reinstate them if anyone is attached to them.

pls rebase on master and repush had an issue with some builds

Travis is green. Are we good with checking whether the `target` is clobbered? And are we OK with raising a `ValueError` instead of `SyntaxError` like `f(a=2, a=3)` does?

FYI monkey patching docs were added by @y-p in this PR: #2530

I am also +1 on removing them... but perhaps someone thinks they are a good idea?

Updated...

added panel tests, section in whatsnew/Highligts. Rebuilding the docs now

ok on removing monkey-patching docs. if you really really want to keep them, then could move to http://pandas.pydata.org/pandas-docs/stable/internals.html#subclassing-pandas-data-structures (FYI need to remove that link as well)

Ok, fixed that newline in the docstring and I just link people to the `pipe` section from `internals.rst` instead of the monkeypathcing.

@TomAugspurger lgtm. merge when ready (see that minor point above though)

looks pretty much good to go for me, too, after a few quick fixes

Ok, I added a section header for `pipe` in whatsnew, linked to that from above. Fixed the (two) typos on function, and added the `...` continuations in the docstring.

We waiting on Travis? It's felt slow today :/

travis IS slow today (on master anyhow). not real sure why.

OK, fixed the accidental deletion of api.rst

@shoyer @jreback merge at your leisure. Or I'll be back online later tonight to push the button :)

http://blog.ibis-project.org/design-composability/

from @wesm