Repository: dask/dask

Issue:
Does ddf.pipe() make sense?
Often times, i end up writing a function that takes in a `dask.dataframe`. pandas implements [`pd.pipe(func)`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.pipe.html) that i find pretty convenient.

This is a pretty easy to implement but i think pipe may be pretty confusing in `dask.dataframe` world especially if someone tries to do columnar reductions. Thoughts on if it makes sense to implement?



Discussion:
cc @tomaugspurger

`pipe` is just sugar that lets you apply a function like a method. So, it seems pretty obvious and sensible to me.

> This is a pretty easy to implement but i think pipe may be pretty confusing in dask.dataframe world especially if someone tries to do columnar reductions.

I'm not sure I understand the concern here. How would this be more confusing for dask.dataframe than for pandas?

I think this makes sense too.

> pipe is just sugar that lets you apply a function like a method

Would dask expect the function to be a delayed function? Or would `pipe` accept any function that takes a`dask.dataframe`?

consider this scenario:

``` python
import pandas as pd
import dask.dataframe as dd

df = pd.DataFrame({'x': [1, 2, 3, 4, 5],
                          'y': [1., 2., 3., 4., 5.]})

ddf = dd.from_pandas(df, npartitions=2)
def f(df):
    return df.x.sum()

ddf.pipe(f).compute()
```

```
0    6
1    9
dtype: int64
```

while pandas `pipe` returns:

```
15
```

What should be the correct behavior here?

I think that dask.dataframe should almost always follow pandas semantics

in that case, it makes sense for the input function to be a delayed method. Thoughts?

```
ddf.pipe(delayed(f)).compute()
```

I would literally copy the implementation of `pipe` from pandas, e.g.,

```
    def pipe(self, func, *args, **kwargs):
        if isinstance(func, tuple):
            func, target = func
            if target in kwargs:
                raise ValueError('%s is both the pipe target and a keyword '
                                 'argument' % target)
            kwargs[target] = self
            return func(*args, **kwargs)
        else:
            return func(self, *args, **kwargs)
```

`df.pipe(func)` should not separately map `func` over partitions no more than calling a `func(df)` does.


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -531,6 +531,20 @@ def reduction(self, chunk, aggregate=None, meta=no_default,
                    meta=meta, token=token, chunk_kwargs=chunk_kwargs,
                    aggregate_kwargs=aggregate_kwargs, **kwargs)
 
+    @derived_from(pd.DataFrame)
+    def pipe(self, func, *args, **kwargs):
+        # Taken from pandas:
+        # https://github.com/pydata/pandas/blob/master/pandas/core/generic.py#L2698-L2707
+        if isinstance(func, tuple):
+            func, target = func
+            if target in kwargs:
+                raise ValueError('%s is both the pipe target and a keyword '
+                                 'argument' % target)
+            kwargs[target] = self
+            return func(*args, **kwargs)
+        else:
+            return func(self, *args, **kwargs)
+
     def random_split(self, p, random_state=None):
         """ Pseudorandomly split dataframe into different pieces row-wise
 


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1541,6 +1541,17 @@ def sum_and_count(x):
     assert eq(res, pd.DataFrame({'sum': df.sum(), 'count': df.count()}))
 
 
+def test_pipe():
+    df = pd.DataFrame({'x': range(50), 'y': range(50, 100)})
+    ddf = dd.from_pandas(df, npartitions=4)
+
+    def f(x, y, z=0):
+        return x + y + z
+
+    assert eq(ddf.pipe(f, 1, z=2), f(ddf, 1, z=2))
+    assert eq(ddf.x.pipe(f, 1, z=2), f(ddf.x, 1, z=2))
+
+
 def test_gh_517():
     arr = np.random.randn(100, 2)
     df = pd.DataFrame(arr, columns=['a', 'b'])