Repository: dask/dask

Issue:
dask.dataframe groupby(as_index=False) does nothing
When doing a groupby, the index gets set even when specifying as_index=False. This combined with issue #734 means that groupby has to be the last operation, otherwise we're losing columns.
Also it seems that groupby works fine with multiple columns, even though multiindex are not supported, which doesn't seem very consistent.



Discussion:
Yes, we should at least raise a `NotImplementedError` here.

Can you give an example of losing columns?

What I mean by losing columns is that once a column becomes the index in dask, I see no way to bring it back as a normal column. If I use set_index(drop=False) that would work, but groupby silently creates an index without giving that option. The example is a little contrived, but hopefully conveys what I'm trying to achieve:

``` python
df = pd.DataFrame({'A': [1,1,2,2,3,3], 'B': [1,2,-1, 2, 3, 4]})
df2= df.groupby('A', as_index=False).max()
df3 = df.groupby('B').mean()
```

In dask, A would be the index of df2, and then I don't see any way to operate on A, which is what I mean by losing columns.

OK, this makes sense to me.  I'm a bit saturated this week but may be able to pick this up next week.  At the very least `reset_index` should be trivial to add if we don't mind having a non-sensical index (e.g. `[0, 1, 2, 3, ... 0, 1, 2, 3, ... 0, 1, 2, 3]`)

Of course it'd be great if anyone else can pick this up in the meantime.

I can pick this up.

I'm confused by this. What was the approach you had in mind @mrocklin ?

We should probably check keyword arguments in groupby and raise notimplementederrors accordingly.  We should probably also implement `reset_index` which will just map `self._partition_type.reset_index` onto the blocks and set `divisions = [None] * (self.npartitions + 1)`


Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -1292,6 +1292,14 @@ def column_info(self):
     def nlargest(self, n=5, columns=None):
         return nlargest(self, n, columns)
 
+    @derived_from(pd.DataFrame)
+    def reset_index(self):
+        new_columns = ['index'] + list(self.columns)
+        reset_index = self._partition_type.reset_index
+        out = self.map_partitions(reset_index, columns=new_columns)
+        out.divisions = [None] * (self.npartitions + 1)
+        return out
+
     @derived_from(pd.DataFrame)
     def groupby(self, key, **kwargs):
         return GroupBy(self, key, **kwargs)
@@ -1823,6 +1831,11 @@ def __init__(self, df, index=None, key=None, **kwargs):
         self.index = index
         self.kwargs = kwargs
 
+        if not kwargs.get('as_index', True):
+            msg = ("The keyword argument `as_index=False` is not supported in "
+                   "dask.dataframe.groupby")
+            raise NotImplementedError(msg)
+
         if isinstance(index, list):
             for i in index:
                 if i not in df.columns:


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -7,6 +7,7 @@
 import pandas.util.testing as tm
 import numpy as np
 import pytest
+from numpy.testing import assert_array_almost_equal
 
 import dask
 from dask.async import get_sync
@@ -2413,3 +2414,22 @@ def test_groupby_index_array():
 
     eq(df.A.groupby(df.index.month).nunique(),
        ddf.A.groupby(ddf.index.month).nunique(), check_names=False)
+
+
+def test_groupby_set_index():
+    df = tm.makeTimeDataFrame()
+    ddf = dd.from_pandas(df, npartitions=2)
+    assert raises(NotImplementedError,
+                  lambda: ddf.groupby(df.index.month, as_index=False))
+
+
+def test_reset_index():
+    df = pd.DataFrame({'x': [1, 2, 3, 4], 'y': [10, 20, 30, 40]})
+    ddf = dd.from_pandas(df, npartitions=2)
+
+    res = ddf.reset_index()
+    exp = df.reset_index()
+
+    assert len(res.index.compute()) == len(exp.index)
+    assert res.columns == tuple(exp.columns)
+    assert_array_almost_equal(res.compute().values, exp.values)