Repository: pandas-dev/pandas
Pull request: BUG: Groupby.apply wasn't allowing for functions which return lists
- [ ] closes #31441 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
TypeError: copy() takes no keyword arguments
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd

df = pd.DataFrame(['A', 'A', 'B', 'B'], columns=['grps'])
df.groupby('grps').apply(lambda x: x.index.to_list())
```

```python
Traceback (most recent call last):
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 735, in apply
    result = self._python_apply_general(f)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 751, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 171, in apply
    result_values, mutated = splitter.fast_apply(f, group_keys)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 925, in fast_apply
    return libreduction.apply_frame_axis0(sdata, f, names, starts, ends)
  File "pandas/_libs/reduction.pyx", line 505, in pandas._libs.reduction.apply_frame_axis0
TypeError: copy() takes no keyword arguments
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3319, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-a3e576516eed>", line 4, in <module>
    df.groupby('grps').apply(lambda x: x.index.to_list())
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 746, in apply
    return self._python_apply_general(f)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 751, in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 171, in apply
    result_values, mutated = splitter.fast_apply(f, group_keys)
  File "/home/bay2rng/miniconda3/envs/p356_rtp_production_scheduling/lib/python3.7/site-packages/pandas/core/groupby/ops.py", line 925, in fast_apply
    return libreduction.apply_frame_axis0(sdata, f, names, starts, ends)
  File "pandas/_libs/reduction.pyx", line 505, in pandas._libs.reduction.apply_frame_axis0
TypeError: copy() takes no keyword arguments

```
#### Problem description

This snippet is working well on version 0.25.3. It breaks in version 1.0.0.

#### Expected Output

```python
A    [0, 1]
B    [2, 3]
dtype: object
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-72-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8
pandas           : 1.0.0
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.1.0.post20200127
Cython           : None
pytest           : 5.3.2
hypothesis       : None
sphinx           : 2.3.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.11.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.15.1
pytables         : None
pytest           : 5.3.2
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : 1.3.13
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None


</details>

Hints from issue discussion:
@BayerSe thanks for the report! 

cc @jbrockmendel 
@BayerSe @jorisvandenbossche I get the same error in 0.25.3
Not on this above mentioned dataframe, but on my own data.
I think this is caused by #28662, cc @dsaxton.

Commit message: Revert patch to original

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.1.rst b/doc/source/whatsnew/v1.0.1.rst
index ff8433c7cafd9..7b08e3ee1f004 100644
--- a/doc/source/whatsnew/v1.0.1.rst
+++ b/doc/source/whatsnew/v1.0.1.rst
@@ -15,7 +15,7 @@ including other versions of pandas.
 
 Bug fixes
 ~~~~~~~~~
-
+- Bug in :meth:`GroupBy.apply` was raising ``TypeError`` if called with function which returned list (:issue:`31441`)
 
 Categorical
 ^^^^^^^^^^^
diff --git a/pandas/_libs/reduction.pyx b/pandas/_libs/reduction.pyx
index 8571761f77265..9b89ddd816f14 100644
--- a/pandas/_libs/reduction.pyx
+++ b/pandas/_libs/reduction.pyx
@@ -1,4 +1,4 @@
-from copy import copy
+from copy import copy, deepcopy
 from distutils.version import LooseVersion
 
 from cython import Py_ssize_t
@@ -499,7 +499,9 @@ def apply_frame_axis0(object frame, object f, object names,
                 # `piece` might not have an index, could be e.g. an int
                 pass
 
-            if not is_scalar(piece):
+            if isinstance(piece, list):
+                piece = deepcopy(piece)
+            elif not is_scalar(piece):
                 # Need to copy data to avoid appending references
                 if hasattr(piece, "copy"):
                     piece = piece.copy(deep="all")
diff --git a/pandas/tests/groupby/test_apply.py b/pandas/tests/groupby/test_apply.py
index c18ef73203914..82a5c3e07e80f 100644
--- a/pandas/tests/groupby/test_apply.py
+++ b/pandas/tests/groupby/test_apply.py
@@ -827,3 +827,11 @@ def test_apply_index_has_complex_internals(index):
     df = DataFrame({"group": [1, 1, 2], "value": [0, 1, 0]}, index=index)
     result = df.groupby("group").apply(lambda x: x)
     tm.assert_frame_equal(result, df)
+
+
+def test_apply_function_returns_list():
+    # GH 31441
+    df = pd.DataFrame(["A", "A", "B", "B"], columns=["groups"])
+    result = df.groupby("groups").apply(lambda x: x.index.to_list())
+    expected = pd.Series([[0, 1], [2, 3]], index=pd.Index(["A", "B"], name="groups"))
+    tm.assert_series_equal(result, expected)


Code review comments:
diff_hunk: @@ -499,7 +499,9 @@ def apply_frame_axis0(object frame, object f, object names,
                 # `piece` might not have an index, could be e.g. an int
                 pass
 
-            if not is_scalar(piece):
+            if isinstance(piece, list):
path: pandas/_libs/reduction.pyx
text: @user1:
Do we want to special case lists specifically here? How about special casing only those times when `copy` is known to have the `deep` keyword for `piece.copy(deep="all")` and letting everything else use a `deepcopy`? (eg., something like `df.groupby("a").apply(lambda x: set(x.index))` would still raise here)

@author:
>  How about special casing only those times when copy is known to have the deep keyword

How - by using `inspect`?

diff_hunk: @@ -499,7 +499,9 @@ def apply_frame_axis0(object frame, object f, object names,
                 # `piece` might not have an index, could be e.g. an int
                 pass
 
-            if not is_scalar(piece):
+            if isinstance(piece, list):
path: pandas/_libs/reduction.pyx
text: @author:
@user1 here's the difference:

```
>>> df = DataFrame({"key": [1, 1, 1, 2, 2, 2, 3, 3, 3], "value": range(9)})

>>> result1 = df.groupby("key", group_keys=True).apply(lambda x: x[:].key)
>>> result2 = df.groupby("key", group_keys=True).apply(lambda x: x.key)
>>> result1
key   
1    0    1
     1    1
     2    1
2    3    2
     4    2
     5    2
3    6    3
     7    3
     8    3

>>> result2
key  0  1  2
key         
1    1  1  1
2    2  2  2
3    3  3  3
```

> Returning a list from a groupby op is thorny to begin with so I think should streamline logic as much as possible

OK, I'll think of a workaround (but I realise that you'd like to get a v1.0.1 release out very soon so if you think this'll take me too long please do feel free to take over)

@author:
I'm somewhat confused here. If I use this, the test case passes:
```
pd._testing.assert_series_equal(deepcopy(piece), piece.copy(deep="all"))
piece = piece.copy(deep="all")
```
but if I use this, it doesn't!
```
pd._testing.assert_series_equal(deepcopy(piece), piece.copy(deep="all"))
piece = deepcopy(piece)
```

will look into it