Repository: pandas-dev/pandas
Pull request: BUG: Fix groupby.apply
- [x] closes #28652
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Makes sure that the output of `groupby.apply` is built up by value instead of by reference in `reduction.pyx` to avoid the behavior from #28652.

Problem statement:
groupby indexing is giving the wrong index
#### Code Sample, a copy-pastable example if possible

```python
a = pd.DataFrame([range(3)]*3,index=map(str, range(3))).T
a.iloc[:,0] = [10, 11, 11]

a.groupby('0').apply(lambda x: print(x.index))

indexes = a.groupby('0').apply(lambda x: x.index)
print(indexes)

index_lengths = a.groupby('0').apply(lambda x: len(x.index))
print(index_lengths)

indexes_lengths = a.groupby('0').apply(lambda x: x.index).apply(len)
print(indexes_lengths)

assert indexes_lengths.equals(index_lengths), "Apply indexes are returning the wrong index"
```
#### Problem description

Groupby indexing is returning the wrong value. It is returning the indexes of the last group for all the groups.

#### Expected Output

```
0
10    Int64Index([0], dtype='int64')
11    Int64Index([1, 2], dtype='int64')
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.8.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.2.0-arch2-1-ARCH
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : C
LOCALE           : None.None

pandas           : 0.25.1
numpy            : 1.15.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : 0.28.5
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : 0.4.0
xlsxwriter       : None
lxml.etree       : 4.3.4
html5lib         : None
pymysql          : None
psycopg2         : 2.8.3 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : 7.5.0
pandas_datareader: None
bs4              : None
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.4
matplotlib       : 2.2.2
numexpr          : 2.6.9
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.13.0
pytables         : None
s3fs             : 0.1.5
scipy            : 1.3.1
sqlalchemy       : None
tables           : 3.4.4
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None


</details>

Hints from issue discussion:

Commit message: Use is_scalar check

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index eb4b72d01d59a..2bdf1e3cde1db 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -270,7 +270,7 @@ Plotting
 Groupby/resample/rolling
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
--
+- Bug in :meth:`DataFrame.groupby.apply` only returning output from a single group in some cases (:issue:`28652`)
 - Bug in :meth:`DataFrame.rolling` not allowing for rolling over datetimes when ``axis=1`` (:issue: `28192`)
 - Bug in :meth:`DataFrame.groupby` not offering selection by column name when ``axis=1`` (:issue:`27614`)
 - Bug in :meth:`DataFrameGroupby.agg` not able to use lambda function with named aggregation (:issue:`27519`)
diff --git a/pandas/_libs/reduction.pyx b/pandas/_libs/reduction.pyx
index 361c21c18c4da..da58f9b0f747a 100644
--- a/pandas/_libs/reduction.pyx
+++ b/pandas/_libs/reduction.pyx
@@ -1,3 +1,4 @@
+from copy import copy
 from distutils.version import LooseVersion
 
 from cython import Py_ssize_t
@@ -16,6 +17,7 @@ cnp.import_array()
 
 cimport pandas._libs.util as util
 from pandas._libs.lib import maybe_convert_objects, values_from_object
+from pandas.api.types import is_scalar
 
 
 cdef _get_result_array(object obj, Py_ssize_t size, Py_ssize_t cnt):
@@ -543,7 +545,10 @@ def apply_frame_axis0(object frame, object f, object names,
                 # `piece` might not have an index, could be e.g. an int
                 pass
 
-            results.append(piece)
+            if is_scalar(piece):
+                results.append(piece)
+            else:
+                results.append(copy(piece))
 
             # If the data was modified inplace we need to
             # take the slow path to not risk segfaults
diff --git a/pandas/tests/frame/test_apply.py b/pandas/tests/frame/test_apply.py
index 0328232213588..f0e565b8888b2 100644
--- a/pandas/tests/frame/test_apply.py
+++ b/pandas/tests/frame/test_apply.py
@@ -1357,3 +1357,11 @@ def test_frequency_is_original(self, num_cols):
         df = DataFrame(1, index=index, columns=range(num_cols))
         df.apply(lambda x: x)
         assert index.freq == original.freq
+
+    def test_func_returns_object(self):
+        # GH 28652
+        df = DataFrame({"a": [1, 2]}, index=pd.Int64Index([1, 2]))
+        result = df.groupby("a").apply(lambda g: g.index)
+
+        tm.assert_index_equal(result[1], pd.Int64Index([1]))
+        tm.assert_index_equal(result[2], pd.Int64Index([2]))


Code review comments:
diff_hunk: @@ -1357,3 +1357,11 @@ def test_frequency_is_original(self, num_cols):
         df = DataFrame(1, index=index, columns=range(num_cols))
         df.apply(lambda x: x)
         assert index.freq == original.freq
+
+    def test_func_returns_object(self):
+        # GH 28652
+        df = DataFrame({"a": [1, 2]}, index=pd.Int64Index([1, 2]))
+        result = df.groupby("a").apply(lambda g: g.index)
+
+        tm.assert_index_equal(result[1], pd.Int64Index([1]))
+        tm.assert_index_equal(result[2], pd.Int64Index([2]))
path: pandas/tests/frame/test_apply.py
text: @user1:
Hmmm...is there a reason why we can't do something like `tm.assert_index_equal(result, expected)` ?

@author:
Updated, I changed it to a single `assert_series_equal`