Repository: pandas-dev/pandas
Pull request: TST: groupby apply called multiple times
closes #31111 
xref https://github.com/pandas-dev/pandas/pull/24748
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ Added test for testing func should be called len(set(groupby_column)) times when passed to groupby.apply(func) ] whatsnew entry


Problem statement:
Strange groupby apply behaviour
#### Minimal reproducible example

```python
import pandas as pd
import numpy as np

tdf = pd.DataFrame({"tree": [0, 0, 0, 0, 1, 1, 1, 1],
                    "into": ["0-2", "0-4", "0-10", np.nan, "1-2", "1-3", "1-7", np.nan],
                    "leaf_value": [0,0,0,3,0,0,0,4]},
                   index=["0-0", "0-2", "0-4", "0-10", "1-0", "1-1", "1-2", "1-7"])

def deduce_tree(df):
    print("DEDUCING TREE WITH INDEX:\n",df.index)
    next_id = df.index[0]
    while isinstance(next_id, str):
        print(next_id)
        next_node = df.loc[next_id, :]
        next_id = df.loc[next_id, "into"]
    print("RETURNING:\n", next_node)
    return next_node

tdf.groupby("tree").apply(deduce_tree)

```
<details>

DEDUCING GREE WITH INDEX:
 Index(['0-0', '0-2', '0-4', '0-10'], dtype='object')
0-0
0-2
0-4
0-10
RETURNING:
 tree            0
into          NaN
leaf_value      3
Name: 0-10, dtype: object
DEDUCING GREE WITH INDEX:
 Index(['1-0', '1-1', '1-2', '1-7'], dtype='object')
1-0
DEDUCING GREE WITH INDEX:
 Index(['0-0', '0-2', '0-4', '0-10'], dtype='object')
0-0
0-2
0-4
0-10
RETURNING:
 tree            0
into          NaN
leaf_value      3
Name: 0-10, dtype: object
DEDUCING GREE WITH INDEX:
 Index(['1-0', '1-1', '1-2', '1-7'], dtype='object')
1-0
1-2
1-7
RETURNING:
 tree            1
into          NaN
leaf_value      4
Name: 1-7, dtype: object

</details>

#### Problem description

This came up when trying to analyze a boosted tree internals. As the apply function gets called with a print statement, when it gets to the line `next_node = df.loc[next_id, :]` it just calls the deduce_tree function again, with the group 0.
it prints out `DEDUCING TREE WITH INDEX` 3 times as opposed to 2, and for some reason interrupts the function.

the result of `tdf.groupby("tree").apply(deduce_tree)` is correct, but it seems to do some unnecessary work and if I want to implement some side effects into `deduce_tree` it gets messed up.

Can anyone explain why it works like this? Is this some bug? How can a .loc interrupt a function?

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.2.1-1.el7.elrepo.x86_64
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.18.1
pytz             : 2018.4
dateutil         : 2.7.3
pip              : 19.2.3
setuptools       : 41.0.1
Cython           : None
pytest           : 4.1.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.8.1 (dt dec pq3 ext lo64)
jinja2           : 2.10
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : 1.2.8
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
Thanks for your report.

> it prints out DEDUCING TREE WITH INDEX 3 times as opposed to 2

It gets printed 4 times when I run it on `master` - could you please include your output?
you can see my output unter the first details tab, turns out, thats also 4 times, I just misread it for some reason. still behavior is strange.
Thanks, will look into it
This seems to have been somehow fixed between 1.0.3 and master (currently - 239b6a721)
Hello,
I've tested the code out on version 1.0.1 and the DEDUCING TREE WITH INDEX printed out 4 times. Since I am a new contributor to Pandas I am still unsure what the masters version is. Can someone let me know what it is? 
Thank you
Hi @nthiagar 

>  I am still unsure what the masters version is

It's the `master` branch of the git repo. It's what you'll get if you build the project from source, rather than, say, with `pip install pandas`.
> This seems to have been somehow fixed between 1.0.3 and master (currently - [239b6a7](https://github.com/pandas-dev/pandas/commit/239b6a7219a4f89637451406819623f0fb631e6f))

If I've done git bisect correctly, it was fixed in fa48f5f098b5ed49a08b9d19f072559f74e1ccc2 (#32611)
take

Commit message: reformatted accordingly, for linting issues

Patch to review:
diff --git a/pandas/tests/groupby/test_apply.py b/pandas/tests/groupby/test_apply.py
index d03b03b3f862c..3c8f8b649e60e 100644
--- a/pandas/tests/groupby/test_apply.py
+++ b/pandas/tests/groupby/test_apply.py
@@ -974,3 +974,24 @@ def test_apply_function_with_indexing_return_column():
     result = df.groupby("foo1", as_index=False).apply(lambda x: x.mean())
     expected = DataFrame({"foo1": ["one", "three", "two"], "foo2": [3.0, 4.0, 4.0]})
     tm.assert_frame_equal(result, expected)
+
+
+def test_apply_function_called_count(capsys):
+    # GH: 31111
+    # groupby-apply need to execute len(set(group_by_columns)) times
+
+    expected = 2  # Number of times `apply` should call a function for the current test
+
+    df = pd.DataFrame(
+        {
+            "group_by_column": [0, 0, 0, 0, 1, 1, 1, 1],
+            "test_column": ["0", "2", "4", "6", "8", "10", "12", "14"],
+        },
+        index=["0", "2", "4", "6", "8", "10", "12", "14"],
+    )
+
+    df.groupby("group_by_column").apply(lambda df: print("function_called"))
+
+    result = capsys.readouterr().out.count("function_called")
+    # If `groupby` behaves unexpectedly, this test will break
+    assert result == expected


Code review comments:
diff_hunk: @@ -974,3 +974,24 @@ def test_apply_function_with_indexing_return_column():
     result = df.groupby("foo1", as_index=False).apply(lambda x: x.mean())
     expected = DataFrame({"foo1": ["one", "three", "two"], "foo2": [3.0, 4.0, 4.0]})
     tm.assert_frame_equal(result, expected)
+
+
+def test_apply_function_called_count(capsys):
path: pandas/tests/groupby/test_apply.py
text: @user1:
move this near test_group_apply_once_per_group and indicate the same issue number (as well)

@user1:
and rename these test to test_group_apply_once_per_group2

@author:
Alright @user1 . Will change it accordingly

@author:
Should remove `GH: 31111` and use `# GH2936, GH7739, GH10519, GH2656, GH12155, GH20084, GH21417` @user1 

@user1:
you can add this issue number as well is ok 

@author:
Alright. fixed accordingly