Repository: pandas-dev/pandas
Pull request: BUG: groupby.agg doesn't include grouping columns in result when selected
- [x] closes #51186 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Problem statement:
REF: de-duplicate iterate_slices
```
# From DataFrameGroupBy
    def _iterate_slices(self) -> Iterable[Series]:
        obj = self._selected_obj
        if self.axis == 1:
            obj = obj.T

        if isinstance(obj, Series) and obj.name not in self.exclusions:
            # Occurs when doing DataFrameGroupBy(...)["X"]
            yield obj
        else:
            for label, values in obj.items():
                if label in self.exclusions:
                    continue

                yield values
```

Iterating over self._selected_obj but then excluding self.exclusons seems a lot like just iterating over self._obj_with_exclusions.  I checked that using obj_with_exclusions _does_ break things, but didn't figure out why.

it'd be nice if we could simplify this, or at least document why we need a third thing.  cc @rhshadrach 

If it were just obj_with_exclusions, then in DataFrameGroupBy._indexed_output_to_ndframe we could just use self._obj_with_exclusions.columns instead of re-constructing an Index (i think this re-construction is largely leftover from a time when we dropped nuisance columns)

### Expected Behavior

NA

### Installed Versions

<details>

Replace this line with the output of pd.show_versions()

</details>

Hints from issue discussion:
I'm seeing 1 test in groupby fail due to this behavior

```
df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 3, 4], 'c': [5, 6, 7]})
gb = df.groupby(['a', 'b'])
result = gb.sum()
print(result)
#       c
# a b    
# 1 3  11
# 2 4   7

gb2 = gb[['a', 'b', 'c']]
result2 = gb2.sum()
print(result2)
#      a  b   c
# a b          
# 1 3  2  6  11
# 2 4  2  4   7
```

In the 2nd example, `'a'` and `'b'` are in the exclusions but this is ignored since `_obj_with_exclusions` is determined via `_selection` in this case:

https://github.com/pandas-dev/pandas/blob/74d5c1950088c2dc0d7cefea382566ea42128765/pandas/core/base.py#L214-L228

When using `.sum()`, we never hit `_iterate_slices`. Within `agg`, `_iterate_slices` causes inconsistent behavior:

```
result3 = gb2.agg(lambda x: x.sum())
print(result3)
#       c
# a b
# 1 3  11
# 2 4   7
```

To get consistent behavior, I think we can just replace `_selected_obj` with `_obj_with_exclusions` and remove any use of `exclusions`.

Commit message: BUG: groupby.agg doesn't include grouping columns in result when selected

Patch to review:
diff --git a/doc/source/whatsnew/v2.0.0.rst b/doc/source/whatsnew/v2.0.0.rst
index a37503460901b..08a9b3b4f8353 100644
--- a/doc/source/whatsnew/v2.0.0.rst
+++ b/doc/source/whatsnew/v2.0.0.rst
@@ -1362,7 +1362,7 @@ Groupby/resample/rolling
 - Bug in :meth:`.DataFrameGroupBy.describe` produced incorrect results when data had duplicate columns (:issue:`50806`)
 - Bug in :meth:`.DataFrameGroupBy.agg` with ``engine="numba"`` failing to respect ``as_index=False`` (:issue:`51228`)
 - Bug in :meth:`DataFrameGroupBy.agg`, :meth:`SeriesGroupBy.agg`, and :meth:`Resampler.agg` would ignore arguments when passed a list of functions (:issue:`50863`)
--
+- Bug in :meth:`DataFrameGroupBy.agg` with multiple groupings would not include groupings in the result when they occurred in the selected columns (:issue:`51186`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
index eecf292e4c3c8..e3490216bf531 100644
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -1320,21 +1320,15 @@ def aggregate(self, func=None, *args, engine=None, engine_kwargs=None, **kwargs)
     agg = aggregate
 
     def _iterate_slices(self) -> Iterable[Series]:
-        obj = self._selected_obj
+        obj = self._obj_with_exclusions
         if self.axis == 1:
             obj = obj.T
 
-        if isinstance(obj, Series) and obj.name not in self.exclusions:
+        if isinstance(obj, Series):
             # Occurs when doing DataFrameGroupBy(...)["X"]
             yield obj
         else:
             for label, values in obj.items():
-                if label in self.exclusions:
-                    # Note: if we tried to just iterate over _obj_with_exclusions,
-                    #  we would break test_wrap_agg_out by yielding a column
-                    #  that is skipped here but not dropped from obj_with_exclusions
-                    continue
-
                 yield values
 
     def _aggregate_frame(self, func, *args, **kwargs) -> DataFrame:
diff --git a/pandas/tests/groupby/aggregate/test_aggregate.py b/pandas/tests/groupby/aggregate/test_aggregate.py
index 4872cc27cde9a..b920e113fe8fd 100644
--- a/pandas/tests/groupby/aggregate/test_aggregate.py
+++ b/pandas/tests/groupby/aggregate/test_aggregate.py
@@ -321,8 +321,9 @@ def func(ser):
 
     with pytest.raises(TypeError, match="Test error message"):
         grouped.aggregate(func)
-    result = grouped[[c for c in three_group if c != "C"]].aggregate(func)
-    exp_grouped = three_group.loc[:, three_group.columns != "C"]
+    columns = ["D", "E", "F"]
+    result = grouped[columns].aggregate(func)
+    exp_grouped = three_group.loc[:, columns]
     expected = exp_grouped.groupby(["A", "B"]).aggregate(func)
     tm.assert_frame_equal(result, expected)
 
@@ -1515,3 +1516,16 @@ def foo2(x, b=2, c=0):
         [[8, 8], [9, 9], [10, 10]], index=Index([1, 2, 3]), columns=["foo1", "foo2"]
     )
     tm.assert_frame_equal(result, expected)
+
+
+def test_agg_groupings_selection():
+    # GH#51186 - a selected grouping should be in the output of agg
+    df = DataFrame({"a": [1, 1, 2], "b": [3, 3, 4], "c": [5, 6, 7]})
+    gb = df.groupby(["a", "b"])
+    selected_gb = gb[["b", "c"]]
+    result = selected_gb.agg(lambda x: x.sum())
+    index = MultiIndex(
+        levels=[[1, 2], [3, 4]], codes=[[0, 1], [0, 1]], names=["a", "b"]
+    )
+    expected = DataFrame({"b": [6, 4], "c": [11, 7]}, index=index)
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -1362,7 +1362,7 @@ Groupby/resample/rolling
 - Bug in :meth:`.DataFrameGroupBy.describe` produced incorrect results when data had duplicate columns (:issue:`50806`)
 - Bug in :meth:`.DataFrameGroupBy.agg` with ``engine="numba"`` failing to respect ``as_index=False`` (:issue:`51228`)
 - Bug in :meth:`DataFrameGroupBy.agg`, :meth:`SeriesGroupBy.agg`, and :meth:`Resampler.agg` would ignore arguments when passed a list of functions (:issue:`50863`)
--
+- Bug in :meth:`DataFrameGroupBy.agg` with multiple groupings would not include groupings in the result when they occurred in the selected columns (:issue:`51186`)
path: doc/source/whatsnew/v2.0.0.rst
text: @user1:
> with multiple groupings

this might not be clear without looking at the new test

@author:
Yea, I wasn't sure how to describe that the bug only occurs with `.groupby([x, y])` and not `.groupby(x)`. Is there better language that could be used here, or maybe just remove it altogether?

@user1:
'after subsetting columns (e.g. `gb[["a", "b"]].agg(...)`'?