Repository: pandas-dev/pandas
Pull request: BUG: ValueError on groupby with categoricals
- [x] closes #34951 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Within `DataFrameGroupBy._cython_agg_blocks`, if it is aggregating a one-column DataFrame, it creates a `SeriresGroupBy`, calls the function on that and takes the returned values. But the `SeriesGroupBy` also does the missing-categories reindexing. The `DataFrameGroupBy` ends up with values that contain the missing categories, and an index that does not. When they are passed into a BlockManager it raises a `ValueError` stating that their lengths don't match. 

Solutions is to have `_cython_agg_blocks` create a `SeriesGroupBy` with `observed=True` so it doesn't do any reindexing. The reindexing is left to the calling `DataFrameGroupBy`

This also explains why error only occurred in `DataFrameGroupBy` but not `SeriesGroupBy`.

Problem statement:
BUG: ValueError on groupby with categoricals
- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

In specific situations involving categorical columns, a groupby() on two or more columns runs into an error:

#### Code Sample

```python
import pandas as pd
col = pd.Categorical([0, 1])
df = pd.DataFrame({'A': col, 'B': col, 'C': col})
grouped = df.groupby(['A', 'B']).first()
# ValueError: Shape of passed values is (4, 1), indices imply (2, 1)
```

#### Expected Output

Expected output would be something like:

```
>>> grouped
       C
A B     
0 0    0
  1  NaN
1 0  NaN
  1    1
```
Instead an exception is thrown, with the error message shown above.

Hints from issue discussion:
Thanks for reporting this - I get the same exception on master. Computing this as a Series via `df.groupby(['A', 'B']).C.first()` works and gives the correct result.
take

Commit message: removed errant pdb.set_trace()

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 5f93e08d51baa..b56d7c2599033 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -1091,6 +1091,7 @@ Groupby/resample/rolling
 - Bug in :meth:`Rolling.apply` where ``center=True`` was ignored when ``engine='numba'`` was specified (:issue:`34784`)
 - Bug in :meth:`DataFrame.ewm.cov` was throwing ``AssertionError`` for :class:`MultiIndex` inputs (:issue:`34440`)
 - Bug in :meth:`core.groupby.DataFrameGroupBy.transform` when ``func='nunique'`` and columns are of type ``datetime64``, the result would also be of type ``datetime64`` instead of ``int64`` (:issue:`35109`)
+- Bug in :meth:'DataFrameGroupBy.first' and :meth:'DataFrameGroupBy.last' that would raise ``ValueError`` grouping on multiple ``Categoricals`` (:issue:`34951`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
index 1f49ee2b0b665..9acea86b14103 100644
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -1058,7 +1058,7 @@ def _cython_agg_blocks(
                     #  reductions; see GH#28949
                     obj = obj.iloc[:, 0]
 
-                s = get_groupby(obj, self.grouper)
+                s = get_groupby(obj, self.grouper, observed=True)
                 try:
                     result = s.aggregate(lambda x: alt(x, axis=self.axis))
                 except TypeError:
diff --git a/pandas/tests/groupby/test_categorical.py b/pandas/tests/groupby/test_categorical.py
index 118d928ac02f4..a87de65800008 100644
--- a/pandas/tests/groupby/test_categorical.py
+++ b/pandas/tests/groupby/test_categorical.py
@@ -1669,3 +1669,27 @@ def test_categorical_transform():
     expected["status"] = expected["status"].astype(delivery_status_type)
 
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize("func", ["first", "last"])
+def test_groupby_first_on_categorical_col_grouped_on_2_categoricals(func: str):
+    # GH 34951
+
+    cat = pd.Categorical([0, 0, 1, 1])
+    val = [0, 1, 1, 0]
+    df = pd.DataFrame({"a": cat, "b": cat, "c": val})
+
+    idx = pd.Categorical([0, 1])
+    idx = pd.MultiIndex.from_product([idx, idx], names=["a", "b"])
+    expected = {
+        "first": pd.Series([0, np.NaN, np.NaN, 1], idx, name="c"),
+        "last": pd.Series([1, np.NaN, np.NaN, 0], idx, name="c"),
+    }
+
+    df_grp = df.groupby(["a", "b"])
+    df_res = getattr(df_grp, func)()
+    tm.assert_frame_equal(df_res, expected[func].to_frame())
+
+    srs_grp = df_grp["c"]
+    srs_res = getattr(srs_grp, func)()
+    tm.assert_series_equal(srs_res, expected[func])


Code review comments:
diff_hunk: @@ -1669,3 +1669,27 @@ def test_categorical_transform():
     expected["status"] = expected["status"].astype(delivery_status_type)
 
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize("func", ["first", "last"])
+def test_groupby_first_on_categorical_col_grouped_on_2_categoricals(func: str):
+    # GH 34951
+
+    cat = pd.Categorical([0, 0, 1, 1])
+    val = [0, 1, 1, 0]
+    df = pd.DataFrame({"a": cat, "b": cat, "c": val})
+
+    idx = pd.Categorical([0, 1])
+    idx = pd.MultiIndex.from_product([idx, idx], names=["a", "b"])
+    expected = {
+        "first": pd.Series([0, np.NaN, np.NaN, 1], idx, name="c"),
+        "last": pd.Series([1, np.NaN, np.NaN, 0], idx, name="c"),
+    }
+
+    df_grp = df.groupby(["a", "b"])
+    df_res = getattr(df_grp, func)()
+    tm.assert_frame_equal(df_res, expected[func].to_frame())
+
+    srs_grp = df_grp["c"]
+    srs_res = getattr(srs_grp, func)()
path: pandas/tests/groupby/test_categorical.py
text: @user1:
can you use 

result = 
expected = 
tm.assert_frame_equal(result, expected)

much easier to read

@author:
Done