Repository: pandas-dev/pandas
Pull request: Fix GroupBy nth Handling with Observed=False
- [X] closes #26385
- [X] tests added / passed
- [] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Problem statement:
GroupBy nth ignores observed keyword for Categorical
Was looking to consolidate nth, first and last in the GroupBy world but was hitting some issues with categorical tests and handling of missing data. Note that the following calls are not equivalent, though they should be:

```python
>>> import numpy as np
>>> import pandas as pd

>>> s1 = pd.Categorical([np.nan, 'a', np.nan, 'a'], categories=['a', 'b', 'c'])
>>> s2 = pd.Series([1, 2, 3, 4])
>>> df = pd.DataFrame({'s1': s1, 's2': s2})
>>> grp = df.groupby('s1', observed=False)
>>> grp.first()
     s2
s1
a   2.0
b   NaN
c   NaN

>>> grp.nth(0)
    s2
s1
a    2
```

Hints from issue discussion:
I get the output - 

```python
>>> grp.nth(0)
     s2
s1    
a    2
c    1
```

Commit message: lint fixup

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.0.rst b/doc/source/whatsnew/v0.25.0.rst
index d38ee7b8b589a..eb6187c158afd 100644
--- a/doc/source/whatsnew/v0.25.0.rst
+++ b/doc/source/whatsnew/v0.25.0.rst
@@ -443,6 +443,7 @@ Groupby/Resample/Rolling
 - Bug in :meth:`pandas.core.groupby.GroupBy.cumsum`, :meth:`pandas.core.groupby.GroupBy.cumprod`, :meth:`pandas.core.groupby.GroupBy.cummin` and :meth:`pandas.core.groupby.GroupBy.cummax` with categorical column having absent categories, would return incorrect result or segfault (:issue:`16771`)
 - Bug in :meth:`pandas.core.groupby.GroupBy.nth` where NA values in the grouping would return incorrect results (:issue:`26011`)
 - Bug in :meth:`pandas.core.groupby.SeriesGroupBy.transform` where transforming an empty group would raise error (:issue:`26208`)
+- Bug in :meth:`pandas.core.groupby.GroupBy.nth` where ``observed=False`` was being ignored for Categorical groupers (:issue:`26385`)
 
 
 Reshaping
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index ae28c23b85a39..a17bd8b2ad2b5 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -42,7 +42,7 @@ class providing the base-class of operations.
 from pandas.core.frame import DataFrame
 from pandas.core.generic import NDFrame
 from pandas.core.groupby import base
-from pandas.core.index import Index, MultiIndex
+from pandas.core.index import CategoricalIndex, Index, MultiIndex
 from pandas.core.series import Series
 from pandas.core.sorting import get_group_index_sorter
 
@@ -839,6 +839,7 @@ def _cython_transform(self, how, numeric_only=True, **kwargs):
     def _cython_agg_general(self, how, alt=None, numeric_only=True,
                             min_count=-1):
         output = {}
+
         for name, obj in self._iterate_slices():
             is_numeric = is_numeric_dtype(obj.dtype)
             if numeric_only and not is_numeric:
@@ -1707,7 +1708,12 @@ def nth(self,
             if not self.as_index:
                 return out
 
-            out.index = self.grouper.result_index[ids[mask]]
+            result_index = self.grouper.result_index
+            out.index = result_index[ids[mask]]
+
+            if not self.observed and isinstance(
+                    result_index, CategoricalIndex):
+                out = out.reindex(result_index)
 
             return out.sort_index() if self.sort else out
 
diff --git a/pandas/tests/groupby/conftest.py b/pandas/tests/groupby/conftest.py
index cb4fe511651ee..559f6b35b9ab6 100644
--- a/pandas/tests/groupby/conftest.py
+++ b/pandas/tests/groupby/conftest.py
@@ -76,3 +76,12 @@ def three_group():
                       'D': np.random.randn(11),
                       'E': np.random.randn(11),
                       'F': np.random.randn(11)})
+
+
+AGG_FUNCS = ['sum', 'prod', 'min', 'max', 'mean', 'median', 'var', 'first',
+             'last', 'nth']  # TODO: ohlc?
+
+
+@pytest.fixture(params=AGG_FUNCS)
+def agg_func(request):
+    return request.param
diff --git a/pandas/tests/groupby/test_categorical.py b/pandas/tests/groupby/test_categorical.py
index 112f7629d735a..fb7f79cd53973 100644
--- a/pandas/tests/groupby/test_categorical.py
+++ b/pandas/tests/groupby/test_categorical.py
@@ -433,6 +433,35 @@ def test_observed_groups_with_nan(observed):
     tm.assert_dict_equal(result, expected)
 
 
+def test_observed_ops(agg_func):
+    cat = pd.Categorical(['a', np.nan, np.nan], categories=['a', 'b', 'c'])
+    ser = pd.Series([1., 2., 3.])
+    df = pd.DataFrame({'cat': cat, 'ser': ser})
+
+    grp = df.groupby('cat', observed=False)['ser']
+    func = getattr(grp, agg_func)
+
+    if agg_func == 'nth':  # Need an argument
+        result = func(0)
+    else:
+        result = func()
+
+    if agg_func == 'sum':  # TODO: maybe a bug?
+        expected_vals = [1., 0., 0.]
+    elif agg_func == 'prod':  # TODO: Definitely seems like a bug
+        expected_vals = [1., 1., 1.]
+    elif agg_func == 'var':
+        expected_vals = [np.nan, np.nan, np.nan]
+    else:
+        expected_vals = [1., np.nan, np.nan]
+
+    index = pd.Categorical(['a', 'b', 'c'], categories=['a', 'b', 'c'])
+    expected = pd.Series(expected_vals, index=index, name='ser')
+    expected.index.name = 'cat'
+
+    tm.assert_series_equal(result, expected)
+
+
 def test_dataframe_categorical_with_nan(observed):
     # GH 21151
     s1 = pd.Categorical([np.nan, 'a', np.nan, 'a'],


Code review comments:
diff_hunk: @@ -76,3 +76,12 @@ def three_group():
                       'D': np.random.randn(11),
                       'E': np.random.randn(11),
                       'F': np.random.randn(11)})
+
+
+AGG_FUNCS = ['sum', 'prod', 'min', 'max', 'mean', 'median', 'var', 'first',
+             'last', 'nth']  # TODO: ohlc?
+
+
+@pytest.fixture(params=AGG_FUNCS)
path: pandas/tests/groupby/conftest.py
text: @user1:
we have these defined in pandas/contest.py, all numeric reductions; add something similiar, but you can use groupby in the name, the reason is that this can then easily be shared with resample tests for example.

also need a description. and a more descriptive name.

@author:
Make sense, though given feedback on doing `nth` separately below I might just defer this for another PR

diff_hunk: @@ -433,6 +433,35 @@ def test_observed_groups_with_nan(observed):
     tm.assert_dict_equal(result, expected)
 
 
+def test_observed_ops(agg_func):
+    cat = pd.Categorical(['a', np.nan, np.nan], categories=['a', 'b', 'c'])
+    ser = pd.Series([1., 2., 3.])
+    df = pd.DataFrame({'cat': cat, 'ser': ser})
+
+    grp = df.groupby('cat', observed=False)['ser']
+    func = getattr(grp, agg_func)
+
+    if agg_func == 'nth':  # Need an argument
path: pandas/tests/groupby/test_categorical.py
text: @user1:
I would do nth separately, its not similar to the other reduction functions, and the test becomes much more straightforward.

diff_hunk: @@ -433,6 +433,35 @@ def test_observed_groups_with_nan(observed):
     tm.assert_dict_equal(result, expected)
 
 
+def test_observed_ops(agg_func):
+    cat = pd.Categorical(['a', np.nan, np.nan], categories=['a', 'b', 'c'])
+    ser = pd.Series([1., 2., 3.])
+    df = pd.DataFrame({'cat': cat, 'ser': ser})
+
+    grp = df.groupby('cat', observed=False)['ser']
path: pandas/tests/groupby/test_categorical.py
text: @user1:
needs to test using the observed fixture (meaning both True and False)