Repository: pandas-dev/pandas
Pull request: BUG: Fix groupby observed=True when aggregating a column
- [x] closes #23970 
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
groupby observed=True not working for aggregating a column
In the following the parameter `observed=True` will be ineffective (in pandas 0.23.4)
```python
import pandas as pd
dd=pd.DataFrame({"c":pd.Series([1,1,2], dtype="category"), "i":[1,2,2], "x":[1, 2, 3]})
gr=dd.groupby(["c", "i"], as_index=False, observed=True)["x"].sum()
gr   # will contain a NaN for c=2, i=1
```
It will only work as expected if I aggregate over all DataFrame columns, i.e. leaving out `["x"]`.

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.23.4
pytest: 3.10.1
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>

Hints from issue discussion:
Seems to be an interaction with `as_index` - investigation/PR welcome!

```python
In [120]: dd.groupby(["c", "i"], observed=True)["x"].sum()
Out[120]: 
c  i
1  1    1
   2    2
2  2    3
Name: x, dtype: int64

In [121]: dd.groupby(["c", "i"], observed=True)["x"].sum().reset_index()
Out[121]: 
   c  i  x
0  1  1  1
1  1  2  2
2  2  2  3
```

Commit message: GH23970 Added relevant docstring to whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v0.24.0.rst b/doc/source/whatsnew/v0.24.0.rst
index a2abda019812a..ec26c9a8a060c 100644
--- a/doc/source/whatsnew/v0.24.0.rst
+++ b/doc/source/whatsnew/v0.24.0.rst
@@ -1560,6 +1560,7 @@ Groupby/Resample/Rolling
 - Bug in :meth:`pandas.core.groupby.GroupBy.rank` with ``method='dense'`` and ``pct=True`` when a group has only one member would raise a ``ZeroDivisionError`` (:issue:`23666`).
 - Calling :meth:`pandas.core.groupby.GroupBy.rank` with empty groups and ``pct=True`` was raising a ``ZeroDivisionError`` (:issue:`22519`)
 - Bug in :meth:`DataFrame.resample` when resampling ``NaT`` in ``TimeDeltaIndex`` (:issue:`13223`).
+- Bug in :meth:`DataFrameGroupBy._gotitem` where `observed` parameter was not passed properly, causing subsequent function calls that rely on `observed` to misbehave if called with ``observed=True`` (:issue:`23970`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
index 33a41ab1cabc4..c5142a4ee98cc 100644
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -1336,7 +1336,8 @@ def _gotitem(self, key, ndim, subset=None):
             return DataFrameGroupBy(subset, self.grouper, selection=key,
                                     grouper=self.grouper,
                                     exclusions=self.exclusions,
-                                    as_index=self.as_index)
+                                    as_index=self.as_index,
+                                    observed=self.observed)
         elif ndim == 1:
             if subset is None:
                 subset = self.obj[key]
diff --git a/pandas/tests/groupby/test_groupby.py b/pandas/tests/groupby/test_groupby.py
index 6d9f60df45ec8..e96d4a170aecc 100644
--- a/pandas/tests/groupby/test_groupby.py
+++ b/pandas/tests/groupby/test_groupby.py
@@ -1694,3 +1694,15 @@ def test_groupby_agg_ohlc_non_first():
     result = df.groupby(pd.Grouper(freq='D')).agg(['sum', 'ohlc'])
 
     tm.assert_frame_equal(result, expected)
+
+
+def test_groupby_agg_observed_lost_err():
+    # GH-23970
+    df = pd.DataFrame({
+        'a': pd.Series([1, 1, 2], dtype='category'),
+        'b': [1, 2, 2], 'x': [1, 2, 3]})
+
+    expected = df
+    result = df.groupby(['a', 'b'], as_index=False, observed=True)['x'].sum()
+
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -1560,6 +1560,7 @@ Groupby/Resample/Rolling
 - Bug in :meth:`pandas.core.groupby.GroupBy.rank` with ``method='dense'`` and ``pct=True`` when a group has only one member would raise a ``ZeroDivisionError`` (:issue:`23666`).
 - Calling :meth:`pandas.core.groupby.GroupBy.rank` with empty groups and ``pct=True`` was raising a ``ZeroDivisionError`` (:issue:`22519`)
 - Bug in :meth:`DataFrame.resample` when resampling ``NaT`` in ``TimeDeltaIndex`` (:issue:`13223`).
+- Bug in :meth:`DataFrameGroupBy._gotitem` where `observed` parameter was not passed properly, causing subsequent function calls that rely on `observed` to misbehave if called with ``observed=True`` (:issue:`23970`)
path: doc/source/whatsnew/v0.24.0.rst
text: @user1:
This message is too low level for the end user. Just mention that `DataFrame.groupby` with `observed=True` did not work correctly when aggregating a specified column.

diff_hunk: @@ -1694,3 +1694,15 @@ def test_groupby_agg_ohlc_non_first():
     result = df.groupby(pd.Grouper(freq='D')).agg(['sum', 'ohlc'])
 
     tm.assert_frame_equal(result, expected)
+
+
+def test_groupby_agg_observed_lost_err():
+    # GH-23970
+    df = pd.DataFrame({
path: pandas/tests/groupby/test_groupby.py
text: @user1:
You can just call this dataframe `expected` so you don't need to reassign `df`