Repository: pandas-dev/pandas
Pull request: BUG: Avoid ambiguous condition in GroupBy.first / last
- [x] closes #32123
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This comparison `val == val` happens in a lot of these groupby operations but only seems to raise here in the presence of `NA`. Are we just always / mostly converting to numpy beforehand in the other cases?

Problem statement:
BUG: GroupBy.first fails with pd.NA on Series with object dtype
#### Code Sample

```python
pd.DataFrame({'x': [1, 1, 2, 2], 'y': [1, 2, 3, pd.NA]}).groupby('x').first()
# *** TypeError: boolean value of NA is ambiguous

pd.DataFrame({'x': [1, 1, 2, 2], 'y': [1, 2, 3, np.nan]}).groupby('x').first()
#     y
# x     
# 1  1.0
# 2  3.0

pd.DataFrame({'x': [1, 1, 2, 2], 'y': [1, 2, 3, pd.NA]}).astype('Int64').groupby('x').first()
#    y
# x   
# 1  1
# 2  3
```
#### Problem description

Applying the `GroupBy.first` aggregation to a `object` dtype column that contains a `pd.NA` causes the method to fail with an exception: `TypeError: boolean value of NA is ambiguous`. Method works fine when using `np.nan` and also works as expected when the column is first converted to an `Int64` dtype column.

#### Expected Output

```
   y
x   
1  1
2  3
```
#### Output of ``pd.show_versions()``

<details>

```
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-76-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : pt_PT.UTF-8

pandas           : 1.0.1
numpy            : 1.17.4
pytz             : 2019.3
dateutil         : 2.8.0
pip              : 20.0.2
setuptools       : 42.0.1
Cython           : None
pytest           : 5.2.3
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.3 (dt dec pq3 ext lo64)
jinja2           : 2.10.3
IPython          : 7.10.1
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.2.3
pyxlsb           : None
s3fs             : None
scipy            : 1.3.2
sqlalchemy       : 1.3.10
tables           : 3.6.1
tabulate         : 0.8.6
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : 0.46.0
```
</details>

Hints from issue discussion:

Commit message: Update whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.2.rst b/doc/source/whatsnew/v1.0.2.rst
index c9031ac1ae9fe..f09107ea3a08c 100644
--- a/doc/source/whatsnew/v1.0.2.rst
+++ b/doc/source/whatsnew/v1.0.2.rst
@@ -45,6 +45,7 @@ Bug fixes
 
 - Fix bug in :meth:`DataFrame.convert_dtypes` for columns that were already using the ``"string"`` dtype (:issue:`31731`).
 - Fixed bug in setting values using a slice indexer with string dtype (:issue:`31772`)
+- Fixed bug where :meth:`GroupBy.first` and :meth:`GroupBy.last` would raise a ``TypeError`` when groups contained ``pd.NA`` in a column of object dtype (:issue:`32123`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/_libs/groupby.pyx b/pandas/_libs/groupby.pyx
index edc44f1c94589..2dfd6b32b636a 100644
--- a/pandas/_libs/groupby.pyx
+++ b/pandas/_libs/groupby.pyx
@@ -22,6 +22,8 @@ from pandas._libs.algos cimport (swap, TiebreakEnumType, TIEBREAK_AVERAGE,
 from pandas._libs.algos import (take_2d_axis1_float64_float64,
                                 groupsort_indexer, tiebreakers)
 
+from pandas._libs.missing import NA
+
 cdef int64_t NPY_NAT = get_nat()
 _int64_max = np.iinfo(np.int64).max
 
@@ -887,7 +889,7 @@ def group_last(rank_t[:, :] out,
             for j in range(K):
                 val = values[i, j]
 
-                if val == val:
+                if (val is not NA) and (val == val):
                     # NB: use _treat_as_na here once
                     #  conditional-nogil is available.
                     nobs[lab, j] += 1
@@ -976,7 +978,7 @@ def group_nth(rank_t[:, :] out,
             for j in range(K):
                 val = values[i, j]
 
-                if val == val:
+                if (val is not NA) and (val == val):
                     # NB: use _treat_as_na here once
                     #  conditional-nogil is available.
                     nobs[lab, j] += 1
diff --git a/pandas/tests/groupby/test_nth.py b/pandas/tests/groupby/test_nth.py
index 0f850f2e94581..0b13d2c298f1d 100644
--- a/pandas/tests/groupby/test_nth.py
+++ b/pandas/tests/groupby/test_nth.py
@@ -530,3 +530,20 @@ def test_nth_nan_in_grouper(dropna):
     )
 
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize("method", ["first", "last"])
+def test_first_last_with_na_object(method):
+    # https://github.com/pandas-dev/pandas/issues/32123
+    groups = pd.DataFrame({"a": [1, 1, 2, 2], "b": [1, 2, 3, pd.NA]}).groupby("a")
+    result = getattr(groups, method)()
+
+    if method == "first":
+        values = {"b": [1, 3]}
+    else:
+        values = {"b": [2, 3]}
+
+    idx = pd.Index([1, 2], name="a")
+    expected = pd.DataFrame(values, index=idx)
+
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -530,3 +530,20 @@ def test_nth_nan_in_grouper(dropna):
     )
 
     tm.assert_frame_equal(result, expected)
+
+
+@pytest.mark.parametrize("method", ["first", "last"])
+def test_first_last_with_na_object(method):
+    # https://github.com/pandas-dev/pandas/issues/32123
+    groups = pd.DataFrame({"a": [1, 1, 2, 2], "b": [1, 2, 3, pd.NA]}).groupby("a")
path: pandas/tests/groupby/test_nth.py
text: @user1:
Should also use nulls_fixture here after #31799

@user1:
Can you merge master? This should be available now

diff_hunk: @@ -887,7 +889,7 @@ def group_last(rank_t[:, :] out,
             for j in range(K):
                 val = values[i, j]
 
-                if val == val:
+                if (val is not NA) and (val == val):
path: pandas/_libs/groupby.pyx
text: @user1:
I think can use the `checknull` function from missing instead

@author:
Maybe just `not checknull(val)` for the whole condition? I think there was an issue with `checknull` not catching `decimal.Decimal("nan")` but I could try to add that here