Repository: pandas-dev/pandas

Issue:
groupby nunique() with dates vs datetimes in presence of NaTs
#### Code Sample

```python

import numpy as np
import pandas as pd
import datetime

df = pd.DataFrame({'a': ['x','x','x','x','x'], 'b': [datetime.date(2019,1,1), pd.NaT, datetime.date(2019,1,1), pd.NaT, datetime.date(2019,1,1)]})
print(df)

# unique() returns only 1 value (plus the NaT) as expected
print('unique() with dates')
print(df.groupby(['a'])['b'].unique())

# nunique() returns 3 instead of the expected 1 (NaTs are dropped). Seems to be due to the NaTs
print('nunique() with dates')
print(df.groupby(['a'])['b'].nunique())

# convert date column to datetime
df['b'] = df.b.astype(np.datetime64)

# with datetime, unique() still returns 1 value (plus the NaT) as expected
print('unique() with datetimes')
print(df.groupby(['a'])['b'].unique())

# with datetime, nunique() returns 1 as expected
print('nunique() with datetimes')
print(df.groupby(['a'])['b'].nunique())

```
#### Problem description
`Series.groupby().nunique()` seems to have a different behavior with dates vs datetimes when NaTs are in the Series. The behavior with `date` seems to return more distinct values than there are.
This is not the same issue as https://github.com/pandas-dev/pandas/issues/14149

#### Expected Output
For nunique() with dates the expected output is:
```
a
x    1
```
#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.18.0-25-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.7.0
pandas_datareader: None
bs4              : 4.8.0
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.6.9
odfpy            : None
openpyxl         : 2.6.2
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.6
tables           : 3.5.2
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.1.8

</details>



Discussion:
Thanks for the report. Interested in debugging where things are going wrong?
I think the problem is that when `numpy` tries to sort
```
array([datetime.date(2019, 1, 1), NaT, datetime.date(2019, 1, 1), NaT,
       datetime.date(2019, 1, 1)], dtype=object)
```
it returns
```
array([datetime.date(2019, 1, 1), NaT, datetime.date(2019, 1, 1), NaT,
       datetime.date(2019, 1, 1)], dtype=object)
```

In `pandas.core.groupby.generic.py` it says
>         # group boundaries are where group ids change
>         # unique observations are where sorted values change

On the other hand,
```
array(['2019-01-01T00:00:00.000000000',                           'NaT',
       '2019-01-01T00:00:00.000000000',                           'NaT',
       '2019-01-01T00:00:00.000000000'], dtype='datetime64[ns]')
```
gets sorted as
```
array([                          'NaT',                           'NaT',
       '2019-01-01T00:00:00.000000000', '2019-01-01T00:00:00.000000000',
       '2019-01-01T00:00:00.000000000'], dtype='datetime64[ns]')
```
One solution (see #27951 ) could be to drop the null values _before_ letting numpy do the sorting?

Resolving patch:
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -1147,6 +1147,10 @@ def nunique(self, dropna=True):
 
         val = self.obj._internal_get_values()
 
+        # GH 27951
+        # temporary fix while we wait for NumPy bug 12629 to be fixed
+        val[isna(val)] = np.datetime64("NaT")
+
         try:
             sorter = np.lexsort((val, ids))
         except TypeError:  # catches object dtypes


Test patch:
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1,4 +1,5 @@
 import builtins
+import datetime as dt
 from io import StringIO
 from itertools import product
 from string import ascii_lowercase
@@ -9,7 +10,16 @@
 from pandas.errors import UnsupportedFunctionCall
 
 import pandas as pd
-from pandas import DataFrame, Index, MultiIndex, Series, Timestamp, date_range, isna
+from pandas import (
+    DataFrame,
+    Index,
+    MultiIndex,
+    NaT,
+    Series,
+    Timestamp,
+    date_range,
+    isna,
+)
 import pandas.core.nanops as nanops
 from pandas.util import _test_decorators as td, testing as tm
 
@@ -1015,6 +1025,42 @@ def test_nunique_with_timegrouper():
     tm.assert_series_equal(result, expected)
 
 
+@pytest.mark.parametrize(
+    "key, data, dropna, expected",
+    [
+        (
+            ["x", "x", "x"],
+            [Timestamp("2019-01-01"), NaT, Timestamp("2019-01-01")],
+            True,
+            Series([1], index=pd.Index(["x"], name="key"), name="data"),
+        ),
+        (
+            ["x", "x", "x"],
+            [dt.date(2019, 1, 1), NaT, dt.date(2019, 1, 1)],
+            True,
+            Series([1], index=pd.Index(["x"], name="key"), name="data"),
+        ),
+        (
+            ["x", "x", "x", "y", "y"],
+            [dt.date(2019, 1, 1), NaT, dt.date(2019, 1, 1), NaT, dt.date(2019, 1, 1)],
+            False,
+            Series([2, 2], index=pd.Index(["x", "y"], name="key"), name="data"),
+        ),
+        (
+            ["x", "x", "x", "x", "y"],
+            [dt.date(2019, 1, 1), NaT, dt.date(2019, 1, 1), NaT, dt.date(2019, 1, 1)],
+            False,
+            Series([2, 1], index=pd.Index(["x", "y"], name="key"), name="data"),
+        ),
+    ],
+)
+def test_nunique_with_NaT(key, data, dropna, expected):
+    # GH 27951
+    df = pd.DataFrame({"key": key, "data": data})
+    result = df.groupby(["key"])["data"].nunique(dropna=dropna)
+    tm.assert_series_equal(result, expected)
+
+
 def test_nunique_preserves_column_level_names():
     # GH 23222
     test = pd.DataFrame([1, 2, 2], columns=pd.Index(["A"], name="level_0"))