Repository: pandas-dev/pandas
Pull request: BUG: groupby nunique changing values
- [ ] closes #31950 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

just reopening #31973 for now - this doesn't special-case NaT. Will think if there's a better way to do it, feel free to close / take over if it's wrong

Problem statement:
BUG: groupby-nunique modifies null values
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np
df = pd.DataFrame({"GROUP": 0, "VALUE": [1.0, np.nan]})
df.groupby("GROUP")["VALUE"].nunique()
print(df)
```
#### Problem description
Original dataframe is modified:
```
   GROUP         VALUE
0      0  1.000000e+00
1      0 -9.223372e+18
```

Issue seems to have been introduced in version 1.0.0, 0.25.3 works as expected.

#### Expected Output
Original dataframe should not be modified.

#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.87-linuxkit-aufs
machine          : x86_64
processor        :
byteorder        : little
LC_ALL           : en_US.UTF-8
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.1
numpy            : 1.17.2
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.2.3
setuptools       : 41.2.0
Cython           : 0.29.13
pytest           : 5.2.1
hypothesis       : 4.23.0
sphinx           : 1.7.9
blosc            : None
feather          : None
xlsxwriter       : 1.2.2
lxml.etree       : 4.4.1
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.8.2 (dt dec pq3 ext lo64)
jinja2           : 2.10.3
IPython          : 7.9.0
pandas_datareader: None
bs4              : None
bottleneck       : 1.2.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.13.0
pytables         : None
pytest           : 5.2.1
pyxlsb           : None
s3fs             : None
scipy            : 1.2.1
sqlalchemy       : 1.3.11
tables           : None
tabulate         : 0.8.5
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : 1.2.2
numba            : 0.45.1
</details>

Hints from issue discussion:
Thanks @thomas-reineking-jdas 
I think this was caused by https://github.com/pandas-dev/pandas/pull/27951. Seems the values get mutated here: https://github.com/pandas-dev/pandas/blob/master/pandas/core/groupby/generic.py#L596
@dsaxton yes, seems that you're right
I think the problem might stem from this:
```
>>> dtarr = np.array([pd.NaT, 1, 2, 3]); dtarr.sort(); dtarr                                                                                                                                                 
array([NaT, 1, 2, 3], dtype=object)                                                                                                                     

>>> dtarr = np.array([np.nan, 1, 2, 3]); dtarr.sort(); dtarr                                                                                                                                                 
array([ 1.,  2.,  3., nan])
```

Can push a fix which removes
```
        # GH 27951
        # temporary fix while we wait for NumPy bug 12629 to be fixed
        val[isna(val)] = np.datetime64("NaT")
```
but it might require pushing the minimum numpy version up, looking into it
cc @jbrockmendel @WillAyd

Commit message: remove file

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.2.rst b/doc/source/whatsnew/v1.0.2.rst
index 07afe60c9c22a..5f06188def4f2 100644
--- a/doc/source/whatsnew/v1.0.2.rst
+++ b/doc/source/whatsnew/v1.0.2.rst
@@ -19,6 +19,7 @@ Fixed regressions
 - Fixed regression in :meth:`Series.align` when ``other`` is a DataFrame and ``method`` is not None (:issue:`31785`)
 - Fixed regression in :meth:`pandas.core.groupby.RollingGroupby.apply` where the ``raw`` parameter was ignored (:issue:`31754`)
 - Fixed regression in :meth:`rolling(..).corr() <pandas.core.window.Rolling.corr>` when using a time offset (:issue:`31789`)
+- Fixed regression in :meth:`DataFrameGroupBy.nunique` which was modifying the original values if ``NaN`` values were present (:issue:`31950`)
 - Fixed regression where :func:`read_pickle` raised a ``UnicodeDecodeError`` when reading a py27 pickle with :class:`MultiIndex` column (:issue:`31988`).
 - Fixed regression in :class:`DataFrame` arithmetic operations with mis-matched columns (:issue:`31623`)
 - Fixed regression in :meth:`GroupBy.agg` calling a user-provided function an extra time on an empty input (:issue:`31760`)
diff --git a/pandas/core/groupby/generic.py b/pandas/core/groupby/generic.py
index 37b6429167646..61d3a3d5a4a77 100644
--- a/pandas/core/groupby/generic.py
+++ b/pandas/core/groupby/generic.py
@@ -591,20 +591,23 @@ def nunique(self, dropna: bool = True) -> Series:
 
         val = self.obj._internal_get_values()
 
-        # GH 27951
-        # temporary fix while we wait for NumPy bug 12629 to be fixed
-        val[isna(val)] = np.datetime64("NaT")
-
-        try:
-            sorter = np.lexsort((val, ids))
-        except TypeError:  # catches object dtypes
-            msg = f"val.dtype must be object, got {val.dtype}"
-            assert val.dtype == object, msg
+        def _object_sorter(val, ids):
             val, _ = algorithms.factorize(val, sort=False)
             sorter = np.lexsort((val, ids))
             _isna = lambda a: a == -1
+            return val, sorter, _isna
+
+        if isna(val).any():
+            val, sorter, _isna = _object_sorter(val, ids)
         else:
-            _isna = isna
+            try:
+                sorter = np.lexsort((val, ids))
+            except TypeError:  # catches object dtypes
+                msg = f"val.dtype must be object, got {val.dtype}"
+                assert val.dtype == object, msg
+                val, sorter, _isna = _object_sorter(val, ids)
+            else:
+                _isna = isna
 
         ids, val = ids[sorter], val[sorter]
 
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
index 6205dfb87bbd0..c402ca194648f 100644
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1017,6 +1017,7 @@ def test_frame_describe_unstacked_format():
 @pytest.mark.parametrize("dropna", [False, True])
 def test_series_groupby_nunique(n, m, sort, dropna):
     def check_nunique(df, keys, as_index=True):
+        original_df = df.copy()
         gr = df.groupby(keys, as_index=as_index, sort=sort)
         left = gr["julie"].nunique(dropna=dropna)
 
@@ -1026,6 +1027,7 @@ def check_nunique(df, keys, as_index=True):
             right = right.reset_index(drop=True)
 
         tm.assert_series_equal(left, right, check_names=False)
+        tm.assert_frame_equal(df, original_df)
 
     days = date_range("2015-08-23", periods=10)
 


Code review comments:
diff_hunk: @@ -591,20 +591,23 @@ def nunique(self, dropna: bool = True) -> Series:
 
         val = self.obj._internal_get_values()
 
-        # GH 27951
-        # temporary fix while we wait for NumPy bug 12629 to be fixed
-        val[isna(val)] = np.datetime64("NaT")
-
-        try:
-            sorter = np.lexsort((val, ids))
-        except TypeError:  # catches object dtypes
-            msg = f"val.dtype must be object, got {val.dtype}"
-            assert val.dtype == object, msg
+        def _object_sorter(val, ids):
path: pandas/core/groupby/generic.py
text: @user1:
let's just simplifiy this

you can factorize any pandas objects, so just

```
codes, _ = algorithms.factorize(val, sort=False)
sorter = np.lexsort((codes, ids))
mask = codes == -1
```
(I renamed val -> codes), but should just work

@author:
Thanks @user1 ! Have updated