Repository: pandas-dev/pandas

Issue:
PERF: NaT groups cause wrong path in grouping
xref #10625

before this patch:

```
In [1]: from string import ascii_lowercase
In [2]: np.random.seed(2718281)
In [3]: n = 1 << 21
In [4]: dr = date_range('2015-08-30', periods=n // 10, freq='T')
In [5]: df = DataFrame({
   ...:         '1st':np.random.choice(list(ascii_lowercase), n),
   ...:         '2nd':np.random.randint(0, 5, n),
   ...:         '3rd':np.random.choice(dr, n)})

In [6]: df.loc[np.random.choice(n, n // 10), '3rd'] = np.nan
In [7]: gr = df.groupby(['1st', '2nd'])

In [8]: %timeit gr.count()
The slowest run took 21.22 times longer than the fastest. This could mean that an intermediate result is being cached 
1 loops, best of 3: 13.3 ms per loop

In [9]: %timeit gr.count()
100 loops, best of 3: 13.8 ms per loop

In [10]: pd.__version__
Out[10]: '0.16.2+521.g207efc2'
```

with this patch:

```
In [8]: %timeit gr.count()
1 loops, best of 3: 144 ms per loop

In [9]: %timeit gr.count()
10 loops, best of 3: 149 ms per loop

In [10]: pd.__version__
Out[10]: '0.16.2+522.g9c2d1a6'
```



Discussion:
cc @larvian 
cc @behzadnouri 

After #11013 this seems ok

is their a asv bench for this? (e.g. count on datetime64 with NaT)?

```
In [11]: %timeit gr.count()
100 loops, best of 3: 7.16 ms per loop

In [12]: pd.__version__
Out[12]: '0.16.2+599.g33530b3'
```

it is only ok for `count` since i removed the cython wrapper. it does still break other cythonized methods


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1523,8 +1523,6 @@ def aggregate(self, values, how, axis=0):
 
         if is_datetime_or_timedelta_dtype(values.dtype):
             values = values.view('int64')
-            values[values == tslib.iNaT] = np.nan
-            # GH 7754
             is_numeric = True
         elif is_bool_dtype(values.dtype):
             values = _algos.ensure_float64(values)
diff --git a/pandas/src/generate_code.py b/pandas/src/generate_code.py
--- a/pandas/src/generate_code.py
+++ b/pandas/src/generate_code.py
@@ -739,7 +739,7 @@ def group_last_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                 val = values[i, j]
 
                 # not nan
-                if val == val:
+                if val == val and val != %(nan_val)s:
                     nobs[lab, j] += 1
                     resx[lab, j] = val
 
@@ -785,7 +785,7 @@ def group_nth_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                 val = values[i, j]
 
                 # not nan
-                if val == val:
+                if val == val and val != %(nan_val)s:
                     nobs[lab, j] += 1
                     if nobs[lab, j] == rank:
                         resx[lab, j] = val
@@ -1013,7 +1013,7 @@ def group_max_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                     val = values[i, j]
 
                     # not nan
-                    if val == val:
+                    if val == val and val != %(nan_val)s:
                         nobs[lab, j] += 1
                         if val > maxx[lab, j]:
                             maxx[lab, j] = val
@@ -1027,7 +1027,7 @@ def group_max_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                 val = values[i, 0]
 
                 # not nan
-                if val == val:
+                if val == val and val != %(nan_val)s:
                     nobs[lab, 0] += 1
                     if val > maxx[lab, 0]:
                         maxx[lab, 0] = val
@@ -1076,7 +1076,8 @@ def group_min_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                     val = values[i, j]
 
                     # not nan
-                    if val == val:
+                    if val == val and val != %(nan_val)s:
+
                         nobs[lab, j] += 1
                         if val < minx[lab, j]:
                             minx[lab, j] = val
@@ -1090,7 +1091,7 @@ def group_min_%(name)s(ndarray[%(dest_type2)s, ndim=2] out,
                 val = values[i, 0]
 
                 # not nan
-                if val == val:
+                if val == val and val != %(nan_val)s:
                     nobs[lab, 0] += 1
                     if val < minx[lab, 0]:
                         minx[lab, 0] = val


Test patch: