Repository: pandas-dev/pandas

Issue:
Inconsistency, NaT included in result of groupby method first but not NaN
`NaT` is included in result of `groupby` method `first` while `NaN`. I am expecting that  first should skip both `NaN` and `NaT` and include the first value where `pandas.isnull` is False.
Demonstration of the inconsistency. (note that both `NaT` and `NaN` in the data frame are produced by `np.nan`, the difference is that the d_t column contains date values).

``` Python
import numpy as np
import pandas as pd
from datetime import datetime as dt

testFrame=DataFrame({'IX':['A','A'],'num':[np.nan,100],'d_t':[np.nan,dt.now()]})
```

Resulting data frame:

```
  IX                     d_t  num
0  A                     NaT  NaN
1  A 2015-07-15 22:47:10.635  100
```

Grouping this data frame on the `IX` column and executing the `first` method results in this data frame which shows the inconsistency between the `d_t` and `num` columns.

```
testFrame.groupby('IX').first()
```

Resulting dataframe:

```
        d_t  num
IX              
A       NaT  100
```

```
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 58 Stepping 9, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.2
nose: 1.3.4
Cython: 0.22
numpy: 1.9.2
scipy: 0.15.1
statsmodels: 0.6.1
IPython: 3.0.0
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.4.2
pytz: 2015.4
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.3
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.6.7
lxml: 3.4.2
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 0.9.9
pymysql: None
psycopg2: None
```



Discussion:
hmm, @sinhrks I thought this was fixed?

I remember some issues when `NaT` is in group key, but not aware the aggregation issure. `.min` might be affected also.

prob needs some adjustment for comparison vs iNaT in the first/last (though i thought it was there)

Confirmed `.min` also affected. @larvian PR is appreciated:)

```
testFrame.groupby('IX').min()
#    d_t  num
# IX         
# A  NaT  100
```

@sinhrks :) I see. Well, I have started to research GitHub and the Pandas source code a bit but right now I unfortunately don't have very much time available and waiting for me could require patience.
If I come to the point when I feel confident that I can contribute to the solution then I will do a PR. If someone else does it I am OK with that too :)


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1532,6 +1532,7 @@ def aggregate(self, values, how, axis=0):
 
         if is_datetime_or_timedelta_dtype(values.dtype):
             values = values.view('int64')
+            values[values == tslib.iNaT] = np.nan
             # GH 7754
             is_numeric = True
         elif is_bool_dtype(values.dtype):
@@ -2761,9 +2762,7 @@ def _cython_agg_blocks(self, how, numeric_only=True):
 
         for block in data.blocks:
 
-            values = block._try_operate(block.values)
-
-            result, _ = self.grouper.aggregate(values, how, axis=agg_axis)
+            result, _ = self.grouper.aggregate(block.values, how, axis=agg_axis)
 
             # see if we can cast the block back to the original dtype
             result = block._try_coerce_and_cast_result(result)


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -5413,6 +5413,24 @@ def test_func(x):
         expected = DataFrame()
         tm.assert_frame_equal(result, expected)
 
+    def test_first_last_max_min_on_time_data(self):
+        # GH 10295
+        # Verify that NaT is not in the result of max, min, first and last on
+        # Dataframe with datetime or timedelta values.
+        from datetime import timedelta as td
+        df_test=DataFrame({'dt':[nan,'2015-07-24 10:10','2015-07-25 11:11','2015-07-23 12:12',nan],
+                           'td':[nan,td(days=1),td(days=2),td(days=3),nan]})
+        df_test.dt=pd.to_datetime(df_test.dt)
+        df_test['group']='A'
+        df_ref=df_test[df_test.dt.notnull()]
+
+        grouped_test=df_test.groupby('group')
+        grouped_ref=df_ref.groupby('group')
+
+        assert_frame_equal(grouped_ref.max(),grouped_test.max())
+        assert_frame_equal(grouped_ref.min(),grouped_test.min())
+        assert_frame_equal(grouped_ref.first(),grouped_test.first())
+        assert_frame_equal(grouped_ref.last(),grouped_test.last())
 
 def assert_fp_equal(a, b):
     assert (np.abs(a - b) < 1e-12).all()