Repository: pandas-dev/pandas
Pull request: TST: Iterrows making incorrect assumptions about datetime
- [x] closes #12222
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ n/a] whatsnew entry


Problem statement:
Iterrows making incorrect assumptions about datetime
This is a very obscure bug in a corner-case, but it's still a bug which I ran into. It's easiest to understand by seeing it in action. I'm running **pandas version 0.17.1**.

```
>>> import pandas as pd
>>> import datetime
>>> df = pd.DataFrame({'a': [datetime.datetime(2015,1,1)], 'b': [None], 'c': [None], 'd': [''], 'e': [[]], 'f': [set()], 'g': [{}]})
>>> df
           a     b     c d   e   f   g
0 2015-01-01  None  None    []  {}  {}
```

The key to causing the bug is a row which starts with a datetime obj followed by at least two None's. Following that is any number of empty strings, empty lists, empty sets, or empty dicts. Indexing into the DataFrame by row works fine.

```
>>>> df.iloc[0]
a    2015-01-01 00:00:00
b                   None
c                   None
d                       
e                     []
f                     {}
g                     {}
Name: 0, dtype: object
```

Iterating over the rows using iterrows causes a weird behavior of converting to a datetime dtype with NaT's. **This is the bug:**

```
>>> df.iterrows().next()
(0, a   2015-01-01
 b          NaT
 c          NaT
 d          NaT
 e          NaT
 f          NaT
 g          NaT
 Name: 0, dtype: datetime64[ns])
```

The following properties **avoid the issue** entirely:
1. Having less than two None's following the datetime.
2. Having any of the items after the two None's be "truthy".
3. Including either a zero or a False in the items after the None's.

Hints from issue discussion:
This is probably working as intended. `.iterrows` returns a Series which must have a single dtype. Pandas will try to cast the items in a row to as precise a dtype as it can so all the Falsey things get cast as `NaT`s. If you want to preserve the items try `DataFrame.itertuples()` which will return namedtuples.

Just one thing though, I don't see

> Having less than two None's following the datetime

not causing it to cast to datetimes:

``` python
In [39]: next(df[['a', 'b']].iterrows())
Out[39]:
(0, a   2015-01-01
 b          NaT
 Name: 0, dtype: datetime64[ns])
```

Can you check on that?

This shows the behavior I was noting:

```
In [39]: df[['a', 'e']].iterrows().next()
Out[39]: 
(0, a    2015-01-01 00:00:00
 e                     []
 Name: 0, dtype: object)

In [40]: df[['a', 'b', 'e']].iterrows().next()
Out[40]: 
(0, a    2015-01-01 00:00:00
 b                   None
 e                     []
 Name: 0, dtype: object)

In [41]: df[['a', 'b', 'c', 'e']].iterrows().next()
Out[41]: 
(0, a   2015-01-01
 b          NaT
 c          NaT
 e          NaT
 Name: 0, dtype: datetime64[ns])
```

Having only `None`s and empty strings always gives `NaT`s.

```
In [43]: df[['a', 'b', 'd', 'c', 'c', 'd', 'b']].iterrows().next()
Out[43]: 
(0, a   2015-01-01
 b          NaT
 d          NaT
 c          NaT
 c          NaT
 d          NaT
 b          NaT
 Name: 0, dtype: datetime64[ns])
```

But including a single `False` or zero in the row avoids the `NaT` conversion, even though those should also be considered "falsy".

```
In [45]: df['h'] = [False]

In [46]: df.iterrows().next()
Out[46]: 
(0, a    2015-01-01 00:00:00
 b                   None
 c                   None
 d                       
 e                     []
 f                     {}
 g                     {}
 h                  False
 Name: 0, dtype: object)

In [47]: df['h'] = [0]

In [48]: df.iterrows().next()
Out[48]: 
(0, a    2015-01-01 00:00:00
 b                   None
 c                   None
 d                       
 e                     []
 f                     {}
 g                     {}
 h                      0
 Name: 0, dtype: object)
```

so the series inferer is a bit aggressive here.

```
In [18]: v
Out[18]: array([Timestamp('2015-01-01 00:00:00'), None, None, []], dtype=object)

In [19]: Series(v)
Out[19]: 
0   2015-01-01
1          NaT
2          NaT
3          NaT
dtype: datetime64[ns]
```

This is called and should not treat as a datetimeliek if this is the case.

```
In [21]: pd.lib.infer_dtype(v)
Out[21]: 'mixed'
```

pull-requests welcome

this is fixed in master. now only needs a test to close issue.

```
>>> import pandas as pd
>>> pd.__version__
'0.25.0.dev0+355.g4814a2875'
>>> import datetime
>>> df = pd.DataFrame({'a': [datetime.datetime(2015,1,1)], 'b': [None], 'c': [None], 'd': [''], 'e': [[]],
 'f': [set()], 'g': [{}]})
>>> next(df.iterrows())
(0, a    2015-01-01 00:00:00
b                   None
c                   None
d
e                     []
f                     {}
g                     {}
Name: 0, dtype: object)
>>> df.values[0]
array([Timestamp('2015-01-01 00:00:00'), None, None, '', list([]), set(),
       {}], dtype=object)
>>> pd.Series(df.values[0])
0    2015-01-01 00:00:00
1                   None
2                   None
3
4                     []
5                     {}
6                     {}
dtype: object
>>>
```

Commit message: TST: Iterrows making incorrect assumptions about datetime

Patch to review:
diff --git a/pandas/tests/frame/test_api.py b/pandas/tests/frame/test_api.py
index 8013ddfeb38f9..808f2000b24fe 100644
--- a/pandas/tests/frame/test_api.py
+++ b/pandas/tests/frame/test_api.py
@@ -1,4 +1,5 @@
 from copy import deepcopy
+import datetime
 import pydoc
 
 import numpy as np
@@ -224,6 +225,16 @@ def test_iterrows_iso8601(self):
             exp = s.loc[k]
             self._assert_series_equal(v, exp)
 
+    def test_iterrows_corner(self):
+        # gh-12222
+        df = DataFrame(
+            {'a': [datetime.datetime(2015, 1, 1)], 'b': [None], 'c': [None],
+             'd': [''], 'e': [[]], 'f': [set()], 'g': [{}]})
+        expected = df.iloc[0]
+        _, result = next(df.iterrows())
+        assert result.dtype == np.object_
+        tm.assert_series_equal(result, expected)
+
     def test_itertuples(self, float_frame):
         for i, tup in enumerate(float_frame.itertuples()):
             s = self.klass._constructor_sliced(tup[1:])


Code review comments:
diff_hunk: @@ -224,6 +225,16 @@ def test_iterrows_iso8601(self):
             exp = s.loc[k]
             self._assert_series_equal(v, exp)
 
+    def test_iterrows_corner(self):
+        # gh-12222
+        df = DataFrame(
+            {'a': [datetime.datetime(2015, 1, 1)], 'b': [None], 'c': [None],
+             'd': [''], 'e': [[]], 'f': [set()], 'g': [{}]})
+        expected = df.iloc[0]
+        _, result = next(df.iterrows())
+        assert result.dtype == np.object_
path: pandas/tests/frame/test_api.py
text: @user1:
We already compare `dtype` in `assert_series_equal`, no?

@author:
since the expected is not explicit, the dtype check is for added assurance.  so the dtype check could have easily been applied to the expected instead. and if the expected was a fixture it would error instead of fail, indicating a bad test.

@user1:
Hmm...perhaps we should construct the expected `Series` explicitly instead of relying on the result of `df.iloc[0]`.

@author:
probably a good idea, i'll change it.