Repository: pandas-dev/pandas

Issue:
powerpc: Test*Parser* fail with AssertionError: 2262-04-10T00:12:45.292707840+0000 != 2262-04-10T00:12:45.292707840+0000
Detected by Debian build farm
https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=powerpc&ver=0.12.0~rc1%2Bgit112-gb79996c-1&stamp=1374189622&file=log

```
======================================================================
FAIL: test_nat_parse (pandas.io.tests.test_parsers.TestCParserHighMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_parsers.py", line 607, in test_nat_parse
    tm.assert_frame_equal(result,df)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 238, in assert_frame_equal
    check_less_precise=check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 197, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 141, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 172, in assert_almost_equal
    assert a == b, "%s != %s" % (a, b)
AssertionError: 2262-04-10T00:12:45.292707840+0000 != 2262-04-10T00:12:45.292707840+0000

======================================================================
FAIL: test_nat_parse (pandas.io.tests.test_parsers.TestCParserLowMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_parsers.py", line 607, in test_nat_parse
    tm.assert_frame_equal(result,df)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 238, in assert_frame_equal
    check_less_precise=check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 197, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 141, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 172, in assert_almost_equal
    assert a == b, "%s != %s" % (a, b)
AssertionError: 2262-04-10T00:12:45.292707840+0000 != 2262-04-10T00:12:45.292707840+0000

======================================================================
FAIL: test_nat_parse (pandas.io.tests.test_parsers.TestPythonParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/io/tests/test_parsers.py", line 607, in test_nat_parse
    tm.assert_frame_equal(result,df)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 238, in assert_frame_equal
    check_less_precise=check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 197, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 141, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/«BUILDDIR»/pandas-0.12.0~rc1+git112-gb79996c/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 172, in assert_almost_equal
    assert a == b, "%s != %s" % (a, b)
AssertionError: 2262-04-10T00:12:45.292707840+0000 != 2262-04-10T00:12:45.292707840+0000
```

good news -- those seems to be the only failures on powerpc (for python 2.7 at least, python 3 did not have chance to get any testing yet due to 2.7 failing)



Discussion:
is this current master?

@jtratner looks like 2 behind (b79996c), but last 2 are merge + y-p doc commit so should be ok

FWIW on s390 the same test fails too
https://buildd.debian.org/status/fetch.php?pkg=pandas&arch=s390x&ver=0.12.0~rc1%2Bgit112-gb79996c-1&stamp=1374197022&file=log
but then the whole build f.cks up somehow upon test_non_unique_loc_memory_error (yet to troubleshoot) and gets killed... so no details on failures and errors are provided

this is again a byteorder issue....

@yarikoptic 

must be something weird going on....on the sparc machine....I can't even build master...it just hangs in the compiler (I turned off quiet)....and goes most of the way thru, but been spinning the cpu for a while (and not out of memory 10% being used)...its the compile step for algos.o....(its pretty large)...maybe that's an issue?
on python 2.7 stock build after I did `scroot`

any ideas?

well -- that box is quite antique by now (somewhere from 2000 or around), so a bit of extra patience would help ;) I just usually leave  it building and come back to it later.  also you might like to make use of installed ccache

I see that it now runs python3.2 for you - so I guess there was progress ;)

ok...will let it build (though 20 min still seems very odd to just compile a file)....

So the issue on that wrap around problem is happening
....let me see if I can fix this

```
(Pdb) p pd.NaT.value
-9223372036854775808L
(Pdb) p df['B'].iloc[3]
Timestamp('2262-04-11 23:47:16.854775807', tz=None)
(Pdb) p df['B'].iloc[3].value
9223372036854775807L
```

these are supposed to be the same


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -42,6 +42,7 @@ class AmbiguousIndexError(PandasError, KeyError):
 _NS_DTYPE = np.dtype('M8[ns]')
 _TD_DTYPE = np.dtype('m8[ns]')
 _INT64_DTYPE = np.dtype(np.int64)
+_DATELIKE_DTYPES = set([ np.dtype(t) for t in ['M8[ns]','m8[ns]'] ])
 
 def isnull(obj):
     """Detect missing values (NaN in numeric arrays, None/NaN in object arrays)
@@ -718,6 +719,12 @@ def _infer_dtype_from_scalar(val):
     return dtype, val
 
 
+def _maybe_cast_scalar(dtype, value):
+    """ if we a scalar value and are casting to a dtype that needs nan -> NaT conversion """
+    if np.isscalar(value) and dtype in _DATELIKE_DTYPES and isnull(value):
+        return tslib.iNaT
+    return value
+
 def _maybe_promote(dtype, fill_value=np.nan):
 
     # if we passed an array here, determine the fill value by dtype
@@ -789,6 +796,7 @@ def _maybe_upcast_putmask(result, mask, other, dtype=None, change=None):
 
     if mask.any():
 
+        other = _maybe_cast_scalar(result.dtype, other)
         def changeit():
 
             # try to directly set by expanding our array to full
@@ -851,6 +859,7 @@ def _maybe_upcast_indexer(result, indexer, other, dtype=None):
         return the result and a changed flag
         """
 
+    other = _maybe_cast_scalar(result.dtype, other)
     original_dtype = result.dtype
     def changeit():
         # our type is wrong here, need to upcast


Test patch:
diff --git a/pandas/tests/test_common.py b/pandas/tests/test_common.py
--- a/pandas/tests/test_common.py
+++ b/pandas/tests/test_common.py
@@ -5,7 +5,7 @@
 import nose
 import unittest
 
-from pandas import Series, DataFrame, date_range, DatetimeIndex
+from pandas import Series, DataFrame, date_range, DatetimeIndex, Timestamp
 from pandas.core.common import notnull, isnull
 import pandas.core.common as com
 import pandas.util.testing as tm
@@ -117,6 +117,24 @@ def test_datetimeindex_from_empty_datetime64_array():
         idx = DatetimeIndex(np.array([], dtype='datetime64[%s]' % unit))
         assert(len(idx) == 0)
 
+def test_nan_to_nat_conversions():
+
+    df = DataFrame(dict({
+        'A' : np.asarray(range(10),dtype='float64'),
+        'B' : Timestamp('20010101') }))
+    df.iloc[3:6,:] = np.nan
+    result = df.loc[4,'B'].value
+    assert(result == iNaT)
+
+    values = df['B'].values
+    result, changed = com._maybe_upcast_indexer(values,tuple([slice(8,9)]),np.nan)
+    assert(isnull(result[8]))
+
+    # numpy < 1.7.0 is wrong
+    from distutils.version import LooseVersion
+    if LooseVersion(np.__version__) >= '1.7.0':
+        assert(result[8] == np.datetime64('NaT'))
+
 def test_any_none():
     assert(com._any_none(1, 2, 3, None))
     assert(not com._any_none(1, 2, 3, 4))