Repository: pandas-dev/pandas

Issue:
"high" datetime conversion incorrect
Hi,

I have a situation where I have "far away" end dates (3 examples included below). Normal Python strptime conversion handles this correctly, but the to_datetime functionality does a Martin McFly and ends up in the 1800s. See this example that reproduces it:

```
# input
time_ser = pd.Series([np.nan, '2013-04-08 00:00:00.000', '2013-06-04 00:00:00.000', '2013-09-06 00:00:00.000', '2013-10-02 00:00:00.000', '2013-10-03 00:00:00.000', '2013-10-30 00:00:00.000', '2013-10-31 00:00:00.000', '2013-11-30 00:00:00.000', '2013-12-02 00:00:00.000', '2013-12-17 00:00:00.000', '2013-12-31 00:00:00.000', '2014-01-15 00:00:00.000', '2014-01-31 00:00:00.000', '2014-02-15 00:00:00.000', '2014-02-28 00:00:00.000', '2014-03-15 00:00:00.000', '2014-03-31 00:00:00.000', '2014-06-15 00:00:00.000', '2014-06-30 00:00:00.000', '2099-12-31 00:00:00.000', '2999-12-31 00:00:00.000', '9990-12-31 00:00:00.000', '9999-12-31 00:00:00.000'])

# see last 3 results
pd.to_datetime(time_ser, '%Y-%m-%d %H:%M:%S.%f')

# normal python
import math
for x in sorted(time_ser.unique()):
    if isinstance(x, basestring):
        print '{} converts to {}'.format( x, datetime.datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f'))

# This is the erroneous result:
21   1830-11-22 00:50:52.580896768
22   1807-03-30 05:56:08.066277376
23   1816-03-29 05:56:08.066277376
```

I'm running Pandas 0.12 on Ubuntu 12.04LTS with Python 2.7.3. I could not find this issue registered yet.

Kind regards,

Carst



Discussion:
try with master
should raise

And for the reason, see here: http://pandas.pydata.org/pandas-docs/dev/gotchas.html#minimum-and-maximum-timestamps

Thanks, just built and installed 0.13-dev, it does not give a raise a warning or error however, but returns the series as an object (still a lot better than 0.12!). try this in master:

```
time_ser = pd.Series([np.nan, '2013-04-08 00:00:00.000', '2013-06-04 00:00:00.000', '2013-09-06 00:00:00.000', '2013-10-02 00:00:00.000', '2013-10-03 00:00:00.000', '2013-10-30 00:00:00.000', '2013-10-31 00:00:00.000', '2013-11-30 00:00:00.000', '2013-12-02 00:00:00.000', '2013-12-17 00:00:00.000', '2013-12-31 00:00:00.000', '2014-01-15 00:00:00.000', '2014-01-31 00:00:00.000', '2014-02-15 00:00:00.000', '2014-02-28 00:00:00.000', '2014-03-15 00:00:00.000', '2014-03-31 00:00:00.000', '2014-06-15 00:00:00.000', '2014-06-30 00:00:00.000', '2099-12-31 00:00:00.000', '2999-12-31 00:00:00.000', '9990-12-31 00:00:00.000', '9999-12-31 00:00:00.000'])
pd.to_datetime(time_ser, '%Y-%m-%d %H:%M:%S.%f')[-3:]
pd.to_datetime(time_ser, '%Y-%m-%d %H:%M:%S.%f').dtype

time_ser = pd.Series([np.nan, '2013-04-08 00:00:00.000', '2013-06-04 00:00:00.000', '2013-09-06 00:00:00.000', '2013-10-02 00:00:00.000', '2013-10-03 00:00:00.000', '2013-10-30 00:00:00.000', '2013-10-31 00:00:00.000', '2013-11-30 00:00:00.000', '2013-12-02 00:00:00.000', '2013-12-17 00:00:00.000', '2013-12-31 00:00:00.000', '2014-01-15 00:00:00.000', '2014-01-31 00:00:00.000', '2014-02-15 00:00:00.000', '2014-02-28 00:00:00.000', '2014-03-15 00:00:00.000', '2014-03-31 00:00:00.000', '2014-06-15 00:00:00.000', '2014-06-30 00:00:00.000', '2099-12-31 00:00:00.000'])
pd.to_datetime(time_ser, '%Y-%m-%d %H:%M:%S.%f')
pd.to_datetime(time_ser, '%Y-%m-%d %H:%M:%S.%f').dtype
```

The second one (without out-of-range dates) will convert, the first one will not. (@joris: I will try to do the period index/frequency solution and see if that works out, also with HDF5 writing/reading)

sorry by default it will do that
you can pass errors='raise' to force it to fail
or coerce=True to make non valid dates be NaT

neither of your examples will be valid for HDF5 FYI

unless you use coerce=True

don't pass them as Series the. it will raise I believe
on e you make it a Series it will automatically be object 

with errors='raise' I indeed get a nice: pandas.tslib.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2999-12-31 00:00:00. :)

0.13 handles this nicely enough to be honest with the object return (0.12 got me rather confused) so this can be closed for me! thank you very much for your help

I'll think on how to handle this for my own specific problem. The most elegant way would be to have a integer-based "date" class that pandas and hdf5 just see as int (but that gives issues with nans though). 
I guess i'll go a bit down a dirty route: we made something to replace all  attribute / non-numeric columns with an integer-index columns that refer to separately saved (unique) value series. Those value series sometimes get pickled anyway by HDF5 because of unicodes, but it also saves a lot of space in general as most of my data often has repeating values in the attribute columns. So I guess I will have to save dates as objects in a standard format there and then later transform them using standard python functionality when needed.


Resolving patch:
diff --git a/pandas/tseries/tools.py b/pandas/tseries/tools.py
--- a/pandas/tseries/tools.py
+++ b/pandas/tseries/tools.py
@@ -112,7 +112,12 @@ def _convert_listlike(arg, box):
 
                 # fallback
                 if result is None:
-                    result = tslib.array_strptime(arg, format, coerce=coerce)
+                    try:
+                        result = tslib.array_strptime(arg, format, coerce=coerce)
+                    except (tslib.OutOfBoundsDatetime):
+                        if errors == 'raise':
+                            raise
+                        result = arg
             else:
                 result = tslib.array_to_datetime(arg, raise_=errors == 'raise',
                                                  utc=utc, dayfirst=dayfirst,


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -804,6 +804,31 @@ def test_to_datetime_default(self):
         xp = datetime(2001, 1, 1)
         self.assert_(rs, xp)
 
+
+    def test_to_datetime_mixed(self):
+
+        # 5863
+        # passing a string format with embedded np.nan
+
+        ts = Series([np.nan, '2013-04-08 00:00:00.000', '9999-12-31 00:00:00.000'])
+        expected = Series([NaT,Timestamp('20130408'),NaT])
+
+        result = to_datetime(ts, format='%Y-%m-%d %H:%M:%S.%f')
+        assert_series_equal(result, ts)
+
+        # raises if specified
+        self.assertRaises(pd.tslib.OutOfBoundsDatetime, to_datetime, ts, format='%Y-%m-%d %H:%M:%S.%f', errors='raise')
+
+        result = to_datetime(ts, format='%Y-%m-%d %H:%M:%S.%f',coerce=True)
+        expected = Series([NaT,Timestamp('20130408'),NaT])
+        assert_series_equal(result,expected)
+
+        # passing integers
+        ts = Series([np.nan, 20130408, '20130409'])
+        result = to_datetime(ts, format='%Y%m%d')
+        expected = Series([NaT,Timestamp('20130408'),Timestamp('20130409')])
+        assert_series_equal(result,expected)
+
     def test_dayfirst(self):
 
         # GH 3341