Repository: pandas-dev/pandas

Issue:
Timestamp() is wrong if np.datetime64 < 1678-01-01
The following should work, but since it doesn't should raise an error...

In [116]: pd.Timestamp(np.datetime64('1677-01-01'),'D')
Out[116]: Timestamp('2261-07-22 23:34:33.709551616', tz=None)

In [117]: pd.Timestamp(np.datetime64('1678-01-01'),'D')
Out[117]: Timestamp('1678-01-01 00:00:00', tz=None)

In [121]: pd.**version**
Out[121]: '0.11.1.dev-06cd915'

In [125]: sys.version
Out[125]: '3.3.2 (default, May 21 2013, 15:40:45) \n[GCC 4.8.0 20130502 (prerelease)]'



Discussion:
I'll try to tackle this since I already mucked around with Timestamp.

great!

The heart of the bug was not doing a _check_dts_bounds() in _get_datetime64_nanos when converting from a datetime64 that wasn't in [ns] units.

But fixing this lead to another issue: the way _check_dts_bounds() worked previously, it did a broad check on the validity of a pandas_datetimestruct.year, and then assuming that triggered, it did a more fine grained check using the integer value of the given datetime64.  However, if you have a datetime64 in another unit, the integer value isn't useful for comparison (because by definition it will map into a valid datetime64 somewhere in the [ns] range), so you don't really know if it is valid or not.

So if you have a datetime64 that might be any unit, the only way to verify if it is in bounds is by just looking at the pandas_datetimestruct and just seeing if the values are in bounds or not.  So I changed _check_dts_bounds() to do bounds checking this way.

can you add some tests with `to_datetime` (which ultimtately call s `array_to_datetime`) with a mix of `np.datetime64` that have some valid, some invalid dtypes? (like you are doing with Timestamp, but with an array of them); the code paths are different, but should yield the same results

e.g.

```
result1 = np.array([ Timestamp(v) for v in test_values ])
result2 = pd.to_datetime(test_values,box=False)
```

assume several cases, test_values are valid (with various types of dtypes), and another case of an invalid value

thxs..

this is great stuff getting all of these bugs out of the way!

Yeah this is a good point, I need to do a little more work here.

When you have a situation like:

```
In [10]: pd.to_datetime(['1000-01-01', '1000-01-02'])
Out[10]: array(['1000-01-01', '1000-01-02'], dtype=object)
```

You give it strings, it gives back strings because it can't convert them to valid datetime64[ns]-es because they are out of bounds.

Currently, if you give it datetime64s it can't deal with, it just maps them randomly into the datetime64[ns] range:

```
In [8]: pd.to_datetime([np.datetime64('1000-01-01')], box=False)
Out[8]: array(['2169-02-08T15:09:07.419103232-0800'], dtype='datetime64[ns]')
```

This is bad, but at least it has 2 nice properties: it doesn't throw errors and it always returns datetime64[ns]s.

So I think there are a few potential solutions.  First of all, in all cases if coerce=True, we just convert the out-of-bounds dt64s into NaTs, so we'll ignore that case:

a) `pd.to_datetime([np.datetime64('1000-01-01')], box=False)` throws an error, pandas can't deal with it

b) `pd.to_datetime([np.datetime64('1000-01-01')], box=False)` returns back an array of datetime64[whatever]s.  This will probably lead to weird situations elsewhere because I think a lot of code assumes it is dealing with datetime64[ns]es.  I just did a quick test and if pd.to_datetime() returns the datetime64[whatevers]s right back, some other aspect of the code maps them to datetime64[ns]es.  I didn't investigate too deeply because I assume it would be a big deal, if not pandas would just support arbitrary units and it doesn't.

c) `pd.to_datetime([np.datetime64('1000-01-01')], box=False)` returns back an array of strings `['1000-01-01']`.  IE, if we get an out-of-bounds datetime64, just convert it to a string and return that.

I like solution (c).  Even though it is kind of surprising, it generally does the right thing and will make sure developers don't have to worry about non datetime64[ns]-es cropped up all over the place in these edge cases.  Then in the future if pandas ever supports datetime64s of various units, it will be easy delete the code which converts the dt64s to strings and just return them.

Though I'm amenable to any of the above solutions or a different solution, no hugely strong feelings.

Also if this is confusing or I am missing something obvious, please let me know.  I'm pretty new to this section of the code.

@danbirken 

c) is the current/right way, but with one additional check

```
In [6]: x = np.datetime64('1000-01-01')

In [7]: x
Out[7]: numpy.datetime64('1000-01-01')

In [8]: x.dtype
Out[8]: dtype('<M8[D]')
```

There is a bug lurking that allows a `np.datetime64` < ns range in on conversion (either the bounds are not checked or the conversion is wrong)

This should raise be `NaT` if `coerce=True,` or raise the bounds error (which ALWAYS raises)

better to let the user know that this is not an acceptable value

side issue is whether their are raises if `coerce=True` and the date is out of bounds (I don't know what it does currently, my thinking is that if you pass `coerce=True`then just make it `NaT`even if out-of-bounds)

@danbirken another related issue: https://github.com/pydata/pandas/issues/3944

related is #4066 as well

Alright, I'm a little confused as to the exact interface you are looking for and while digging through the code and all of these related issues i've found a few more potential fixes, so here are some clarification statements.  Let me know if any of these are wrong:

1) `pd.to_datetime(value) == pd.to_datetime([value])[0]` should be true in all cases (there are some cases right now where this isn't true)

2) `pd.to_datetime('1000-01-01', errors='ignore', box=False)` should return `'1000-01-01'`

3) `pd.to_datetime('1000-01-01', errors='raise', box=False)` should throw ValueError

4) `pd.to_datetime(np.datetime64('1000-01-01'), errors='ignore', box=False)` should return `'1000-01-01'` --- ie, datetime64 converted to a string because it can't be represented as a datetime64[ns]

5) `pd.to_datetime(np.datetime64('1000-01-01'), errors='raise', box=False)` should throw ValueError

6) `pd.to_datetime('1000-01-01', coerce=True)` should return a NaT

7) `pd.to_datetime(np.datetime64('1000-01-01'), coerce=True)` should return a NaT

can you explain 4 again?

when is 1) not true?

all others look good

For 1)

```
In [8]: pd.to_datetime([np.datetime64('2000-01-01')], box=False)[0].dtype
Out[8]: dtype('<M8[ns]')

In [9]: pd.to_datetime(np.datetime64('2000-01-01'), box=False).dtype
Out[9]: dtype('<M8[D]')
```

IE pd.to_datetime() will not fix the datetime64[] range type if it isn't wrapped in an array.  It is a simple fix, I'm just double checking the behavior.

For 4)

It is basically this case:

```
import pandas as pd
import numpy as np

ix = [np.datetime64('1000-01-01', 'D'), np.datetime64('1000-01-02', 'D')]
df = pd.DataFrame({'col1': [1, 2]}, index=ix)
print df
```

This is the current behavior:

```
                            col1
2169-02-08 23:09:07.419103     1
2169-02-09 23:09:07.419103     2
```

as far as I see, this could:

a) throw an error

b) don't change it, let the current behavior continue

c) convert the datetime64s to strings (or maybe datetime.datetimes?), so you get this output:

```
            col1
1000-01-01     1
1000-01-02     2
```

d) something else

ok....I understand 1); thought it was fixed, but good catch

for 4)

I think if you detect this (an 'invalid' `np.datetime64`):

if coerce is True, then obviously just `NaT` them
if errors='raise', then raise

otherwise, return convert to an object array of datetimes (these won't be further converted anywhere)

np.datetime64('1000-01-01').item() is the datetime

```
In [52]: ix2 = np.array([datetime.date(1000,1,1),datetime.date(1000,1,1)])

In [53]: df = pd.DataFrame({'col1': [1, 2]}, index=ix2)

In [54]: df
Out[54]: 
            col1
1000-01-01     1
1000-01-01     2

In [55]: df.index
Out[55]: Index([1000-01-01, 1000-01-01], dtype=object)

In [56]: df.index[0]
Out[56]: datetime.date(1000, 1, 1)
```

I am not 100% in love with this, becuase its 'probably' a mistake

but ok for now

@danbirken how's this coming?

I'm 90% done but I'm currently in Munich for Octoberfest which is slowing
my progress.  I'll have an updated patch within a couple days.  Sorry about
the delay.

-Dan

On Thu, Sep 26, 2013 at 1:35 PM, jreback notifications@github.com wrote:

> @danbirken https://github.com/danbirken how's this coming?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/4065#issuecomment-25201401
> .

@danbirken enjoy the beer!

Alright, well it seems my change has triggered some incompatibilities on other platforms which I need to investigate (I just setup python2.6 build env and reproduced them locally, so that is good).  Unfortunately I'm still on vacation and having less time for work than expected so I can't get to this now.

So basically, you can ignore this change for the time being.

np...lmk....


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -348,6 +348,13 @@ def _pickle_array(arr):
 
 def _unpickle_array(bytes):
     arr = read_array(BytesIO(bytes))
+
+    # All datetimes should be stored as M8[ns].  When unpickling with
+    # numpy1.6, it will read these as M8[us].  So this ensures all
+    # datetime64 types are read as MS[ns]
+    if is_datetime64_dtype(arr):
+        arr = arr.view(_NS_DTYPE)
+
     return arr
 
 
@@ -1780,6 +1787,14 @@ def is_datetime64_dtype(arr_or_dtype):
         tipo = arr_or_dtype.dtype.type
     return issubclass(tipo, np.datetime64)
 
+def is_datetime64_ns_dtype(arr_or_dtype):
+    if isinstance(arr_or_dtype, np.dtype):
+        tipo = arr_or_dtype
+    elif isinstance(arr_or_dtype, type):
+        tipo = np.dtype(arr_or_dtype)
+    else:
+        tipo = arr_or_dtype.dtype
+    return tipo == _NS_DTYPE
 
 def is_timedelta64_dtype(arr_or_dtype):
     if isinstance(arr_or_dtype, np.dtype):
diff --git a/pandas/tseries/index.py b/pandas/tseries/index.py
--- a/pandas/tseries/index.py
+++ b/pandas/tseries/index.py
@@ -204,7 +204,7 @@ def __new__(cls, data=None,
                 data = _str_to_dt_array(data, offset, dayfirst=dayfirst,
                                         yearfirst=yearfirst)
             else:
-                data = tools.to_datetime(data)
+                data = tools.to_datetime(data, errors='raise')
                 data.offset = offset
                 if isinstance(data, DatetimeIndex):
                     if name is not None:
@@ -243,14 +243,14 @@ def __new__(cls, data=None,
                 subarr = data.view(_NS_DTYPE)
         else:
             try:
-                subarr = tools.to_datetime(data)
+                subarr = tools.to_datetime(data, box=False)
             except ValueError:
                 # tz aware
-                subarr = tools.to_datetime(data, utc=True)
+                subarr = tools.to_datetime(data, box=False, utc=True)
 
             if not np.issubdtype(subarr.dtype, np.datetime64):
-                raise TypeError('Unable to convert %s to datetime dtype'
-                                % str(data))
+                raise ValueError('Unable to convert %s to datetime dtype'
+                                 % str(data))
 
         if isinstance(subarr, DatetimeIndex):
             if tz is None:
@@ -934,7 +934,7 @@ def join(self, other, how='left', level=None, return_indexers=False):
                                         'mixed-integer-float', 'mixed')):
             try:
                 other = DatetimeIndex(other)
-            except TypeError:
+            except (TypeError, ValueError):
                 pass
 
         this, other = self._maybe_utc_convert(other)
@@ -1051,7 +1051,7 @@ def intersection(self, other):
         if not isinstance(other, DatetimeIndex):
             try:
                 other = DatetimeIndex(other)
-            except TypeError:
+            except (TypeError, ValueError):
                 pass
             result = Index.intersection(self, other)
             if isinstance(result, DatetimeIndex):
diff --git a/pandas/tseries/tools.py b/pandas/tseries/tools.py
--- a/pandas/tseries/tools.py
+++ b/pandas/tseries/tools.py
@@ -89,13 +89,12 @@ def _convert_listlike(arg, box):
         if isinstance(arg, (list,tuple)):
             arg = np.array(arg, dtype='O')
 
-        if com.is_datetime64_dtype(arg):
+        if com.is_datetime64_ns_dtype(arg):
             if box and not isinstance(arg, DatetimeIndex):
                 try:
                     return DatetimeIndex(arg, tz='utc' if utc else None)
-                except ValueError as e:
-                    values, tz = tslib.datetime_to_datetime64(arg)
-                    return DatetimeIndex._simple_new(values, None, tz=tz)
+                except ValueError:
+                    pass
 
             return arg
 


Test patch:
diff --git a/pandas/tseries/tests/test_timeseries.py b/pandas/tseries/tests/test_timeseries.py
--- a/pandas/tseries/tests/test_timeseries.py
+++ b/pandas/tseries/tests/test_timeseries.py
@@ -1,5 +1,5 @@
 # pylint: disable-msg=E1101,W0612
-from datetime import datetime, time, timedelta
+from datetime import datetime, time, timedelta, date
 import sys
 import os
 import unittest
@@ -952,6 +952,81 @@ def test_to_datetime_list_of_integers(self):
 
         self.assert_(rng.equals(result))
 
+    def test_to_datetime_dt64s(self):
+        in_bound_dts = [
+            np.datetime64('2000-01-01'),
+            np.datetime64('2000-01-02'),
+        ]
+
+        for dt in in_bound_dts:
+            self.assertEqual(
+                pd.to_datetime(dt),
+                Timestamp(dt)
+            )
+
+        oob_dts = [
+            np.datetime64('1000-01-01'),
+            np.datetime64('5000-01-02'),
+        ]
+
+        for dt in oob_dts:
+            self.assertRaises(ValueError, pd.to_datetime, dt, errors='raise')
+            self.assertRaises(ValueError, tslib.Timestamp, dt)
+            self.assert_(pd.to_datetime(dt, coerce=True) is NaT)
+
+    def test_to_datetime_array_of_dt64s(self):
+        dts = [
+            np.datetime64('2000-01-01'),
+            np.datetime64('2000-01-02'),
+        ]
+
+        # Assuming all datetimes are in bounds, to_datetime() returns
+        # an array that is equal to Timestamp() parsing
+        self.assert_(
+            np.array_equal(
+                pd.to_datetime(dts, box=False),
+                np.array([Timestamp(x).asm8 for x in dts])
+            )
+        )
+
+        # A list of datetimes where the last one is out of bounds
+        dts_with_oob = dts + [np.datetime64('9999-01-01')]
+
+        self.assertRaises(
+            ValueError,
+            pd.to_datetime,
+            dts_with_oob,
+            coerce=False,
+            errors='raise'
+        )
+
+        self.assert_(
+            np.array_equal(
+                pd.to_datetime(dts_with_oob, box=False, coerce=True),
+                np.array(
+                    [
+                        Timestamp(dts_with_oob[0]).asm8,
+                        Timestamp(dts_with_oob[1]).asm8,
+                        iNaT,
+                    ],
+                    dtype='M8'
+                )
+            )
+        )
+
+        # With coerce=False and errors='ignore', out of bounds datetime64s
+        # are converted to their .item(), which depending on the version of
+        # numpy is either a python datetime.datetime or datetime.date
+        self.assert_(
+            np.array_equal(
+                pd.to_datetime(dts_with_oob, box=False, coerce=False),
+                np.array(
+                    [dt.item() for dt in dts_with_oob],
+                    dtype='O'
+                )
+            )
+        )
+
     def test_index_to_datetime(self):
         idx = Index(['1/1/2000', '1/2/2000', '1/3/2000'])
 
diff --git a/pandas/tseries/tests/test_tslib.py b/pandas/tseries/tests/test_tslib.py
--- a/pandas/tseries/tests/test_tslib.py
+++ b/pandas/tseries/tests/test_tslib.py
@@ -4,7 +4,7 @@
 import numpy as np
 
 from pandas import tslib
-from datetime import datetime
+import datetime
 
 from pandas.core.api import Timestamp
 
@@ -15,19 +15,53 @@
 from pandas import _np_version_under1p7
 
 
-class TestDatetimeParsingWrappers(unittest.TestCase):
-    def test_verify_datetime_bounds(self):
-        for year in (1, 1000, 1677, 2262, 5000):
-            dt = datetime(year, 1, 1)
-            self.assertRaises(
-                ValueError,
-                tslib.verify_datetime_bounds,
-                dt
-            )
+class TestTimestamp(unittest.TestCase):
+    def test_bounds_with_different_units(self):
+        out_of_bounds_dates = (
+            '1677-09-21',
+            '2262-04-12',
+        )
+
+        time_units = ('D', 'h', 'm', 's', 'ms', 'us')
 
-        for year in (1678, 2000, 2261):
-            tslib.verify_datetime_bounds(datetime(year, 1, 1))
+        for date_string in out_of_bounds_dates:
+            for unit in time_units:
+                self.assertRaises(
+                    ValueError,
+                    tslib.Timestamp,
+                    np.datetime64(date_string, dtype='M8[%s]' % unit)
+                )
+
+        in_bounds_dates = (
+            '1677-09-23',
+            '2262-04-11',
+        )
 
+        for date_string in in_bounds_dates:
+            for unit in time_units:
+                tslib.Timestamp(
+                    np.datetime64(date_string, dtype='M8[%s]' % unit)
+                )
+
+    def test_barely_oob_dts(self):
+        one_us = np.timedelta64(1)
+
+        # By definition we can't go out of bounds in [ns], so we
+        # convert the datetime64s to [us] so we can go out of bounds
+        min_ts_us = np.datetime64(tslib.Timestamp.min).astype('M8[us]')
+        max_ts_us = np.datetime64(tslib.Timestamp.max).astype('M8[us]')
+
+        # No error for the min/max datetimes
+        tslib.Timestamp(min_ts_us)
+        tslib.Timestamp(max_ts_us)
+
+        # One us less than the minimum is an error
+        self.assertRaises(ValueError, tslib.Timestamp, min_ts_us - one_us)
+
+        # One us more than the maximum is an error
+        self.assertRaises(ValueError, tslib.Timestamp, max_ts_us + one_us)
+
+class TestDatetimeParsingWrappers(unittest.TestCase):
     def test_does_not_convert_mixed_integer(self):
         bad_date_strings = (
             '-50000',
@@ -97,15 +131,45 @@ def test_number_looking_strings_not_into_datetime(self):
         arr = np.array(['1', '2', '3', '4', '5'], dtype=object)
         self.assert_(np.array_equal(tslib.array_to_datetime(arr), arr))
 
-    def test_dates_outside_of_datetime64_ns_bounds(self):
-        # These datetimes are outside of the bounds of the
-        # datetime64[ns] bounds, so they cannot be converted to
-        # datetimes
-        arr = np.array(['1/1/1676', '1/2/1676'], dtype=object)
-        self.assert_(np.array_equal(tslib.array_to_datetime(arr), arr))
+    def test_coercing_dates_outside_of_datetime64_ns_bounds(self):
+        invalid_dates = [
+            datetime.date(1000, 1, 1),
+            datetime.datetime(1000, 1, 1),
+            '1000-01-01',
+            'Jan 1, 1000',
+            np.datetime64('1000-01-01'),
+        ]
 
-        arr = np.array(['1/1/2263', '1/2/2263'], dtype=object)
-        self.assert_(np.array_equal(tslib.array_to_datetime(arr), arr))
+        for invalid_date in invalid_dates:
+            self.assertRaises(
+                ValueError,
+                tslib.array_to_datetime,
+                np.array([invalid_date], dtype='object'),
+                coerce=False,
+                raise_=True,
+            )
+            self.assert_(
+                np.array_equal(
+                    tslib.array_to_datetime(
+                        np.array([invalid_date], dtype='object'), coerce=True
+                    ),
+                    np.array([tslib.iNaT], dtype='M8[ns]')
+                )
+            )
+
+        arr = np.array(['1/1/1000', '1/1/2000'], dtype=object)
+        self.assert_(
+            np.array_equal(
+                tslib.array_to_datetime(arr, coerce=True),
+                np.array(
+                    [
+                        tslib.iNaT,
+                        '2000-01-01T00:00:00.000000000-0000'
+                    ],
+                    dtype='M8[ns]'
+                )
+            )
+        )
 
     def test_coerce_of_invalid_datetimes(self):
         arr = np.array(['01-01-2013', 'not_a_date', '1'], dtype=object)
@@ -130,11 +194,11 @@ def test_coerce_of_invalid_datetimes(self):
         )
 
 
-class TestTimestamp(unittest.TestCase):
+class TestTimestampNsOperations(unittest.TestCase):
     def setUp(self):
         if _np_version_under1p7:
             raise nose.SkipTest('numpy >= 1.7 required')
-        self.timestamp = Timestamp(datetime.utcnow())
+        self.timestamp = Timestamp(datetime.datetime.utcnow())
 
     def assert_ns_timedelta(self, modified_timestamp, expected_value):
         value = self.timestamp.value