Repository: pandas-dev/pandas

Issue:
np.datetime64 values cast to datetime.date iff dtype == M8
This seems a little inconsistent to me. IMHO, it would be good to be able to rely on the the fact that if you specify a numpy array of _any_ datetime64 type the result will still be a datetime64 type and not cast to a different type dependant on the exact dtype passed in.

``` python
In [1]: dates = pd.date_range('01-Jan-2015', '01-Dec-2015', freq='M')
   ...: values1 = dates.view(np.ndarray).astype('M8[D]')
   ...: values2 = dates.view(np.ndarray).astype('datetime64[ns]')
   ...: series1 = pd.TimeSeries(values1, dates)
   ...: series2 = pd.TimeSeries(values2, dates)
   ...: 

In [2]: series1
Out[2]: 
2015-01-31    2015-01-31
2015-02-28    2015-02-28
2015-03-31    2015-03-31
2015-04-30    2015-04-30
2015-05-31    2015-05-31
2015-06-30    2015-06-30
2015-07-31    2015-07-31
2015-08-31    2015-08-31
2015-09-30    2015-09-30
2015-10-31    2015-10-31
2015-11-30    2015-11-30
Freq: M, dtype: object

In [3]: series2
Out[3]: 
2015-01-31   2015-01-31
2015-02-28   2015-02-28
2015-03-31   2015-03-31
2015-04-30   2015-04-30
2015-05-31   2015-05-31
2015-06-30   2015-06-30
2015-07-31   2015-07-31
2015-08-31   2015-08-31
2015-09-30   2015-09-30
2015-10-31   2015-10-31
2015-11-30   2015-11-30
Freq: M, dtype: datetime64[ns]

In [4]: series1.values
Out[4]: array([datetime.date(2015, 1, 31), datetime.date(2015, 2, 28), datetime.date(2015, 3, 31), datetime.date(2015, 4, 30), datetime.date(2015, 5, 31), datetime.date(2015, 6, 30), datetime.date(2015, 7, 31), datetime.date(2015, 8, 31), datetime.date(2015, 9, 30), datetime.date(2015, 10, 31), datetime.date(2015, 11, 30)], dtype=object)

In [5]: series2.values
Out[5]: array(['2015-01-31T00:00:00.000000000+0000', '2015-02-28T00:00:00.000000000+0000', '2015-03-31T01:00:00.000000000+0100', '2015-04-30T01:00:00.000000000+0100', '2015-05-31T01:00:00.000000000+0100', '2015-06-30T01:00:00.000000000+0100', '2015-07-31T01:00:00.000000000+0100', '2015-08-31T01:00:00.000000000+0100', '2015-09-30T01:00:00.000000000+0100', '2015-10-31T00:00:00.000000000+0000', '2015-11-30T00:00:00.000000000+0000'], dtype='datetime64[ns]')

In [6]: pd.__version__
Out[6]: '0.13.1-339-g6c3755b'
```



Discussion:
The problem is that some people actually want to keep an actual `datetime.date` value in a Series, even though its pretty much useless and inefficient.

note that series1.dtype is `object`, while series2 is converted to `datetime64[ns]`.

Their is no way to 'tell' what should be done here.

IMHO we _should_ convert these, and we could, but that would break backward usage. thoughts?

The issue isn't _keeping_ `datetime.date` instances about - there weren't any in the first place until the `TimeSeries` class cast the `datetime64[D]` instances to `datetime.date` instances. So it's actually _changing_ the type of the input you're supplying to it, to, as you've noted, a pretty useless and non-performant type.

As for backward compatibility, I picked this up from a breakage in our unittests - It looks like it is a regression between 13.0 and 13.1, though 12.0 seems to have the most consistent behaviour as it doesn't change the dtype of the array at all - but that could equally be a np17 vs np18 issue.

**pandas 0.12.0**

``` python
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.12.0'
>>> dates = pd.date_range('01-Jan-2015', '01-Dec-2015', freq='M')
>>> values1 = dates.view(np.ndarray).astype('M8[D]')
>>> series1 = pd.TimeSeries(values1, dates)
>>> series1
2015-01-31   2015-01-31 00:00:00
2015-02-28   2015-02-28 00:00:00
2015-03-31   2015-03-31 00:00:00
2015-04-30   2015-04-30 00:00:00
2015-05-31   2015-05-31 00:00:00
2015-06-30   2015-06-30 00:00:00
2015-07-31   2015-07-31 00:00:00
2015-08-31   2015-08-31 00:00:00
2015-09-30   2015-09-30 00:00:00
2015-10-31   2015-10-31 00:00:00
2015-11-30   2015-11-30 00:00:00
Freq: M, dtype: datetime64[D]
>>>
```

**pandas 0.13.0**

``` python
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.13.0'
>>> dates = pd.date_range('01-Jan-2015', '01-Dec-2015', freq='M')
>>> values1 = dates.view(np.ndarray).astype('M8[D]')
>>> series1 = pd.TimeSeries(values1, dates)
>>> series1
2015-01-31   2015-01-31 00:00:00
2015-02-28   2015-02-28 00:00:00
2015-03-31   2015-03-31 00:00:00
2015-04-30   2015-04-30 00:00:00
2015-05-31   2015-05-31 00:00:00
2015-06-30   2015-06-30 00:00:00
2015-07-31   2015-07-31 00:00:00
2015-08-31   2015-08-31 00:00:00
2015-09-30   2015-09-30 00:00:00
2015-10-31   2015-10-31 00:00:00
2015-11-30   2015-11-30 00:00:00
Freq: M, dtype: datetime64[ns]
>>>
```

**pandas 0.13.1**

``` python
>>> import numpy as np
>>> import pandas as pd
>>> pd.__version__
'0.13.1'
>>> dates = pd.date_range('01-Jan-2015', '01-Dec-2015', freq='M')
>>> values1 = dates.view(np.ndarray).astype('M8[D]')
>>> series1 = pd.TimeSeries(values1, dates)
>>> series1
2015-01-31    2015-01-31
2015-02-28    2015-02-28
2015-03-31    2015-03-31
2015-04-30    2015-04-30
2015-05-31    2015-05-31
2015-06-30    2015-06-30
2015-07-31    2015-07-31
2015-08-31    2015-08-31
2015-09-30    2015-09-30
2015-10-31    2015-10-31
2015-11-30    2015-11-30
Freq: M, dtype: object
>>>
```

what numpy are these under? 1.7.1/2? 1.8?

0.12.0 is wrong; no numpy datetime dtypes can live except for `datetime64[ns]` they all must be converted upon input.

I can prob put back to 0.13.0 which seems to make the most sense (if you are actually passing `datetime.date` then I would expect those to stay).

Agreed on the fact that `datetime.date` can stay if they're explicitly passed in. I don't mind the dates being cast to `datetime64[ns]` - so long as they're still numpy `datetime64` instances.
FWIW the (simplified) test which works under 0.12.0 and 0.13.0 but breaks with 0.13.1 was:

``` python
def test_datetime64_values_arent_cast():
    dates = pd.date_range('01-Jan-2015', '01-Dec-2015', freq='M').view(np.ndarray).astype('M8[D]')
    series = pd.Series(dates)
    assert np.all(series.values == dates)
```

NB: I'm using anaconda so pandas 0.12.0 was tested against numpy 1.7.1 and 0.13+ against numpy 1.8


Resolving patch:
diff --git a/pandas/core/common.py b/pandas/core/common.py
--- a/pandas/core/common.py
+++ b/pandas/core/common.py
@@ -124,7 +124,7 @@ def isnull(obj):
 
     See also
     --------
-    pandas.notnull: boolean inverse of pandas.isnull    
+    pandas.notnull: boolean inverse of pandas.isnull
     """
     return _isnull(obj)
 
@@ -272,7 +272,7 @@ def notnull(obj):
     isnulled : array-like of bool or bool
         Array or bool indicating whether an object is *not* null or if an array
         is given which of the element is *not* null.
-    
+
     See also
     --------
     pandas.isnull : boolean inverse of pandas.notnull
@@ -1727,10 +1727,7 @@ def _possibly_cast_to_datetime(value, dtype, coerce=False):
             dtype = value.dtype
 
             if dtype.kind == 'M' and dtype != _NS_DTYPE:
-                try:
-                    value = tslib.array_to_datetime(value)
-                except:
-                    raise
+                value = value.astype(_NS_DTYPE)
 
             elif dtype.kind == 'm' and dtype != _TD_DTYPE:
                 from pandas.tseries.timedeltas import \


Test patch:
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -600,6 +600,25 @@ def test_constructor_dtype_datetime64(self):
         self.assertEqual(result['a'], Timestamp('20130101'))
         self.assertEqual(result['b'], 1)
 
+        # GH6529
+        # coerce datetime64 non-ns properly
+        dates = date_range('01-Jan-2015', '01-Dec-2015', freq='M')
+        values2 = dates.view(np.ndarray).astype('datetime64[ns]')
+        expected = Series(values2, dates)
+
+        # numpy < 1.7 is very odd about astyping
+        if not _np_version_under1p7:
+            for dtype in ['s','D','ms','us','ns']:
+                values1 = dates.view(np.ndarray).astype('M8[{0}]'.format(dtype))
+                result = Series(values1, dates)
+                assert_series_equal(result,expected)
+
+        # leave datetime.date alone
+        dates2 = np.array([ d.date() for d in dates.to_pydatetime() ],dtype=object)
+        series1 = Series(dates2, dates)
+        self.assert_numpy_array_equal(series1.values,dates2)
+        self.assertEqual(series1.dtype,object)
+
     def test_constructor_dict(self):
         d = {'a': 0., 'b': 1., 'c': 2.}
         result = Series(d, index=['b', 'c', 'd', 'a'])