Repository: pandas-dev/pandas

Issue:
BUG: coercion of non-M8[ns] in datetime ops
```
import datetime
s = pd.Series(pd.date_range('20130101',periods=3))
s-pd.Timestamp('20130101')
s-datetime.datetime(2013,1')
```

This fails as the datetime64 is not converted properly (because numpy datetime ops suck)

```
s-np.datetime64('20130101')
```

e.g. `np.datetime64('20130101').astype('M8[ns]')` is a bug, no?



Discussion:
I am not fully following here. Isn't this just a limitation of numpy's string parsing? `np.datetime64('2013-01-01').astype('M8[ns]')` looks OK to me.  

Numpy only parses ISO timestrings, so `np.datetime64('20130101')` will never work? Doing `np.datetime64('2013-01-01').astype('M8[ns]')`  does not look wrong. Or is that not what you mean?

But in any case `s-np.datetime64('2013-01-01')` should still ideally work. Can pandas work around this? (by always doing a `.astype('M8[ns]')` in `__rsub__`?)

no I think it's just broken in numpy

np.datetime64('2013-01-01') make this dtype of M8[D] but I don't think it allows astype to M8[ns](correctly) 

it still IS possible though I think you just have to get the value and figure it out based in the dtype (which is where pandas can handle it - if the astype worked then it would be easy)

```
In [94]: np.datetime64('2013-01-01')
Out[94]: numpy.datetime64('2013-01-01')

In [95]: np.datetime64('2013-01-01').astype('M8[ms]')
Out[95]: numpy.datetime64('2013-01-01T01:00:00.000+0100')

In [96]: np.datetime64('2013-01-01').astype('M8[ns]')
Out[96]: numpy.datetime64('2013-01-01T01:00:00.000000000+0100')
```

This looks like astype to `M8[ns]` works? 

hmm maybe doesn't work in 1.7 I think

that was in 1.7.1 (and 1.8.1)

try exactly how I have it (the format)

```
In [98]: np.datetime64('20130101').astype('M8[ns]')
Out[98]: numpy.datetime64('2151-06-04T08:32:39.009206272+0200')
```

That is indeed bullshit output, but that is because numpy does not support non-iso string parsing, not because of the `astype` not working. 
Without the `astype`, it is also not really working (see that the datetime is not parse, but it _is_ strange it does not give an error here):

```
In [12]: np.datetime64('20130101')
Out[12]: numpy.datetime64('20130101')
```

That seems a bug in numpy, that is does not raise:

```
In [99]: np.datetime64('20130101')
Out[99]: numpy.datetime64('20130101')

In [100]: np.datetime64('20130101 10:00')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-100-6968cb1137d9> in <module>()
----> 1 np.datetime64('20130101 10:00')

ValueError: Error parsing datetime string "20130101 10:00" at position 8

In [101]: np.datetime64('2013-01-01 10:00')
Out[101]: numpy.datetime64('2013-01-01T10:00+0100')
```

it does raise when there is also an hour and not only date. Or is there a reason a date-only would allow more flexible string parsing?

ahh ok

but we still don't handle this input correctly (a non ns numpy datetime input)

yes, so it a legitimate issue :-) (only the comment about `astype` was not correct)

@jreback Figured out the strange behaviour of numpy :-)

It is interpreting `np.datetime64('20130101')` as the _year_ 20,130,101, so it is logical this does not raise an error about malformated date, and that it does not fit in a `ns` range:

```
In [16]: np.datetime64('20130101').dtype
Out[16]: dtype('<M8[Y]')

In [17]: np.datetime64('20130101').astype('M8[D]')
Out[17]: numpy.datetime64('20130101-01-01')
```

But it apparantly does not give an out-of-range date error when astyping to ns range.

For copy/pasting, the OP has a typo in line `s-datetime.datetime(2013,1')`.

AFAICT the `np.datetime64('20130101')` is unsalvageable and the open issue is `s-np.datetime64('2013-01-01')`.  Is that correct?

Slightly different kind of wrong when using a DatetimeIndex instead of Series:
```
>>> dti = pd.date_range('20130101',periods=3)
>>> dti - np.datetime64('2013-01-01')
DatetimeIndex(['1970-01-01', '1970-01-02', '1970-01-03'], dtype='datetime64[ns]', freq=None)
```
> the open issue is s-np.datetime64('2013-01-01'). Is that correct?

Yes, it is:

```
In [2]: import datetime
   ...: s = pd.Series(pd.date_range('20130101',periods=3))
   ...: 

In [3]: s-pd.Timestamp('20130101')
Out[3]: 
0   0 days
1   1 days
2   2 days
dtype: timedelta64[ns]

In [5]: s-datetime.datetime(2013,1,1)
Out[5]: 
0   0 days
1   1 days
2   2 days
dtype: timedelta64[ns]

In [6]: s-np.datetime64('2013-01-01')
Out[6]: 
0   15705 days 23:59:59.999984
1   15706 days 23:59:59.999984
2   15707 days 23:59:59.999984
dtype: timedelta64[ns]
```
I'm about to submit a PR that fixes this.  It works for Series but still fails for DataFrame.

Resolving patch:
diff --git a/pandas/core/ops.py b/pandas/core/ops.py
--- a/pandas/core/ops.py
+++ b/pandas/core/ops.py
@@ -28,7 +28,7 @@
     is_datetimelike_v_numeric,
     is_integer_dtype, is_categorical_dtype,
     is_object_dtype, is_timedelta64_dtype,
-    is_datetime64_dtype, is_datetime64tz_dtype,
+    is_datetime64_dtype, is_datetime64tz_dtype, is_datetime64_ns_dtype,
     is_bool_dtype, is_datetimetz,
     is_list_like, is_offsetlike,
     is_scalar,
@@ -527,6 +527,11 @@ def _convert_to_array(self, values, name=None, other=None):
             elif not (isinstance(values, (np.ndarray, ABCSeries)) and
                       is_datetime64_dtype(values)):
                 values = libts.array_to_datetime(values)
+            elif (is_datetime64_dtype(values) and
+                  not is_datetime64_ns_dtype(values)):
+                # GH#7996 e.g. np.datetime64('2013-01-01') is datetime64[D]
+                values = values.astype('datetime64[ns]')
+
         elif inferred_type in ('timedelta', 'timedelta64'):
             # have a timedelta, convert to to ns here
             values = to_timedelta(values, errors='coerce', box=False)


Test patch:
diff --git a/pandas/tests/series/test_operators.py b/pandas/tests/series/test_operators.py
--- a/pandas/tests/series/test_operators.py
+++ b/pandas/tests/series/test_operators.py
@@ -1004,6 +1004,33 @@ def test_operators_timedelta64_with_timedelta_invalid(self, scalar_td):
 
 
 class TestDatetimeSeriesArithmetic(object):
+    @pytest.mark.parametrize(
+        'box, assert_func',
+        [(Series, tm.assert_series_equal),
+         (pd.Index, tm.assert_index_equal)])
+    def test_sub_datetime64_not_ns(self, box, assert_func):
+        # GH#7996
+        dt64 = np.datetime64('2013-01-01')
+        assert dt64.dtype == 'datetime64[D]'
+
+        obj = box(date_range('20130101', periods=3))
+        res = obj - dt64
+        expected = box([Timedelta(days=0), Timedelta(days=1),
+                        Timedelta(days=2)])
+        assert_func(res, expected)
+
+        res = dt64 - obj
+        assert_func(res, -expected)
+
+    @pytest.mark.xfail(reason='GH#7996 datetime64 units not converted to nano')
+    def test_frame_sub_datetime64_not_ns(self):
+        df = pd.DataFrame(date_range('20130101', periods=3))
+        dt64 = np.datetime64('2013-01-01')
+        assert dt64.dtype == 'datetime64[D]'
+        res = df - dt64
+        expected = pd.DataFrame([Timedelta(days=0), Timedelta(days=1),
+                                 Timedelta(days=2)])
+        tm.assert_frame_equal(res, expected)
 
     def test_operators_datetimelike(self):
         def run_ops(ops, get_ser, test_ser):