PROPOSAL: use datetime64 and timedelta64 to eval
State: closed

This PR aims to use datetime64 and timedelta64 in evaluation directly, instead of casting to integer and then casting back. 

Benefits:
1. `values` is saved in `ndarray[datetime64]`, casting to `int` to eval is non-intuitive. It seems back to the time before numpy 1.7 which implements `datetime64`.
2. `numpy.ufunc` can handle all the `dtype` change, e.g.  `datetime64 - datetime64 = timedelta64`. Additonally, nonsense operations are prohibited, e.g. `datetime64 + datetime64`.
3. numpy implements `datetime64('NaT')`, seems `values_mask` and `other_mask` are no longer needed for `nan`s.
4. `Series` can share the same call path with `DataFrame` in `Block`. Now `Series` is using `dispatch_to_index_op` to wrap as `Index`, but `Index` is 1d-array while `DataFrame` is 2d-array, so `DataFrame` can't use the same wrapper. `ndarray([Index])` is `dtype='O'` so can't be benefited from `numpy.ufunc`.

This commit has passed `tests/{internals,frame,dtypes}` , but not `tests/series`, because `dispatch_to_index_op` breaks it.

Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
> `Series` can share the same call path with `DataFrame` in `Block`

We actually want to go in the other direction.  We want to make `DataFrame` dispatch to the `Series` (or `DatetimeIndex`/`TimedeltaIndex`) implementations.  Is there a viable way to do that here?  Maybe dispatch to `self._holder`?

>what problem are you trying to solve here?

@jreback #22005 will no longer exist. But the main purpose is simplify the code, and make `DataFrame` and `Series` operations share same call path in `Block`.

> We actually want to go in the other direction. We want to make DataFrame dispatch to the Series (or DatetimeIndex/TimedeltaIndex) implements. Is there a viable way to do that here? Maybe dispatch to self._holder?

@jbrockmendel 
As pointed out in Benefits (4),  `Series (or DatetimeIndex/TimedeltaIndex)` can only handle 1d-array. If have to make DataFrame dispatch into  1d-array, there is 2 possible implementations:
1. Reduce 2d-ndarray into 1d-ndarray, eval with `numpy.ufunc`, then raise back. This will need to due with shape changing and possible data copying. And it will break ops between `1d-ndarray` and `2d-ndarray` (need a different call path).
2. Make it into `ndarray[DatetimeIndex]`(`1d-ndarray` of `DatetimeIndex`). But it will be treat as `dtype='O'` in `numpy`, the call path will be
    1. make `2d-ndarray[dtype='m8'`] into `1d-ndarray[DatetimeIndex]`
    2. `ufunc` with `1d-ndarray[dtype='O']`
    3. `eval` with `DatetimeIndex`
    4.  `ufunc` with `1d-ndarray[dtype='m8']`

    This PR will simply be 
    1. `ufunc` with `2d-ndarray[dtype='m8']`

As `numpy` already handle data operating well, why do we need an extra implement with less efficiency? In fact, dispatching to `DatetimeIndex` is ending up with `ufunc with 1d-ndarray` too, what `DatetimeIndex` do can be done (already implemented, seems only exception is tz-detection) in `Block` and `ufunc`.

Could someone show me the benefits of the `Index`-dispatch implement?

---
I have write a 2nd commit,which implements:
1. `.tz` info save as `DatetimeTZBlock.tz`, no longer need a `DatetimeIndex`. Multi-columns can share a same `DatetimeTZBlock` if they have same `tz`. (NotImplemented)
2. `_try_coerce_args` is implemented in `DatetimeLikeBlockMixin`, those override in subclass are removed. So as `_try_coerce_result`. (Benefited from `numpy` handing data)
3. `values_mask` and `other_mask` are also removed. (`numpy` can handle `NaT`)

Tests haven't pass all, those failed mainly is `DataFrame` columns converting to `Series`.
Hello @holymonson! Thanks for updating the PR.









Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on August 12, 2018 at 08:41 Hours UTC
> can only handle 1d-array

Can you expand on why this is a compelling reason?  Maybe there is some characteristic np.ufunc that I'm not considering?

> so `DataFrame` can't use the same wrapper

The idea is that `DataFrame` dispatches column-wise, like it already does in `DataFrame._combine_frame` and `DataFrame._compare_frame`

> Could someone show me the benefits of the Index-dispatch implement?

See #18824.  As you've noticed, there are many operations (particularly arithmetic and comparisons) where `DataFrame` behavior is incorrect, particularly for datetime64 and datetime64tz dtypes.  Until a few months ago, the same was true for Series.  We fixed the Series problems by having it dispatch to the Index subclasses, since the DatetimeIndex and TimedeltaIndex implementations are much more careful.  Now we're down to two implementations instead of three; we still should get it down to one.

@holymonson can you rebase?  Arithmetic ops have seen a lot of cleanup in the last couple months.
Closing as stale. Ping if you'd like to pick this back up