Implement numpy_helper functions directly in cython
State: closed

cleanup unused bits of src/datetime.pxd

TL;DR this PR implements `get_datetime64_value`, `get_timedelta64_value`, and `get_datetime64_unit` directly in cython instead of getting them via C.  There is no performance penalty.  Weaning off of the src directory will make the build less of a hassle.

AFAICT the main reason why `get_datetime64_unit`, `get_datetime64_value`, and `get_timedelta64_value` (as well as `is_float_object`, is_datetime64_object`...) are implemented in numpy_helper.h and np_datetime.c is because cython/numpy does not make it easy to cimport the relevant symbols: https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/src/numpy.pxd#L481

```
    # Cannot be supported due to ## ## in macro:
    # bint PyArray_IsScalar(object, verbatim work)
```

After some wrangling and [stackoverflowing](https://stackoverflow.com/questions/46015489/using-numpys-pyarray-isscalar-in-cython/46016420#46016420), I found a way to implement these directly in cython.  Importantly, the C code generated is functionally identical to the existing versions, so there shouldn't be a performance hit.

A big part of the diff comes from moving some declarations from np_datetime's pyx file to the pxd file.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=h1) Report
> Merging [#18059](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/881ee30b443f8b9651068b819ecd43adca5789b5?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18059/graphs/tree.svg?width=650&height=150&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18059      +/-   ##
==========================================
- Coverage   91.25%   91.23%   -0.02%     
==========================================
  Files         163      163              
  Lines       50115    50115              
==========================================
- Hits        45730    45721       -9     
- Misses       4385     4394       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.04% <ø> (ø)` | :arrow_up: |
| #single | `40.23% <ø> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=footer). Last update [881ee30...ea4bef0](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=h1) Report
> Merging [#18059](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/bc69dc69b168bae59d0ed6a461f18222b8137fa1?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18059/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18059      +/-   ##
==========================================
+ Coverage   91.25%   91.26%   +<.01%     
==========================================
  Files         163      163              
  Lines       50124    50124              
==========================================
+ Hits        45742    45745       +3     
+ Misses       4382     4379       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.07% <ø> (+0.02%)` | :arrow_up: |
| #single | `40.32% <ø> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.2% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=footer). Last update [bc69dc6...13b85d8](https://codecov.io/gh/pandas-dev/pandas/pull/18059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

> so there shouldn't be a performance hit.

Seem reasonable, but let's back that claim up with `asv` metrics to be sure.
instead of this, is there a reason you simply don't move datetime.pxd into np_datetime.pxd? (certainly can do some of the code cleanup, but after)
> Seem reasonable, but let's back that claim up with asv metrics to be sure.

Will do.  Even with affinity my asv results probably merit outside confirmation.
> instead of this, is there a reason you simply don't move datetime.pxd into np_datetime.pxd? (certainly can do some of the code cleanup, but after)

That may be worth doing, but I see it as orthogonal to this.  Cutting numpy_helper out of the equation simplifies things quite a bit (among other things I'm thinking about setup.py where 'include' and 'pxd' specifications seem pretty fragile)
With appropriate skepticism:
```
taskset 5 asv continuous -E virtualenv -f 1.1 master HEAD -b timestamp
[...]
      before           after         ratio
     [d80a4b81]       [ea4bef00]
+     5.95±0.02μs      6.79±0.05μs     1.14  timestamp.TimestampProperties.time_is_leap_year
+     5.71±0.03μs         6.28±1μs     1.10  timestamp.TimestampProperties.time_is_quarter_end
-     7.09±0.02μs      6.17±0.02μs     0.87  timestamp.TimestampProperties.time_is_year_start
-        195±50ns        109±0.2ns     0.56  timestamp.TimestampProperties.time_microsecond
-        862±40ms         159±10ms     0.18  binary_ops.Timeseries.time_timestamp_ops_diff2
-        443±10ms         77.5±4ms     0.18  binary_ops.TimeseriesTZ.time_timestamp_ops_diff2
-        469±30ms         63.1±7ms     0.13  binary_ops.TimeseriesTZ.time_timestamp_ops_diff1
-         155±8ms       19.1±0.2ms     0.12  binary_ops.Timeseries.time_timestamp_ops_diff1

taskset 5 asv continuous -E virtualenv -f 1.1 master HEAD -b timeseries
[...]
       before           after         ratio
     [d80a4b81]       [ea4bef00]
+      17.3±0.9ms         21.6±1ms     1.25  timeseries.TimeSeries.time_add_irregular
+     46.3±0.09μs       54.3±0.2μs     1.17  timeseries.SemiMonthOffset.time_begin_incr
+        132±10ms            150ms     1.13  timeseries.DatetimeIndex.time_dti_tz_factorize
-      20.3±0.1μs      18.4±0.09μs     0.90  timeseries.Offsets.time_custom_bday_incr
-      20.6±0.1μs       17.8±0.1μs     0.86  timeseries.Offsets.time_timeseries_day_apply

```

Digging out the C code produced by cython for the implementation of `get_datetime64_value` here (and stripping out the traceback comments)

```
static CYTHON_INLINE npy_datetime __pyx_f_6pandas_5_libs_6tslibs_11np_datetime_get_datetime64_value(PyObject *__pyx_v_obj) {
  npy_datetime __pyx_r;

  __pyx_r = ((PyDatetimeScalarObject *)__pyx_v_obj)->obval;
  goto __pyx_L0;

  /* function exit code */
  __pyx_L0:;
  return __pyx_r;
}
```

Whereas the version being replaced is:
```
PANDAS_INLINE npy_datetime get_datetime64_value(PyObject* obj) {
        return ((PyDatetimeScalarObject*)obj)->obval;
    }
```

cython is really verbose, but I pinged their mailing list and was told that the verbosity should be stripped out by the compiler.

Same story for `get_datetime64_unit` and `get_timedelta64_value`.
@chris-b1 can you review
rebase as always on re-push
Thoughts on this?  I'd like to follow-up by moving a few more pieces of src/datetime.pxd over to np_datetime.pxd
ping @jreback
thanks! you will prob need to rebase your other PR's on this just to make sure.