Repository: numpy/numpy

Issue:
Add `isfinite` support for `datetime64` and `timedelta64`
`numpy.isfinite` should support `datetime64` and `timedelta64` objects.  It should return `True` for any ordinary value, and `False` for `datetime64('NaT')` and `timedelta64('NaT')`.

As a side-effect, this will make `isclose` and `allclose` usable for `datetime64` and `timedelta64` objects.

```
In [97]: numpy.isfinite(numpy.datetime64(10**9, 's'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-97-8555b18406e5> in <module>()
----> 1 numpy.isfinite(numpy.datetime64(10**9, 's'))

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```



Discussion:
I made some progress on this today: https://github.com/numpy/numpy/commit/cc99cdd53de418cb7a00d580676c87246108e096 / 
https://github.com/shoyer/numpy/tree/datetime64-isfinite

It works for unit-less datetime64/timedelta64, but fails for dates or times with units attached:

```
In [1]: import numpy as np

In [2]: np.isfinite(np.timedelta64(10))
Out[2]: True

In [3]: np.isfinite(np.timedelta64('NaT'))
Out[3]: False

In [4]: np.isfinite(np.timedelta64('NaT', 'us'))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-3998f88199cc> in <module>()
----> 1 np.isfinite(np.timedelta64('NaT', 'us'))

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```

It appears that this needs something with the ufunc dtype resolution to handle datetimes with units, but I'm not quite sure what that is. Somehow `PyUFunc_SimpleBinaryComparisonTypeResolver` and `PyUFunc_SimpleUnaryOperationTypeResolver` can handle datetime64 with units but the default `PyUFunc_DefaultTypeResolver` cannot.

@shoyer I was wondering whether you got somewhere with this? I fear that if we change the NaT comparison handling, we probably cannot avoid implementing `isnan` and `isfinite` for it.
Otherwise even our own tests should not like them (though of course we could work around that).

EDIT: See also gh-5222

@seberg nope, I got stuck on ufunc type resolution (in the linked branch). I do agree this would be useful, though I usually find checking for `x != x` is a good way to identify NaN/NaT even without a dedicated function.

I just hit this bug. I hope it can be fixed.

I took a look at @shoyer's branch at https://github.com/numpy/numpy/commit/cc99cdd53de418cb7a00d580676c87246108e096.  The issue seems to be that the default type resolver ends up invoking [`linear_search_type_resolver`](https://github.com/numpy/numpy/blob/master/numpy/core/src/umath/ufunc_type_resolution.c#L1867), which essentially executes something like:

```
for type in ufunc.registered_types:
    if ufunc_loop_matches(input_data, type, ...):
       <use type>
```

(the actual code here is https://github.com/numpy/numpy/blob/master/numpy/core/src/umath/ufunc_type_resolution.c#L1867).

For at least the scalar case, `ufunc_loop_matches` essentially boils down to a call to `PyArray_CanCastTypeTo`: https://github.com/numpy/numpy/blob/master/numpy/core/src/umath/ufunc_type_resolution.c#L1529-L1535 with a casting rule of `'safe'`.

The issue ends up being that the input dtype is `datetime64[<unit>]` or `timedelta64[<unit>]`, but the type registered for the ufunc is just vanilla `datetime64` or `timedelta64`.  Since casting from a datetime with a unit to one without a unit isn't deemed, safe, we decide that we don't have a ufunc registered for the input data.  (This is why the dispatch works correctly on unit-less datetimes/timedeltas.).

I think the easiest fix here is probably to just add a new ufunc type resolver for `isfinite`.  It doesn't seem worthwhile to try to shoehorn datetime/timedelta-handling logic into the default resolver, since almost all ufuncs that work with datetimes/timedeltas seem to need their own custom behavior to deal with unit conversion (there are already custom resolvers used for `add`, `subtract` and `multiply` for example.

@ssanderson would it also work to register the datetime types as types to `np.isfinite` and `np.isnan` (assuming we are OK using those names)? Honestly, have not tried to understand the issues yet really.

If you ahve some time, we all always appreciate pull requests!

@seberg I don't think it's possible to register ufuncs for the specific unit.  Looking at the signature for `PyUFunc_FromFuncAndData`, it looks like you can only provide type numbers, but the units are metadata attached to the type number.

I made some progress on a custom resolver last night; I'll see if I can tie off that work tonight.

It's possibly worth figuring out what path in ufunc_type_resolution `isnan`/`isfinite` normally take (for float data) and why that doesn't work for datetime64.

@shoyer I think my long post above outlines the path they take.  

The root issue is that part of (at least the generic case) of ufunc resolution is checking whether the input dtype can be safely cast to a dtype for which there's a registered ufunc for the function being dispatched.  ufuncs that work on datetime/timedelta are always just registered with a generic dtype, and it's not considered safe to cast from `datetime64[<unit>]` to generic `datetime64` (which seems correct in general?) which means that, as far as I can tell, every ufunc that works on datetime/timedelta needs a custom resolver to work correctly.  

The only exception seems to be ufuncs dispatched with `PyUFunc_SimpleBinaryComparisonTypeResolver`, which directly invokes `PyArray_ResultType` to figure out how to coerce the input, and then always yields `NPY_BOOL` for the output.  Something similar, e.g. a `PyUFunc_SimpleUnaryPredicateTypeResolver` for functions that always yield NPY_BOOL would probably work for `isfinite`, and `isnan` if we wanted to make that work for datetime (I'd personally be :-1: on that: I'd rather have a generic `isnull` than have `isnan` work on non-floating values).

@shoyer specifically, the reason we end up trying to check coercions between a generic datetime and a unit-having datetime is that we get our target type in `ufunc_loop_matches` by calling [`PyArray_DescrFromType`](https://github.com/numpy/numpy/blob/master/numpy/core/src/umath/ufunc_type_resolution.c#L1512) on the typenum being checked.  The dtype created there has no unit set, so it behaves like a generic datetime when we try to check coercions against unit-aware inputs [later on in the function](https://github.com/numpy/numpy/blob/master/numpy/core/src/umath/ufunc_type_resolution.c#L1529-L1541).

So now I'm wondering if maybe the right fix here is to check if we're resolving a datetime/timedelta and if so copy the metadata from our input to the temporary dtype used for the casting check...

Here's an attempt that gets the isfinite tests working for datetime and friends: https://github.com/numpy/numpy/pull/7856.

I'm not totally sure whether the short-circuit on equal typenums I added is safe.  The alternative, I think, is to figure out how to correctly populate the temp dtype's metadata by copying it from the input array.  As far as I can tell, that requires a new publicly-visible function in numpy_api.py.  At the very least, I couldn't figure out how to call a private function defined in `multiarray` from a function defined in `umath`.

@ssanderson, do we have any movement on this issue?  It would be great if we could get this in sometime soon because it is affecting our workflow (https://github.com/MPAS-Dev/MPAS-Analysis/issues/269).  Can we get some idea of what is needed to resolve this for the short-term?  For instance, https://github.com/numpy/numpy/pull/7856 has gone stale-- what help (if any) is needed to revitalize the PR?  Or, is an alternative approach needed?
By now we have `np.isnat`, a PR is probably good. It might be that you can steal almost everything from there. Dunno if there is any discussion necessary about whether or not we should add this, but allowing it in isfinite seems OK.
Bump. Ran into np.isnan failing on pd.datetimeindex, tried converting it to a datetime, pandas.Timestamp, and np.datetime64 but same issue. 
Yep, just ran into this issue today too.
I think what this requires is including the datetime and timedelta dtypes in the loops understood by `isfinite`, with the loops themselves essentially copied from `isnat` (but obviously with the opposite result). In principle, not difficult, though I labeled it as "intermediate" since anything to do with ufuncs is non-trivial.
@mhvk I've decided to try to solve this issue. It's high time I learned about ufuncs and the C API and I would like to use this as a stepping stone to make it possible to histogram datetimes.
Great!
There is some discusssion around the idea that ufunc inner loops should have access to the operand dtypes. It is not clear how to do this yet in a backward-compatible fashion, perhaps extending the signature of [`PyUFuncGenericFunction`](http://www.numpy.org/devdocs/reference/c-api.ufunc.html?highlight=pyufuncgenericfunction#functions) to  `void loopfunc(char** args, npy_intp* dimensions, npy_intp* steps, void* data, PyArray_Descr * dtypes)`? Then datetime64 loops could acces the units. Theoretically, we could repurpose the rarely-used `data` argument, but that would break code like [`PyUFunc_e_e`](https://github.com/numpy/numpy/blob/fe3412fde8990ff08405f3324d309e521aff1d7d/numpy/core/src/umath/loops.c.src#L261) and friends, which are used in downstream projects.

Edit: add link to `PyUFuncGenericFunction`
They need access to it, but I wonder if they don't need a hook already
before (during/after the dtype resolution step) anyway.
Also, it might be nice to try to think ahead and see that such a new
hook is at a point where it could in principle modify a flexible output
type to be correct.

What I mean is, we have dtype resolution step (and specializations for
ufuncs). After a specific loop is decided (eg. 's3s5->s'), that loop
could get a hook to decide on things (or throw errors), so make it
's3s5->s8' (allowing output allocation to work).
Then that function could also set a data void* (maybe a new one) to
whatever it signal whatever its specific inner loop needs.

@seberg 
> Then that function could also set a data void*

Do you mean the dtype resolution function? Yes, I think I see where you are going with that, the resolution function could set the return dtype and also set the `data` pointer to a specific function as determined from the operand dtypes, this specific function would be called in the inner loop like PyUFunc_e_e and friends, as in the end of this [example](http://www.numpy.org/devdocs/user/c-info.ufunc-tutorial.html#example-numpy-ufunc-with-structured-array-dtype-arguments) but not hard-coding the data pointer.

Error handling: the specific function could set a `PyErr` exception, and abort the inner loop function and then we could check after the loop execution `PyErr_Occurred` to raise the exception in the ufunc machinery. Note there is no return value from the inner loop functions, so there is no direct way to know that a loop erred out. This does in some way break the ufunc design that inner loops are not python-capi-aware, but I don't know if that was an explicit design goal.
Yes, for the most part. Except I do not know why we would need to pass
in anything to the inner loop (except assuming the inner loop needs it,
such as the string length.

Say we have a units dtype, then they can already throw the error
during/after/as part of the type resolution step. I do not think it
needs to pass a function to the inner loop. At least not normally.

(Of course some stranger things might pass in functions for
functionality)

Now the question is also, if we need an "after" hook in some sense, but
probably not really.

As an example:

`unit + unit` goes into the type resolvers and it says finds it is the:

"unit,unit->unit" registered loop, not it will call:

unit_finalize_dtype_for_add(*actual_out_dtype_if_it_is_flexible,
                            *actual_loop)

That tests if the units make sense (and throws an error otherwise), and
sets the output dtype to have the correct unit, etc.
It then just assigns the normal `d,d->d` loop (or actually, it does not
and that is already fixed during registration).
But the thing is: the unit dtype should be able to reuse the double
inner loops.

My sense is that there is no real need to adjust the inner loops themselves for dealing with types - there is already the general purpose `*data` (I've used it for a prototype chained ufunc); so far, the only real weakness I have found with the whole ufunc mechanism is the inability to assign output arrays, but I think all that requires is moving the hooks for type resolution up earlier in the code, and allowing code to do more in that hook (using the type resolver also seems to have been the conclusion in this thread).

One part where a standard for passing to inner loops might be more useful is for `where` - working via a write-back buffer prevents meaningful use of `where` in reductions.

(It does seem we're a bit far off track of the actual issue here! It would seem useful to collect cases where the current ufunc structure doesn't work so well in a wiki; for what it is worth, for units I think chained ufuncs are the way to go, i.e., one calls an upper inner loop which calls more standard inner loops below).
Just got hit by this bug.  My use case is simple: `np.histogram(ndarray_of_datetime64)`.  Without being able to do this, I basically have no reason to use the datetime64 support.  Please fix!
I think #12988 might address this.
@concretevitamin: `np.histogram` will work fine as long as you compute the bins for it yourself. There are more things blocking that than just nan-checking. `linspace` is another.
@eric-wieser Sure - make ndarray of datetime64 function more like first-class citizen will be welcome!

Resolving patch:
diff --git a/numpy/core/code_generators/generate_umath.py b/numpy/core/code_generators/generate_umath.py
--- a/numpy/core/code_generators/generate_umath.py
+++ b/numpy/core/code_generators/generate_umath.py
@@ -844,8 +844,8 @@ def english_upper(s):
 'isfinite':
     Ufunc(1, 1, None,
           docstrings.get('numpy.core.umath.isfinite'),
-          None,
-          TD(nodatetime_or_obj, out='?'),
+          'PyUFunc_IsFiniteTypeResolver',
+          TD(noobj, out='?'),
           ),
 'signbit':
     Ufunc(1, 1, None,


Test patch:
diff --git a/numpy/core/tests/test_datetime.py b/numpy/core/tests/test_datetime.py
--- a/numpy/core/tests/test_datetime.py
+++ b/numpy/core/tests/test_datetime.py
@@ -2208,6 +2208,27 @@ def test_isnat_error(self):
                 continue
             assert_raises(TypeError, np.isnat, np.zeros(10, t))
 
+    def test_isfinite(self):
+        assert_(not np.isfinite(np.datetime64('NaT', 'ms')))
+        assert_(not np.isfinite(np.datetime64('NaT', 'ns')))
+        assert_(np.isfinite(np.datetime64('2038-01-19T03:14:07')))
+
+        assert_(not np.isfinite(np.timedelta64('NaT', "ms")))
+        assert_(np.isfinite(np.timedelta64(34, "ms")))
+
+        res = np.array([True, True,  False])
+        for unit in ['Y', 'M', 'W', 'D',
+                     'h', 'm', 's', 'ms', 'us',
+                     'ns', 'ps', 'fs', 'as']:
+            arr = np.array([123, -321, "NaT"], dtype='<datetime64[%s]' % unit)
+            assert_equal(np.isfinite(arr), res)
+            arr = np.array([123, -321, "NaT"], dtype='>datetime64[%s]' % unit)
+            assert_equal(np.isfinite(arr), res)
+            arr = np.array([123, -321, "NaT"], dtype='<timedelta64[%s]' % unit)
+            assert_equal(np.isfinite(arr), res)
+            arr = np.array([123, -321, "NaT"], dtype='>timedelta64[%s]' % unit)
+            assert_equal(np.isfinite(arr), res)
+
     def test_corecursive_input(self):
         # construct a co-recursive list
         a, b = [], []