PERF/CLN: Improve datetime-like index ops perf
State: closed

Add `_isnan` to `DatetimeIndexOpsMixin` to cache `NaT` mask. 

This leads to some perf improvement which is noticeable in larger data.
### after fix

```
import pandas as pd
import numpy as np

np.random.seed(1)
idx = pd.DatetimeIndex(np.append(np.array([pd.tslib.iNaT]),
                       np.random.randint(500, 1000, size=100000000)))
%timeit idx.max()
1 loops, best of 3: 599 ms per loop

%timeit idx.min()
1 loops, best of 3: 608 ms per loop
```
### before fix:

```
%timeit idx.max()
1 loops, best of 3: 940 ms per loop

%timeit idx.min()
1 loops, best of 3: 883 ms per loop
```


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Performance
description: Memory or execution speed performance

Comments:
ohh nice!

do we have sufficient benches for this?

Updated top description. It is harder to notice in small size of `Index`. Shoud I add vbench for this? 

yes I would add a vbench for this.

btw, I think its actually useful to use `.hasnans` to skip some of the masking operations

`min_stamp = self[~self._isnan].asi8.min()`
can be

```
if self._hasnans:
    self = self[~self._isnan]
min_stamp = self.asi8.min()
```

on a NO nan series that would be a big win (assume that this is not the first operation, e.g. it obviously needs caching).

here's another thing. I think you can actually do a lot of this not in the constructor but in certain operations
(this may require you to keep another attribute)

e.g.

if you construct from a `date_range` you KNOW that you don't have any `NaT` already. 
On `insert/append` you can see if you have any nans (in the appended one and the existing), so you don't actually have to do the check.

So it can still be lazy but in some cases you can set it a-priori w/o actually doing the check.

@jreback Exactly! As a first step, added `hasnans` conditions before mask (`_isnan`)

> I think you can actually do a lot of this not in the constructor but in certain operations
> (this may require you to keep another attribute)

I assume it means adding new property to `_attributes` to show whether data has `NaT` or not. From my understanding, implementation should be:
- Adding a flag which shows whether the data MAY contains `NaT`, like `_maybe_hasnans`.
- Update the value to `False` when we can confirm the value DOESN'T contains `NaT`
- When `_maybe_hasnans` is `False`, `hasnans` returns `False` without checking `_isnan`.
- When `_maybe_hasnans` is `True`, `hasnans` checks `_isnan` and return the result, cache itself and change `_maybe_hasnans`.

@sinhrks yep, exactly. For instance, `Int64Index`, will always be `False` for this.

status?

@sinhrks status?

I couldn't undertake to add an attribute which indicates `NaN`` possibility. 

Because this is PR improve a perf somewhat, can I do it separately?

I think this needs to wait for `Index.fillna` yes?

Correct, will finish #11343 first.

@sinhrks I belive we can put this in now? move whatsnew note to 0.18.0

@sinhrks status?

@jreback Rebased, and ready for review.

I misunderstood about `fillna`. It can't be used because the target is `np.ndarray`. 

thanks!