BUG: get_indexer_non_unique with np.datetime64("NaT") and np.timedelta64("NaT") 
State: closed

- [x] closes #43869
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)
> @jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)

first guess is to check if https://github.com/numpy/numpy/issues/12550 is biting you.  i.e. when you construct `Index([np_nat_fixture], dtype=object)`, some numpy versions might be silently casting that np_nat_fixture to an integer.
> > @jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)
> 
> first guess is to check if [numpy/numpy#12550](https://github.com/numpy/numpy/issues/12550) is biting you. i.e. when you construct `Index([np_nat_fixture], dtype=object)`, some numpy versions might be silently casting that np_nat_fixture to an integer.

You were right -- it was because the older version of numpy was down-casting the NaT. I was able to get the CI passing except for Windows (timed-out) -- is this a consequence of performance?
> I was able to get the CI passing except for Windows (timed-out) -- is this a consequence of performance?

windows timeouts are pretty common on the CI; almost certainly unrelated
@jbrockmendel @jreback I was able to simplify all the na checks into one (float/complex, Decimal, np.datetime64(“NaT”) and np.timedelta64(“NaT”)) - and looks like performance increased as well!

```
asv continuous -f 1.1 upstream/master HEAD -b ^indexing


       before           after         ratio
     [c4cce9b7]       [da4300cc]
     <master>         <nui-dt64nat-td64nat>
-      97.9±0.4ms       85.7±0.7ms     0.88  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.Float64Index'>, 'nonunique_monotonic_inc')
-      99.9±0.5ms         87.4±2ms     0.87  indexing.NumericSeriesIndexing.time_getitem_array(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-     1.41±0.03μs      1.22±0.03μs     0.87  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Float32Engine'>, <class 'numpy.float32'>), 'monotonic_incr', True, 2000000)
-         102±1ms         83.8±1ms     0.82  indexing.NumericSeriesIndexing.time_getitem_lists(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-        99.3±1ms       80.7±0.7ms     0.81  indexing.NumericSeriesIndexing.time_getitem_array(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-         101±1ms       82.0±0.7ms     0.81  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-       101±0.9ms       82.2±0.9ms     0.81  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-         102±2ms         81.4±1ms     0.80  indexing.NumericSeriesIndexing.time_getitem_lists(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-      29.8±0.5ms       22.6±0.8ms     0.76  indexing.DataFrameNumericIndexing.time_loc_dups

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```
lgtm. @jbrockmendel good here?
LGTM really nice work here
fixed up some typos in the comments + adding test cases (failing on master, but this PR should resolve). I'll ping on green @jbrockmendel @jreback 
ping @jreback @jbrockmendel
thanks @alexhlim really nice!
> LGTM really nice work here

your suggestion of using sets + is_matching_na really helped me out!!
thanks for taking the time to review @jreback @jbrockmendel !