Implement NA.__array_ufunc__
State: closed

This gives us consistent comparisons with NumPy scalars.

I have a few implementation questions:

1. I wanted to reuse `maybe_dispatch_ufunc_to_dunder_op`. That was in `core/ops/dispatch.py`, but NA is defend in `_libs/missing.pyx`. For now I just moved it to `_libs/missing.pyx`, is there a better home for it? I don't think `_libs/ops.pyx` is an option, as it imports missing.
2. How should we handle comparisons with ndarrays? I see a few options:

  a. Return an `ndarray[object]`, where each element is NA
  b. Really take over and return our extension arrays. So `np.array([1, 2]) == pd.NA` would return an `IntegerArray([NA, NA])`. Boolean results would return a BooleanArray. And float results would return a ... PandasArray? (so maybe this isn't a good idea. But worth considering)
3. What scalars should we handle (in `__array_ufunc__` and in the regular methods)? Any of date time, Timestamp, time delta, Period, Interval? 


One item 2, I seem to have messed something up, as we return just a scalar `NA` right now. Will want to sort that out.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> How should we handle comparisons with ndarrays?

https://github.com/pandas-dev/pandas/pull/30245/commits/46f2327fa36dd110f5adae311dce487a06dc03f6 restores the behavior on master. We return an object-dtype ndarray filled with NA values. Not sure if that's the right choice though.
Anybody else seeing the CI check failure when linting `.pyx` files?

```
flake8 --version
3.7.9 (flake8-comprehensions: 3.1.4, mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.3 on Linux
Linting .py code
Linting .py code DONE
Linting .pyx code
Traceback (most recent call last):
  File "/home/runner/miniconda3/envs/pandas-dev/bin/flake8", line 11, in <module>
    sys.exit(main())

 ...

    self.formatter.handle(error)
  File "/home/runner/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/flake8/formatting/base.py", line 91, in handle
    line = self.format(error)
  File "/home/runner/miniconda3/envs/pandas-dev/lib/python3.7/site-packages/flake8/formatting/default.py", line 39, in format
    "col": error.column_number,
ValueError: unsupported format character ':' (0x3a) at index 41
```

Not able to reproduce locally with those versions.
All green now. Any thoughts on what, e.g. `np.array([True, False]) | pd.NA` should return? Our choices are an object-dtype ndarray, or a BooleanArray. Is it strange to return a non-ndarray?
> How should we handle comparisons with ndarrays? I see a few options:

Another option might also be to raise right now? (or is that also surprising?) That might be the safest option to allow choosing a more specific behaviour later. 

Otherwise returning an object array of NAs seems OK to me. But from numpy's point of view, such an object array is also strange, as you can't do anything boolean with it like boolean indexing.


> But from numpy's point of view, such an object array is also strange, as you can't do anything boolean with it like boolean indexing.

Right. An ndarray full of pd.NA is a fairly useless thing I think. That's my main motivation for considering anything fancy like returning an IntegerArray, etc.

We do have precedence for returning non-ndarrays from `__array_ufunc__`.

```python
In [8]: a = pd.array([2000, 2001], dtype="Period[D]")

In [9]: np.add(a, 1)
Out[9]: array([Period('2000-01-02', 'D'), Period('2001-01-02', 'D')], dtype=object)
```

Raising for ndarrays is also fine by me. Just need to choose whether a 0-dim ndarray is an array or scalar (probably a scalar).
> We do have precedence for returning non-ndarrays from __array_ufunc__.

What you show is actually returning a numpy object array (but I would agree that should/could return a PeriodArray)
Oh... it's too early here :) I could have sworn we returned an EA there.
Thanks for the review @jorisvandenbossche. Things should be good here.

I've deferred on returning anything too exotic from NA comparisons with ndarrays. We just return an ndarray of NA now.
should getting this in for 1.0 be a priority?
> should getting this in for 1.0 be a priority?

All open NA-related PRs are blockers for 1.0 I think
FYI I’ll be offline most of the next week. Feel free to push commits here, otherwise I’ll pick this up when I can. 
This should be good to go, assuming CI passes.
All green.
Planning to merge this later today if there aren't any objections.
comment about L482 vs L483, otherwise LGTM
Fixed. Merging once https://github.com/pandas-dev/pandas/pull/30245/files/0f4e121a229e3ede7d3b0fa4a974f5e2dcc8e30e#diff-c4b0cf199cc6d5bc2362ceb737159458 is resolved @jorisvandenbossche.
Merging in a few hours if there aren't any objections. Can revisit the line in https://github.com/pandas-dev/pandas/pull/30245/files/0f4e121a229e3ede7d3b0fa4a974f5e2dcc8e30e#diff-c4b0cf199cc6d5bc2362ceb737159458 if needed.
Checks are failing: the benchmark run failure seems unrelated, the typing failure looks unrelated as well but I am not very knowledgeable about that.
thanks @TomAugspurger very nice!