Use new NA scalar in BooleanArray
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/29597 and https://github.com/pandas-dev/pandas/pull/29555, now actually using the `pd.NA` scalar in `BooleanArray`.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
One question that comes up here (but the same question applies to string and integer arrays): what "NA value" do we want to use when converting to object dtype numpy arrays? `None` or `pd.NA` ? 

In the initial BooleanArray PR, I used `None` (since `pd.NA` was not there yet), so you get a numpy array like `np.array([True, False, None])`. 
Now we can start using `pd.NA`, which is closer to the pandas representation (and what you would get from iteration or conversion to list `np.array([i for i in arr])`). But on the other hand, `None` can be easier to handle for cases where you need a numpy array (functionality that needs numpy arrays will typically also not recognize or handle correctly  `pd.NA`).


I'm not sure, but my initial preference is for having `pd.NA`, with an option to get other values for NA upon request. I think that means we should have a somewhat standard `to_numpy` method

```python
def to_numpy(self, dtype=object, na_value=pd.NA):
    ... 
```

Then if the user wants None / NaN, they can request it relatively easily.

> do we want to use when converting to object dtype numpy arrays?

What are the user-actions that hit this?

1. `np.array(boolarray, dtype=...)`
2. `boolarray.astype(np_dtype)`
3. ...?

Examples where the `pd.NA` in a numpy array gives problems: https://github.com/pandas-dev/pandas/issues/29976 (pyarrow's C conversion code does not know it) and the factorize errors in https://github.com/pandas-dev/pandas/pull/29964 (our cython hashing code for object arrays cannot handle it). 

Now, both are easy to solve (certainly since both can easily be solved on our side; the hashing code can recognize pd.NA and for the conversion to pyarrow we can ensure to use None before passing to pyarrow). But it are examples of how other code can break. 

Still not sure that we should therefore use None as the default in `__array__`, but at least I think it is important to have this `to_numpy(.., na_value=..)`.
@TomAugspurger updated this
See https://github.com/pandas-dev/pandas/issues/30043 for the CI failures. OK to ignore for now.
One more note: we'll need to handle reductions like `any` and `all`. They look somewhat buggy on master though.

```python
In [16]: pd.array([True, None])._reduce('all')
Out[16]: True

In [17]: pd.array([True, None])._reduce('all', skipna=False)
Out[17]: True

In [18]: pd.array([False, None])._reduce('all', )
Out[18]: False

In [19]: pd.array([False, None])._reduce('all', skipna=False)
Out[19]: False
```

Do we want to do that here or as a followup?
Let's do that as a follow-up, that's a remainder from the initial implementation, it's noted as a to do item in https://github.com/pandas-dev/pandas/issues/29556
Sounds good to me.
OK, going to merge this then, so the Kleene PR can then be updated. 
Once the IntegerArray PR is in, I will also take a look at consolidating both classes
I'll update the Kleene PR now, and the Integer one after if I have a chance.
I did a quick PR for the any/all: https://github.com/pandas-dev/pandas/pull/30062