ENH: add NA scalar for missing value indicator, use in StringArray.
State: closed

xref https://github.com/pandas-dev/pandas/issues/28095, https://github.com/pandas-dev/pandas/issues/28778

This PR adds a `pd.NA` singleton with the behaviour as discussed in above issues. For now, it's only used in StringArray in this PR.

Labels:
name: Enhancement
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Any chance you could change StringDtype.na_value to use this, and see what breaks?

There are a few places where we use `np.nan` when we should be referring to `StringDtype.na_value` instead.

I suspect that `_libs.lib.Validator.is_valid_null` or `util.is_nan` will need to be updated to recognice pd.NA as NA. So not a big deal if it's a bunch of work to prototype.
core.missing and _libs.missing will need to recognize this
Playing with using this in StringArray, will push something (here or a PR against this) in a bit.

Anecdotally, having `__bool__` raise is kinda annoying since it breaks reductions like `all`. 

```python
(Pdb) pp left_value
<StringArray>
[NA, NA]
Length: 2, dtype: string
(Pdb) np.all(left_value == right_value)
*** TypeError: boolean value of NA is ambiguous
```

But maybe that's a good thing. I'd like to develop some more experience here.
@jbrockmendel I moved it to cython (well, still mostly copy paste of the python version into the cython file) and let missing functions recognize this (isna), so feedback on that part is very welcome
> Anecdotally, having __bool__ raise is kinda annoying since it breaks reductions like all.

We will need to rethink any/all in general as well. In principle, those are reducing operations, and reductions skip NAs, so your example should be equivalent to `all` of an empty Series/array. 
Although in the case of any/all, skipping NAs also doesn't fully feel as the best behaviour in many cases (eg for boolean array `all([True, NA]) == True` would be also be somewhat strange as the NA could be False, so you don't know the result). 
This also recently came up for object dtypes IIRC, and was planning to open a new issue for it (or find an existing).
If anyone wants a preview of what getting this working with StringArray looks like: https://github.com/jorisvandenbossche/pandas/pull/1/files.

It'll mean a larger diff, but I think we'll want to merge that PR against Joris' branch into here, so that we can get a feel for what actually using this is going to look like before merging it into master.
> I suspect that _libs.lib.Validator.is_valid_null or util.is_nan will need to be updated to recognice pd.NA as NA. So not a big deal if it's a bunch of work to prototype.

util.is_nan is specifically for float-nan.  The scalar-recognizing functions that need to recognize this are in _libs.missing.
Joris merged in my changes to make StringArray use `pd.NA`. It wasn't too large of a change so far.

I want to call out one behavior change we may want to make: Right now, `Series[string].str` methods that return numeric output (like `.str.count`) return either int or float dtype, depending on whether there are NAs. See https://github.com/pandas-dev/pandas/pull/29597/files#diff-683f8dacd08abda4913680fafe3a4ea7R3512.

I'm proposing that we change that behavior to always return an integer-na dtype in those cases for `string` dtype (no change to object dtype).
> I'm proposing that we change that behavior to always return an integer-na dtype in those cases for string dtype (no change to object dtype).

+1 

It's also not a "breaking" change, as string dtype is still new. In general, with those new dtypes I think we should try to return as much as possible also new dtypes for the result (eg also for boolean results, once boolean array has landed)
@jorisvandenbossche I'm not checking everything but it occurred to me - do the changes you have test things like `fillna()`, `dropna()` and `isna()`, and how do the semantics change with those methods when we used to have `np.nan` mean NA versus `pd.NA`?
@Dr-Irv so right now this only implements the NA object itself (and that it is recognized in `isna`), but it's not yet actually used much as missing value indicator. Right now only in the StringArray (which Tom added to this PR). So those things work (and are tested through the string dtype's tests):

```
In [1]: s = pd.Series(['a', pd.NA, 'c'], dtype='string') 

In [2]: s 
Out[2]: 
0      a
1    NaN
2      c
dtype: string

In [3]: s.isna()  
Out[3]: 
0    False
1     True
2    False
dtype: bool

In [4]: s.dropna()
Out[4]: 
0    a
2    c
dtype: string

In [5]: s.fillna('dd') 
Out[5]: 
0     a
1    dd
2     c
dtype: string

In [7]: np.asarray(s)  
Out[7]: array(['a', NA, 'c'], dtype=object)
```

(note the repr still shows it as "NaN", but it's actually pd.NA that is used, that's something we still need to fix)

For those methods, there should be no behavioural changes. It's eg in comparison ops that there is a change (the NA now gets propagated instead of being False):

```
In [8]: s == 'a' 
Out[8]: 
0     True
1      NaN
2    False
dtype: object   # this will become boolean (there is another PR open to add BooleanArray)

In [9]: s.astype(object) == 'a'   # the typical way where strings are stored as object dtype
Out[9]: 
0     True
1    False
2    False
dtype: bool
```


(I need to add those examples as docs!)
Merged in master, so this has the new behavior `.str` ops on StringArray returning an IntegerArray.
Thanks! I will try to push some docs on pd.NA here.
Do people have thoughts on the behavior of `float(pd.NA)`? We raise right now, which is probably the right move, but we could consider returning `np.nan`. I'm concerned that this conflates NaN and NA even more than we already do.

Right now, if you have a `np.array([0, 1, pd.NA], dtype="object")`, it's a bit hard to get a float dtype ndarray. You would need something like `arr[pd.isna(arr)] = np.nan; arr.astype(float)`

Assuming we don't want `float(NA)` to be `np.nan`, is `np.asarray(integer_array, dtype=float)` enough of a hint that users want to treat NA as NaN?
> Do people have thoughts on the behavior of float(pd.NA)

I think raising by default is certainly a reasonable option. Although if we have float array with mask-NA in the future, that might be even more annoying default behaviour there.

(and should add a test for this behaviour)

We should probably add options to `to_numpy` to be more flexible than the single choice of `__array__`? 
For example, could add a `NA_as_NaN` (or `na_as_nan`, `use_nan`, ..) option to indicate you want to use np.nan as the missing value indicator upon conversion to numpy.


> Do people have thoughts on the behavior of `float(pd.NA)`?

As I said on the call, the _semantics_ of `pd.NA` vs. `np.nan` need to be carefully handled in a variety of places.  IMHO, I think we go for the drastic change of saying that `np.nan` will no longer indicate a missing value, and that it only means "not a number" without any option to allow people to do it the "old" way. This also means that `np.isna()` is different than `pd.isna()`.  So in this case, either `float(pd.NA)` returns `pd.NA`, because `pd.NA` has no type, or we raise an exception.

Here's a bit radical idea to make the difference even more obvious, which is instead of using `pd.NA`, we call it `pd.MV` for "missing value", and change the API everywhere such that any keyword or method that uses the "NA" abbreviation is changed to "MV".
> So in this case, either `float(pd.NA)` returns pd.NA, because pd.NA has no type, or we raise an exception.

We can in principle control the behaviour of `foat(..)` (with `__float__`), but if we return something, I think it should return a float (that seems like the contract of `float(..)` to me). So then I would raise an error.

> Here's a bit radical idea to make the difference even more obvious, which is instead of using pd.NA, we call it pd.MV for "missing value",

The problem with that is that the big advantage of NA is that it is already used almost everywhere in our API (dropna, fillna, isna, skipna, ...) ..

> We should probably add options to to_numpy to be more flexible than the single choice of __array__?

Yes, I think that's best. Though the question remains: does `asarray(..., dtype="float")` imply `na_as_nan=True`?

If we add an `IntegerArray.to_numpy(dtype, na_as_nan=False)`, what's the interaction between dtype and na_as_nan?

* dtype=float, na_as_nan=T/F: fine
* dtype=int/bool, na_as_nan=T: raise regardless, since NaN isn't an int / bool? Or only if there are NAs?
* dtype=int/bool, na_as_nan=False: raise iff there are missing values, since NA isn't an int / bool?

All-in-all, for now I'm OK with float(NA) raising, and requiring explicit opt-in from users when converting from NA to NaN, via an `*Array.to_numpy(dtype=float, na_as_nan=True)`.
> > Here's a bit radical idea to make the difference even more obvious, which is instead of using pd.NA, we call it pd.MV for "missing value",
> 
> The problem with that is that the big advantage of NA is that it is already used almost everywhere in our API (dropna, fillna, isna, skipna, ...) ..

I get that, but maybe we create new methods (dropmv, fillmv, ismv, skipmv), etc., as the replacements, keep the old ones alive, and just map the "na" named methods to "mv" named methods, and tell people that the "na" based methods will be deprecated in the future.

I realize it's a bit radical, but my sense is that the only reason "na" was used was because numpy used "na", and pandas decided to make `np.nan` mean "missing value", but now's the time to make it really clear that `np.nan` and `pd.MV` are different.

> because numpy used "na",

Numpy does not use "na", but "nan" (np.isnan, np.nanmean, etc). I wasn't yet involved at the time, but I suppose the main reason was the terminology from R to use NA.

Personally, I am not convinced that changing all those APIs is worth that. 
> but if we return something, I think it should return a float

Python will raise if you don't return a float. So it's either NaN or raise on our own I think.

---

FWIW, I'd like to punt on the interaction of NA and NaN for now. It doesn't directly affect this issue, as we're only using NA in arrays that will not use NaN. And it's a complicated enough discussion that it warrants its own issue.

That said, @Dr-Irv if you think there are things that *are* related to this issue, then continue to raise them. But I think that something like changing the behavior of `isna`, `dropna`, isn't really on the table right now.
> FWIW, I'd like to punt on the interaction of NA and NaN for now

Yes, and the conversion to numpy arryay will only come up in a PR to use it in IntegerArray. 
And let's open a separate issue for NA / NaN
More comments on the *code* in this PR?

I know I need to fix divmod (should return two NAs), will try to fix that this evening.
> > FWIW, I'd like to punt on the interaction of NA and NaN for now
> 
> Yes, and the conversion to numpy arryay will only come up in a PR to use it in IntegerArray.
> And let's open a separate issue for NA / NaN

Fair enough.

IMHO, I think that converting `pd.NA` to anything (`str`, `float`, `int`) should always raise.  (Probably could debate about `str(pd.NA)`) .  And in terms of arrays, here's some interesting `numpy` behavior:
```
>>> import numpy as np
>>> np.array([5.0, 2.0, None, np.nan], dtype='float')
array([ 5.,  2., nan, nan])
>>> np.array([5.0, 2.0, None, np.nan])
array([5.0, 2.0, None, nan], dtype=object)
>>> np.array([5, 2, None, np.nan], dtype='int')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
>>> np.array([5, 2, np.nan], dtype='int')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: cannot convert float NaN to integer
```

So maybe the way that `numpy` handles `None` is a guide for the behavior we want in pandas when converting to `numpy` arrays?

> More comments on the code in this PR?

In case it helps, see https://github.com/TomAugspurger/pandas/compare/NA-scalar...TomAugspurger:NA-scalar+IntegerArray for integrating this into IntegerArray.

The only change that may be relevant here is the behavior of `int(NA)` and `float(NA)`. Previously, `int(np.nan)` raised a ValueError, since `np.nan` is a float, and some floats can be converted to int, but `np.nan` can't. I think that a TypeError is appropriate since there are no `NA` is the only instance of NAType.

So perhaps just tests for that.
Just to scope this PR, I think that using `pd.NA` in BooleanArray should be done as a followup PR.
I added some initial docs (whatsnew + section in missing_data
@jorisvandenbossche I've looked at your doc changes.  You may want to say something about that when you read data, the new `pd.NA` type won't be used, until it gets addressed by closing the issue I created #29752 .
cc @pandas-dev/pandas-core more comments on this?