Repository: pandas-dev/pandas
Pull request: API: Uses pd.NA in IntegerArray
Just a WIP for now. I haven't closely reviewed this since pd.NA was merged into master.

cc @jorisvandenbossche 

Closes https://github.com/pandas-dev/pandas/issues/15375

Problem statement:
ENH: Changing Series and DataFrame `repr` for NaN values
With future pandas internal improvements in contemplation, I have often wondered if it would be worth changing the `NaN` outputs to be `NA` or `NULL` instead to reflect the actual semantics of the data. This could be something that's configurable in `pandas.options` (i.e. showing the semantic value or the physical value).

Hints from issue discussion:
I would be in favor of that (NA instead of NaN). 
The only thing I am wondering is if we already know whether we want to make a distinction between NA and NaN for float data in pandas 2.0, where this would theoretically be possible when using bitmasks for NA. Because if we want that distinction, it might make sense to wait with the other repr until 2.0 instead 1.0.
It's somewhat out of scope for this issue, but I've been thinking about the NaN problem in pandas 2.0, and I think for backwards compatibility reasons we're going to get forced to make `NaN` and NA / NULL equivalent during the transition period. We could later add warnings when `NaN` is being treated as NA in operations like `s[...] = np.nan` (which I can attest litters people's pandas code), and then later add an option (where the default is that NaN and NA are differnet), and then later remove the option. 

I am pretty confident that using bitmaps everywhere will make our code much simpler and faster (i.e. can use SIMD operations on the bitmaps to deal with null analytics and propagation). 
As one example of why things will be faster, we can use bitmaps to eliminate branching in aggregations:

```c++
sum_x += values[i] * GetBit(bitmap, i);
```

compared with

```c++
if (values[i] == values[i]) sum_x += values[i];
```
Yes, we could have a 'nan_as_missing' option that is first True and could possibly later change (similar to how we now have `pd.options.mode.use_inf_as_null` option that is False by default).
The issue to discuss this is probably: https://github.com/pandas-dev/pandas2/issues/46

Commit message: fixup value counts

Patch to review:
diff --git a/pandas/core/arrays/integer.py b/pandas/core/arrays/integer.py
index 12b76df9a5983..22a6cdaef1d78 100644
--- a/pandas/core/arrays/integer.py
+++ b/pandas/core/arrays/integer.py
@@ -1,10 +1,10 @@
 import numbers
-from typing import Type
+from typing import Any, Tuple, Type
 import warnings
 
 import numpy as np
 
-from pandas._libs import lib
+from pandas._libs import lib, missing as libmissing
 from pandas.compat import set_function_name
 from pandas.util._decorators import cache_readonly
 
@@ -43,7 +43,7 @@ class _IntegerDtype(ExtensionDtype):
     name: str
     base = None
     type: Type
-    na_value = np.nan
+    na_value = libmissing.NA
 
     def __repr__(self) -> str:
         sign = "U" if self.is_unsigned_integer else ""
@@ -377,14 +377,28 @@ def __getitem__(self, item):
             return self._data[item]
         return type(self)(self._data[item], self._mask[item])
 
-    def _coerce_to_ndarray(self):
+    def _coerce_to_ndarray(self, dtype=None, na_value=libmissing.NA):
         """
         coerce to an ndarary of object dtype
         """
-
         # TODO(jreback) make this better
-        data = self._data.astype(object)
-        data[self._mask] = self._na_value
+        if dtype is None:
+            dtype = object
+        elif is_float_dtype(dtype) and na_value is libmissing.NA:
+            # XXX: Do we want to implicitly treat NA as NaN here?
+            # We should be deliberate in this decision.
+            na_value = np.nan
+
+        data = self._data.astype(dtype)
+
+        if (
+            is_integer_dtype(dtype)
+            and na_value is libmissing.NA
+            and not self._mask.any()
+        ):
+            return data
+        else:
+            data[self._mask] = na_value
         return data
 
     __array_priority__ = 1000  # higher than ndarray so ops dispatch to us
@@ -394,7 +408,7 @@ def __array__(self, dtype=None):
         the array interface, return my values
         We return an object array here to preserve our scalar values
         """
-        return self._coerce_to_ndarray()
+        return self._coerce_to_ndarray(dtype=dtype)
 
     def __arrow_array__(self, type=None):
         """
@@ -510,7 +524,7 @@ def isna(self):
 
     @property
     def _na_value(self):
-        return np.nan
+        return self.dtype.na_value
 
     @classmethod
     def _concat_same_type(cls, to_concat):
@@ -549,7 +563,7 @@ def astype(self, dtype, copy=True):
             return type(self)(result, mask=self._mask, copy=False)
 
         # coerce
-        data = self._coerce_to_ndarray()
+        data = self._coerce_to_ndarray(dtype=dtype)
         return astype_nansafe(data, dtype, copy=None)
 
     @property
@@ -604,12 +618,17 @@ def value_counts(self, dropna=True):
             # w/o passing the dtype
             array = np.append(array, [self._mask.sum()])
             index = Index(
-                np.concatenate([index.values, np.array([np.nan], dtype=object)]),
+                np.concatenate(
+                    [index.values, np.array([self.dtype.na_value], dtype=object)]
+                ),
                 dtype=object,
             )
 
         return Series(array, index=index)
 
+    def _values_for_factorize(self) -> Tuple[np.ndarray, Any]:
+        return self._coerce_to_ndarray(na_value=np.nan), np.nan
+
     def _values_for_argsort(self) -> np.ndarray:
         """Return values for sorting.
 
@@ -673,7 +692,8 @@ def _reduce(self, name, skipna=True, **kwargs):
         # coerce to a nan-aware float if needed
         if mask.any():
             data = self._data.astype("float64")
-            data[mask] = self._na_value
+            # We explicitly use NaN within reductions.
+            data[mask] = np.nan
 
         op = getattr(nanops, "nan" + name)
         result = op(data, axis=0, skipna=skipna, mask=mask, **kwargs)
diff --git a/pandas/tests/arrays/test_integer.py b/pandas/tests/arrays/test_integer.py
index 443a0c7e71616..d8ca06d83a50d 100644
--- a/pandas/tests/arrays/test_integer.py
+++ b/pandas/tests/arrays/test_integer.py
@@ -108,13 +108,17 @@ def test_repr_array_long():
 
 
 class TestConstructors:
+    def test_uses_pandas_na(self):
+        a = pd.array([1, None], dtype=pd.Int64Dtype())
+        assert a[1] is pd.NA
+
     def test_from_dtype_from_float(self, data):
         # construct from our dtype & string dtype
         dtype = data.dtype
 
         # from float
         expected = pd.Series(data)
-        result = pd.Series(np.array(data).astype("float"), dtype=str(dtype))
+        result = pd.Series(np.array(data, dtype="float"), dtype=str(dtype))
         tm.assert_series_equal(result, expected)
 
         # from int / list
@@ -464,7 +468,8 @@ def test_astype(self, all_data):
 
         # coerce to same numpy_dtype - mixed
         s = pd.Series(mixed)
-        with pytest.raises(ValueError):
+        with pytest.raises(TypeError):
+            # XXX: Should this be TypeError or ValueError?
             s.astype(all_data.dtype.numpy_dtype)
 
         # coerce to object
@@ -507,7 +512,7 @@ def test_frame_repr(data_missing):
 
     df = pd.DataFrame({"A": data_missing})
     result = repr(df)
-    expected = "     A\n0  NaN\n1    1"
+    expected = "    A\n0  NA\n1   1"
     assert result == expected
 
 
@@ -523,7 +528,7 @@ def test_conversions(data_missing):
     # we assert that we are exactly equal
     # including type conversions of scalars
     result = df["A"].astype("object").values
-    expected = np.array([np.nan, 1], dtype=object)
+    expected = np.array([pd.NA, 1], dtype=object)
     tm.assert_numpy_array_equal(result, expected)
 
     for r, e in zip(result, expected):
@@ -750,9 +755,11 @@ def test_reduce_to_float(op):
 def test_astype_nansafe():
     # see gh-22343
     arr = integer_array([np.nan, 1, 2], dtype="Int8")
-    msg = "cannot convert float NaN to integer"
+    # XXX: determine the proper exception here, from int(NA).
+    # msg = "cannot convert float NaN to integer"
+    msg = ""
 
-    with pytest.raises(ValueError, match=msg):
+    with pytest.raises(TypeError, match=msg):
         arr.astype("uint32")
 
 
diff --git a/pandas/tests/extension/test_integer.py b/pandas/tests/extension/test_integer.py
index d051345fdd12d..7900649675610 100644
--- a/pandas/tests/extension/test_integer.py
+++ b/pandas/tests/extension/test_integer.py
@@ -34,7 +34,7 @@
 
 
 def make_data():
-    return list(range(1, 9)) + [np.nan] + list(range(10, 98)) + [np.nan] + [99, 100]
+    return list(range(1, 9)) + [pd.NA] + list(range(10, 98)) + [pd.NA] + [99, 100]
 
 
 @pytest.fixture(
@@ -65,7 +65,7 @@ def data_for_twos(dtype):
 
 @pytest.fixture
 def data_missing(dtype):
-    return integer_array([np.nan, 1], dtype=dtype)
+    return integer_array([pd.NA, 1], dtype=dtype)
 
 
 @pytest.fixture
@@ -75,18 +75,18 @@ def data_for_sorting(dtype):
 
 @pytest.fixture
 def data_missing_for_sorting(dtype):
-    return integer_array([1, np.nan, 0], dtype=dtype)
+    return integer_array([1, pd.NA, 0], dtype=dtype)
 
 
 @pytest.fixture
 def na_cmp():
-    # we are np.nan
-    return lambda x, y: np.isnan(x) and np.isnan(y)
+    # we are pd.NA
+    return lambda x, y: x is pd.NA and y is pd.NA
 
 
 @pytest.fixture
 def na_value():
-    return np.nan
+    return pd.NA
 
 
 @pytest.fixture
@@ -94,7 +94,7 @@ def data_for_grouping(dtype):
     b = 1
     a = 0
     c = 2
-    na = np.nan
+    na = pd.NA
     return integer_array([b, b, na, na, a, a, b, c], dtype=dtype)
 
 
@@ -129,7 +129,7 @@ def _check_op(self, s, op, other, op_name, exc=NotImplementedError):
             expected = s.combine(other, op)
 
             if op_name in ("__rtruediv__", "__truediv__", "__div__"):
-                expected = expected.astype(float)
+                expected = expected.fillna(np.nan).astype(float)
                 if op_name == "__rtruediv__":
                     # TODO reverse operators result in object dtype
                     result = result.astype(float)
@@ -142,6 +142,11 @@ def _check_op(self, s, op, other, op_name, exc=NotImplementedError):
                 # combine method result in 'biggest' (int64) dtype
                 expected = expected.astype(s.dtype)
                 pass
+            if op_name == "__rpow__":
+                # TODO: https://github.com/pandas-dev/pandas/issues/29997
+                # pow(1, NA) is NA or 1?
+                pytest.skip("TODO-29997")
+
             if (op_name == "__rpow__") and isinstance(other, pd.Series):
                 # TODO pow on Int arrays gives different result with NA
                 # see https://github.com/pandas-dev/pandas/issues/22022
@@ -163,6 +168,7 @@ def test_error(self, data, all_arithmetic_operators):
 
 class TestComparisonOps(base.BaseComparisonOpsTests):
     def check_opname(self, s, op_name, other, exc=None):
+        pytest.skip(msg="TODO: NA comparisions")
         super().check_opname(s, op_name, other, exc=None)
 
     def _compare_other(self, s, data, op_name, other):


Code review comments:
diff_hunk: @@ -377,14 +377,28 @@ def __getitem__(self, item):
             return self._data[item]
         return type(self)(self._data[item], self._mask[item])
 
-    def _coerce_to_ndarray(self):
+    def _coerce_to_ndarray(self, dtype=None, na_value=libmissing.NA):
         """
         coerce to an ndarary of object dtype
         """
-
         # TODO(jreback) make this better
-        data = self._data.astype(object)
-        data[self._mask] = self._na_value
+        if dtype is None:
+            dtype = object
+        elif is_float_dtype(dtype) and na_value is libmissing.NA:
path: pandas/core/arrays/integer.py
text: @author:
We need to discuss this point. I think it comes up in something like `IntegerArray.astype(float)`. Should that raise or treat `NA` as `NaN`?

@user1:
On the one hand, if we want to distinguish NaN and NA later, it seems strange to now automatically convert NA to NaN. On the other hand, this is of course also just pragmatic and what most users will want.

(BTW, this might actually be a reason to use `lib._no_default` so you can explicitly ask for NA and not have it converted (which would error in this case))

diff_hunk: @@ -142,6 +142,11 @@ def _check_op(self, s, op, other, op_name, exc=NotImplementedError):
                 # combine method result in 'biggest' (int64) dtype
                 expected = expected.astype(s.dtype)
                 pass
+            if op_name == "__rpow__":
+                # TODO: https://github.com/pandas-dev/pandas/issues/29997
path: pandas/tests/extension/test_integer.py
text: @author:
Should `1 ** pd.NA` be 1 or `NA`? See https://github.com/pandas-dev/pandas/issues/29997.

R returns 1. Julia returns missing.

diff_hunk: @@ -163,6 +168,7 @@ def test_error(self, data, all_arithmetic_operators):
 
 class TestComparisonOps(base.BaseComparisonOpsTests):
     def check_opname(self, s, op_name, other, exc=None):
+        pytest.skip(msg="TODO: NA comparisions")
path: pandas/tests/extension/test_integer.py
text: @author:
Should I implement NA comparisons here, or leave that as a followup? Right now we have the `np.nan` always-false behavior.

```python
In [6]: pd.Series([1, 2, pd.NA], dtype="Int64") > 1
Out[6]:
0    False
1     True
2    False
dtype: bool
```

@user1:
I think you started adding it here? (which is fine I think), but so can remove this skip again. 
Although you might want to skip those tests fully given the deviating new behaviour, it might be more annoying than worth it to try to get those tests working. 
I see that in the BooleanArray PR I skip the two tests explicitly: https://github.com/pandas-dev/pandas/pull/29961/files#diff-c2269e718ef70677ad49e7aea9beda42R163-R170 (so to keep the third test which is still relevant, or if there are added others in the future)

diff_hunk: @@ -377,14 +377,28 @@ def __getitem__(self, item):
             return self._data[item]
         return type(self)(self._data[item], self._mask[item])
 
-    def _coerce_to_ndarray(self):
+    def _coerce_to_ndarray(self, dtype=None, na_value=libmissing.NA):
         """
         coerce to an ndarary of object dtype
         """
-
         # TODO(jreback) make this better
-        data = self._data.astype(object)
-        data[self._mask] = self._na_value
+        if dtype is None:
+            dtype = object
+        elif is_float_dtype(dtype) and na_value is libmissing.NA:
+            # XXX: Do we want to implicitly treat NA as NaN here?
+            # We should be deliberate in this decision.
+            na_value = np.nan
+
+        data = self._data.astype(dtype)
+
+        if (
+            is_integer_dtype(dtype)
+            and na_value is libmissing.NA
+            and not self._mask.any()
+        ):
+            return data
path: pandas/core/arrays/integer.py
text: @author:
I believe this retains the behavior on master of

```
np.asarray(IntegerArray, dtype=int)
```

succeeding if there are no NA values, and failing otherwise. Is that the behavior we want? If so, I'll ensure we have a better error message.


@user1:
> succeeding if there are no NA values, and failing otherwise. Is that the behavior we want?

I think so, yes.

diff_hunk: @@ -750,9 +755,11 @@ def test_reduce_to_float(op):
 def test_astype_nansafe():
     # see gh-22343
     arr = integer_array([np.nan, 1, 2], dtype="Int8")
-    msg = "cannot convert float NaN to integer"
+    # XXX: determine the proper exception here, from int(NA).
path: pandas/tests/arrays/test_integer.py
text: @author:
Should `int(NA)` be a TypeError or ValueError?

@user1:
`int(NA)` being a TypeError is the default that you get from Python (I suppose we could change this through `__index__`), but to me that seems fine to keep. 

In the astype, however, we can in principle have control over which error gets raised? 
I see that in Boolean.astype I explicitly check if the dtype is integer-like, and then raise a ValueError with a custom message if there are NAs present.