Repository: pandas-dev/pandas
Pull request: ENH: support downcasting of nullable EAs in pd.to_numeric
- [x] closes #33013
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Picking up #33435.

In `pd.to_numeric` when encountering an `IntegerArray` and `FloatingArray` (or `Series` built from them) we downcast `_data` then reconstruct the array using the downcast `_data` and the `_mask` from the original array.

Problem statement:
ENH: Support downcasting of nullable dtypes in to_numeric
This currently does not yet work:

```
In [5]: pd.to_numeric(pd.Series([1, 2, 3], dtype="Int64"), downcast='integer') 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-64a04efaaac0> in <module>
----> 1 pd.to_numeric(pd.Series([1, 2, 3], dtype="Int64"), downcast='integer')

~/scipy/pandas/pandas/core/tools/numeric.py in to_numeric(arg, errors, downcast)
    179             for dtype in typecodes:
    180                 if np.dtype(dtype).itemsize <= values.dtype.itemsize:
--> 181                     values = maybe_downcast_to_dtype(values, dtype)
    182 
    183                     # successful conversion

~/scipy/pandas/pandas/core/dtypes/cast.py in maybe_downcast_to_dtype(result, dtype)
    141         dtype = np.dtype(dtype)
    142 
--> 143     converted = maybe_downcast_numeric(result, dtype, do_round)
    144     if converted is not result:
    145         return converted

~/scipy/pandas/pandas/core/dtypes/cast.py in maybe_downcast_numeric(result, dtype, do_round)
    234                     return new_result
    235             else:
--> 236                 if np.allclose(new_result, result, rtol=0):
    237                     return new_result
    238 

<__array_function__ internals> in allclose(*args, **kwargs)

~/miniconda3/envs/dev/lib/python3.7/site-packages/numpy/core/numeric.py in allclose(a, b, rtol, atol, equal_nan)
   2169 
   2170     """
-> 2171     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2172     return bool(res)
   2173 

<__array_function__ internals> in isclose(*args, **kwargs)

~/miniconda3/envs/dev/lib/python3.7/site-packages/numpy/core/numeric.py in isclose(a, b, rtol, atol, equal_nan)
   2268 
   2269     xfin = isfinite(x)
-> 2270     yfin = isfinite(y)
   2271     if all(xfin) and all(yfin):
   2272         return within_tol(x, y, atol, rtol)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
```

I suppose because under the hood a conversion to object dtype array is happening somewhere.

Hints from issue discussion:
take
Interestingly, this worked with pandas < version 1.0. I recently upgraded our existing code base from 0.2x to 1.1 and now the existing and unmodified code fails with exactly this error. So, this is a regression:

Before 1.0:
```
In [7]: pd.to_numeric(pd.Series([42], dtype="Int64"), downcast="integer")
Out[7]: 
0    42
dtype: int8

In [8]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.5.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.7.12-arch1-1
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.3
numpy            : 1.19.1
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.1.1
setuptools       : 47.1.1
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : 7.17.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
```

After:
```
In [2]: pd.to_numeric(pd.Series([42], dtype="Int64"), downcast="integer")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-0969c270a2a4> in <module>
----> 1 pd.to_numeric(pd.Series([42], dtype="Int64"), downcast="integer")

/usr/lib/python3.8/site-packages/pandas/core/tools/numeric.py in to_numeric(arg, errors, downcast)
    180             for dtype in typecodes:
    181                 if np.dtype(dtype).itemsize <= values.dtype.itemsize:
--> 182                     values = maybe_downcast_to_dtype(values, dtype)
    183 
    184                     # successful conversion

/usr/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in maybe_downcast_to_dtype(result, dtype)
    149         dtype = np.dtype(dtype)
    150 
--> 151     converted = maybe_downcast_numeric(result, dtype, do_round)
    152     if converted is not result:
    153         return converted

/usr/lib/python3.8/site-packages/pandas/core/dtypes/cast.py in maybe_downcast_numeric(result, dtype, do_round)
    241                     return new_result
    242             else:
--> 243                 if np.allclose(new_result, result, rtol=0):
    244                     return new_result
    245 

<__array_function__ internals> in allclose(*args, **kwargs)

/usr/lib/python3.8/site-packages/numpy/core/numeric.py in allclose(a, b, rtol, atol, equal_nan)
   2187 
   2188     """
-> 2189     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2190     return bool(res)
   2191 

<__array_function__ internals> in isclose(*args, **kwargs)

/usr/lib/python3.8/site-packages/numpy/core/numeric.py in isclose(a, b, rtol, atol, equal_nan)
   2286 
   2287     xfin = isfinite(x)
-> 2288     yfin = isfinite(y)
   2289     if all(xfin) and all(yfin):
   2290         return within_tol(x, y, atol, rtol)

TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

In [3]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : d9fff2792bf16178d4e450fe7384244e50635733
python           : 3.8.5.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.7.12-arch1-1
Version          : #1 SMP PREEMPT Fri, 31 Jul 2020 17:38:22 +0000
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.1.0
numpy            : 1.19.1
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.1.1
setuptools       : 49.6.0
Cython           : 0.29.21
pytest           : 6.0.1
hypothesis       : 5.25.0
sphinx           : 3.2.0
blosc            : None
feather          : None
xlsxwriter       : 1.3.2
lxml.etree       : 4.5.2
html5lib         : 1.1
pymysql          : None
psycopg2         : 2.8.5 (dt dec pq3 ext lo64)
jinja2           : 2.11.2
IPython          : 7.17.0
pandas_datareader: 0.10.0dev0
bs4              : 4.9.1
bottleneck       : 1.3.2
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : 3.3.1
numexpr          : 2.7.1
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pyxlsb           : None
s3fs             : None
scipy            : 1.5.2
sqlalchemy       : 1.3.18
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None
```
Note that it didn't *fully* work (but didn't raise an error, for sure), as the resulting type is not a nullable dtype anymore. And when missing values were present, no downcasting happened at all:

```
In [2]: pd.to_numeric(pd.Series([42, None], dtype="Int64"), downcast="integer")   
Out[2]: 
0     42
1    NaN
dtype: Int64
```

Commit message: replace to_numpy use with _data

Patch to review:
diff --git a/pandas/core/tools/numeric.py b/pandas/core/tools/numeric.py
index 08cdfde7df58d..efa2c1165f789 100644
--- a/pandas/core/tools/numeric.py
+++ b/pandas/core/tools/numeric.py
@@ -7,6 +7,9 @@
     ensure_object,
     is_datetime_or_timedelta_dtype,
     is_decimal,
+    is_extension_array_dtype,
+    is_float_dtype,
+    is_integer_dtype,
     is_number,
     is_numeric_dtype,
     is_scalar,
@@ -15,6 +18,8 @@
 from pandas.core.dtypes.generic import ABCIndex, ABCSeries
 
 import pandas as pd
+from pandas.core.arrays.numeric import NumericArray
+from pandas.core.construction import extract_array
 
 
 def to_numeric(arg, errors="raise", downcast=None):
@@ -118,10 +123,14 @@ def to_numeric(arg, errors="raise", downcast=None):
     is_series = False
     is_index = False
     is_scalars = False
+    is_numeric_extension_dtype = False
 
     if isinstance(arg, ABCSeries):
         is_series = True
         values = arg.values
+        if is_extension_array_dtype(arg) and isinstance(values, NumericArray):
+            is_numeric_extension_dtype = True
+            values = extract_array(arg)
     elif isinstance(arg, ABCIndex):
         is_index = True
         if needs_i8_conversion(arg.dtype):
@@ -142,6 +151,12 @@ def to_numeric(arg, errors="raise", downcast=None):
     else:
         values = arg
 
+    if is_numeric_extension_dtype or (
+        is_extension_array_dtype(arg) and isinstance(values, NumericArray)
+    ):
+        is_numeric_extension_dtype = True
+        mask, values = values._mask, values._data
+
     values_dtype = getattr(values, "dtype", None)
     if is_numeric_dtype(values_dtype):
         pass
@@ -188,6 +203,16 @@ def to_numeric(arg, errors="raise", downcast=None):
                     if values.dtype == dtype:
                         break
 
+    if is_numeric_extension_dtype:
+        if is_integer_dtype(values):
+            from pandas.core.arrays import IntegerArray
+
+            values = IntegerArray(values, mask)
+        elif is_float_dtype(values):
+            from pandas.core.arrays import FloatingArray
+
+            values = FloatingArray(values, mask)
+
     if is_series:
         return arg._constructor(values, index=arg.index, name=arg.name)
     elif is_index:
diff --git a/pandas/tests/tools/test_to_numeric.py b/pandas/tests/tools/test_to_numeric.py
index f89958f7723ef..db947aafc5a07 100644
--- a/pandas/tests/tools/test_to_numeric.py
+++ b/pandas/tests/tools/test_to_numeric.py
@@ -725,3 +725,19 @@ def test_to_numeric_from_nullable_string(values, expected):
     s = Series(values, dtype="string")
     result = to_numeric(s)
     tm.assert_series_equal(result, expected)
+
+
+@pytest.mark.parametrize(
+    "data, input_dtype, downcast, expected_dtype",
+    (
+        ([1, 1], "Int64", "integer", "Int8"),
+        ([450, 300], "Int64", "integer", "Int16"),
+        ([1, 1], "Float64", "float", "Float32"),
+        ([1, 1], "Float64", "integer", "Int8"),
+    ),
+)
+def test_downcast_nullable_numeric(data, input_dtype, downcast, expected_dtype):
+    arr = pd.array(data, dtype=input_dtype)
+    result = pd.to_numeric(arr, downcast=downcast)
+    expected = pd.array(data, dtype=expected_dtype)
+    tm.assert_extension_array_equal(result, expected)


Code review comments:
diff_hunk: @@ -118,10 +123,14 @@ def to_numeric(arg, errors="raise", downcast=None):
     is_series = False
     is_index = False
     is_scalars = False
+    is_numeric_extension_dtype = False
 
     if isinstance(arg, ABCSeries):
         is_series = True
         values = arg.values
+        if is_extension_array_dtype(arg) and isinstance(values, NumericArray):
+            is_numeric_extension_dtype = True
+            values = extract_array(arg)
path: pandas/core/tools/numeric.py
text: @user1:
Is this extract_array line needed? (the `values = arg.values` from above should have worked fine, I think)

@author:
It's not, I reverted this bit