Repository: pandas-dev/pandas
Pull request: BUG: Fix replacing in `string` series with NA (pandas-dev#32621)
The pd.NA values are replaced with np.nan before comparing the arrays/scalars

- [X] closes #32621 
- [X] tests passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Problem statement:
BUG: Replace in `string` series with NA
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np

rep = {'one': '1', 'two': '2'}
a = pd.Series(['one', 'two'], dtype='string')
b = pd.Series(['one', 'two', np.nan])
c = pd.Series(['one', 'two', np.nan], dtype='string')


# A: this works
a.replace(to_replace=rep)

# B: this also works
b.replace(to_replace=rep)

# C: this throws exception
c.replace(to_replace=rep)
# TypeError: Cannot compare types 'ndarray(dtype=object)' and 'str'
```
#### Problem description

`pandas.Series.replace` cannot be used in series of type `string` that contain `<NA>`

#### Expected Output

I would expect `C` to output the same as `B` but with `string` dtype

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.4.0-112-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : es_ES.UTF-8
LOCALE           : es_ES.UTF-8

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 41.4.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
@albertotb Thanks for the report!

Slightly related to https://github.com/pandas-dev/pandas/issues/32075

It seems this works with scalars:

```
In [18]: c.replace('one', '1')                                                                                                                                                                                     
Out[18]: 
0       1
1     two
2    <NA>
dtype: string
```

but not with lists/dicts
Hi, can I try to pick this up?
Yes, that would be welcome!
So, here's a few things I've noticed so far:

- When the dtype of a `pd.Series` is defined as a string, then the nan values of the array (which is to be stored under a series object) are converted to`pd.NA`. 
- When a dictionary/list is used to define the replacements (see variable `rep` in the first example) a different function is called underneath to apply those replacements. That's why the version specified by @jorisvandenbossche is working fine. Apparently, the replace function is very different in that case (not sure yet why that is though).
- The function used to find where to apply the replacements is using `operator.eq`. That operator fails to make the element-wise comparison between a value and pd.NA and returns a single scalar (instead of an array consisted of boolean values). And that's why the TypeError exception is raised.

I'm now investigating what's the best way of resolving this.
I also noticed another issue, have a look at this code:
```python
import pandas as pd
import numpy as np

replacements = {'one': '1', 'two': '2'}

series_a = pd.Series(['one', 'two'], dtype='string')
for rep in replacements:
    series_a = series_a.replace(rep, replacements[rep])

series_b = pd.Series(['one', 'two'], dtype='string')
series_b = series_b.replace(to_replace=replacements)
```
It produces the following output:
`In [4]: series_a`
`Out[4]:`
`0    1    `
`1    2`
`dtype: string`
`In [5]: series_b`
`Out[5]:`
`0    1`
`1    2`
`dtype: object`

Shouldn't they both return a `pd.Series` with a dtype of string?
Here's an explanation of my current solution (https://github.com/chrispe92/pandas/commit/47f66760879469dec6c0b8a5aa2a8d389e5ed99f).  In order to enable comparison with arrays containing `pd.NA`, I make sure to replace all of the missing values with `np.na`:
```python
# Replace all definitions of missing values (isna=True) to a numpy.nan
# Where x is an array of values
x = np.where(isna(x), np.nan, x)
```

Commit message: BUG: Fix replacing in `string` series with NA (pandas-dev#32621)

Added test for the reported bug

Patch to review:
diff --git a/pandas/core/internals/managers.py b/pandas/core/internals/managers.py
index 66e96af05eb71..ff0b4b66f8e90 100644
--- a/pandas/core/internals/managers.py
+++ b/pandas/core/internals/managers.py
@@ -1936,6 +1936,19 @@ def _compare_or_regex_search(a, b, regex=False):
     is_a_array = isinstance(a, np.ndarray)
     is_b_array = isinstance(b, np.ndarray)
 
+    def _get_nan_value(x):
+        if np.issubdtype(x.dtype, np.datetime64):
+            return np.datetime64("NaT")
+        elif np.issubdtype(x.dtype, np.timedelta64):
+            return np.timedelta64("NaT")
+        return np.nan
+
+    # Replace all definitions of missing values (isna=True) to a numpy.nan
+    if is_a_array:
+        a = np.where(isna(a), _get_nan_value(a), a)
+    if is_b_array:
+        b = np.where(isna(b), _get_nan_value(b), b)
+
     if is_datetimelike_v_numeric(a, b) or is_numeric_v_string_like(a, b):
         # GH#29553 avoid deprecation warnings from numpy
         result = False
diff --git a/pandas/tests/series/methods/test_replace.py b/pandas/tests/series/methods/test_replace.py
index 904a455870ab1..0cc4d0d146cc1 100644
--- a/pandas/tests/series/methods/test_replace.py
+++ b/pandas/tests/series/methods/test_replace.py
@@ -241,6 +241,13 @@ def test_replace2(self):
         assert (ser[6:10] == -1).all()
         assert (ser[20:30] == -1).all()
 
+    def test_replace_with_dictlike_and_string_dtype(self):
+        # GH 32621
+        s = pd.Series(["one", "two", np.nan], dtype="string")
+        expected = pd.Series(["1", "2", np.nan])
+        result = s.replace({"one": "1", "two": "2"})
+        tm.assert_series_equal(expected, result)
+
     def test_replace_with_empty_dictlike(self):
         # GH 15289
         s = pd.Series(list("abcd"))


Code review comments:
diff_hunk: @@ -1936,6 +1936,19 @@ def _compare_or_regex_search(a, b, regex=False):
     is_a_array = isinstance(a, np.ndarray)
     is_b_array = isinstance(b, np.ndarray)
 
+    def _get_nan_value(x):
path: pandas/core/internals/managers.py
text: @user1:
this is not a good strategy to do at all

rather NA should simply be handled 

@author:
Indeed, the behaviour of `pd.NA` seems to be causing the issue. For example:
```
>>> 1 == pd.NA
<NA>
>>> 1 == np.nan
False
```
The element-wise comparison seems to fail because equality check with `pd.NA` always returns `pd.NA` instead of a boolean value. However, looking at the documentation (https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html#operations) 
I see that this behaviour is intentional.

So, in which way do you suggest to handle the `pd.NA` values? In the current solution I resolve this by replacing the `pd.NA` with the "right" type of missing value in numpy (which is determined by the dtype of the array).

Thanks!

@author:
@user1: Do you mean something like the following?
```python
from pandas.core.api import NA
is_pd_NA = np.vectorize(lambda x: x is NA)
if is_a_array:
    a = np.where(is_pd_NA(a), _get_nan_value(a), a)
if is_b_array:
    b = np.where(is_pd_NA(b), _get_nan_value(b), b)
```

@user1:
so this is what you need to do.

compute the result (line 1956), then apply a mask

your mask is like this (may need adjustments if we have scalars)
```
mask = isna(a) | isna(b)

# import this at the top
na_value = pandas.core.dtypes.na_value_for_dtype(compat=False)

result[mask] = na_value
```

@author:
Shouldn't the mask also be applied to both arrays (a,b) before the computation of the result takes place? Otherwise the element-wise comparison is going to fail and the value of the result will be a scalar instead of an array. I will commit the suggested changes so that you can have a look.

See changes here: https://github.com/pandas-dev/pandas/blob/0405f6a369a7401bc59df1f61ee23e0c57345722/pandas/core/internals/managers.py#L1948