Repository: pandas-dev/pandas
Pull request: BUG: Fixed bug, where pandas._libs.lib.maybe_convert_objects function improperly handled arrays with bools and NaNs
- [x] closes #32146
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
REGR: Series repr of object Index with bools and NaN is wrong
#### Code Sample, a copy-pastable example if possible

```
import pandas as pd
# a series with booleans and nan
pd.Series([False,True,True,pd.NA]).value_counts(dropna=False)
True     2
False    1
True     1
dtype: int64

# check the actual index of the value_counts result
pd.Series([False,True,True,pd.NA]).value_counts(dropna=False).index
Index([True, False, nan], dtype='object')

# a similar Serie, with ints instead of nans, seems to work ok
pd.Series([0,1,1,pd.NA]).value_counts(dropna=False)
1.0    2
0.0    1
NaN    1
dtype: int64
```
#### Problem description
As shown in the example, the __repr__ of the value_counts result is apparently wrong when booleans are in the serie. It should report nan as a possible value, instead it maps it to True.
Note that this is limited to series with boolean. A similar example with ints works ok.


#### Output of ``pd.show_versions()``

<details>
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.1.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-76-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.2
setuptools       : 45.2.0
Cython           : None
pytest           : 5.3.5
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.4 (dt dec pq3 ext lo64)
jinja2           : None
IPython          : 7.12.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.3
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.3.5
pyxlsb           : None
s3fs             : None
scipy            : 1.4.1
sqlalchemy       : 1.3.13
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None

</details>

Hints from issue discussion:
take
Not sure if this is related but it seems similar:

`numpy.nan` shows as  `True` in index.

```python
>>> import pandas as pd
>>> import numpy as np
>>> s = pd.Series(range(3), index=[True, False, np.nan])
>>> s
True     0
False    1
True     2
dtype: int64
```

This `True`  is different from the regular `True` in the value count:

```python
>>> s.index.value_counts(dropna=False)
True     1
False    1
True     1
dtype: int64
```

But the correct value is still used "under the hood":
```python
>>> s.index.unique()
Index([True, False, nan], dtype='object')
```

Yes, so it is not directly related to `value_counts`, but a bug in the representation of an object index with NaN inside (@timschulz91's first example above):

```
>>> s = pd.Series(range(3), index=[True, False, np.nan])
>>> s
True     0
False    1
True     2
dtype: int64
```

And this seems to be a regression compared to 0.25, so tagged it as such (and with 1.0.2 milestone)
And the bug should be somewhere in here:

```
In [1]: idx = pd.Index([True, False, np.nan], dtype=object) 

In [2]: idx.format() 
Out[2]: ['True ', 'False', 'False']
```
And there, the `maybe_convert_objects` function is called, and it is this one that changed behaviour:

```
In [6]: pd._libs.lib.maybe_convert_objects(idx.values, safe=1)                                                                                                                                                     
Out[6]: array([ True, False,  True])
```
(this returns  `array([True, False, nan], dtype=object)` in 0.25.3)

I suppose this is caused by https://github.com/pandas-dev/pandas/pull/27335

Commit message: BUG: Fixed bug, where pandas._libs.lib.maybe_convert_objects function improperly handled arrays with bools and NaNs

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 13827e8fc4c33..83a52f9c162df 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -98,6 +98,7 @@ Performance improvements
 Bug fixes
 ~~~~~~~~~
 
+- Fixed bug where :func:`pandas._libs.lib.maybe_convert_objects` improperly handled arrays with bools and NaNs. (:issue:`32146`)
 
 Categorical
 ^^^^^^^^^^^
diff --git a/pandas/_libs/lib.pyx b/pandas/_libs/lib.pyx
index 1990ef66a6bf1..e2035738756b8 100644
--- a/pandas/_libs/lib.pyx
+++ b/pandas/_libs/lib.pyx
@@ -2279,7 +2279,7 @@ def maybe_convert_objects(ndarray[object] objects, bint try_float=0,
                                 return uints
                             else:
                                 return ints
-                elif seen.is_bool:
+                elif seen.is_bool and not seen.nan_:
                     return bools.view(np.bool_)
 
     return objects
diff --git a/pandas/tests/dtypes/test_inference.py b/pandas/tests/dtypes/test_inference.py
index 48ae1f67297af..c9e8ea3883cf1 100644
--- a/pandas/tests/dtypes/test_inference.py
+++ b/pandas/tests/dtypes/test_inference.py
@@ -568,6 +568,12 @@ def test_maybe_convert_objects_nullable_integer(self, exp):
 
         tm.assert_extension_array_equal(result, exp)
 
+    def test_maybe_convert_objects_bool_nan(self):
+        # GH32146
+        arr = pd.Index([True, False, np.nan], dtype=object)
+        exp = np.array([True, False, np.nan], dtype=object)
+        tm.assert_numpy_array_equal(lib.maybe_convert_objects(arr.values, safe=1), exp)
+
     def test_mixed_dtypes_remain_object_array(self):
         # GH14956
         array = np.array([datetime(2015, 1, 1, tzinfo=pytz.utc), 1], dtype=object)


Code review comments:
diff_hunk: @@ -568,6 +568,12 @@ def test_maybe_convert_objects_nullable_integer(self, exp):
 
         tm.assert_extension_array_equal(result, exp)
 
+    def test_maybe_convert_objects_bool_nan(self):
+        # GH32146
+        arr = pd.Index([True, False, np.nan], dtype=object)
+        exp = np.array([True, False, np.nan], dtype=object)
+        tm.assert_numpy_array_equal(lib.maybe_convert_objects(arr.values, safe=1), exp)
path: pandas/tests/dtypes/test_inference.py
text: @user1:
can you do `result = lib.maybe...` on its own line