Repository: pandas-dev/pandas
Pull request: BUG: Fix bug, where BooleanDtype columns are converted to Int64
- [x] closes #32287
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
BUG: convert_dtypes changes BooleanDtype to Int64
#### Code Sample, a copy-pastable example if possible

```python
>>> import pandas as pd
>>> df = pd.DataFrame(data=[["abc", 123, True]])
>>> print(df)
     0    1     2
0  abc  123  True
>>> print(df.dtypes)
0    object
1     int64
2      bool
dtype: object
>>> df = df.convert_dtypes()
>>> print(df)
     0    1     2
0  abc  123  True
>>> print(df.dtypes)
0     string
1      Int64
2    boolean
dtype: object
>>> df = df.convert_dtypes()
>>> print(df)
        0    1  2
0  b'abc'  123  1
>>> print(df.dtypes)
0    object
1     Int64
2     Int64
dtype: object

```
#### Problem description

Applying convert_dtypes() to a column with dtype `string` converts it to a column dtype 'object' (and the individual values from `str` type to `bytes` type).

Applying convert_dtypes() to a column with dtype `boolean` converts it to a column dtype 'Int64' (and the individual values from `bool` type to `int` type).

#### Expected Output

convert_dtypes() should keep StringDtype columns as StringDtype and BooleanDtype columns as BooleanDtype.

#### Output of ``pd.show_versions()``

<details>


pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.1.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : English_Malaysia.1252

pandas           : 1.0.1
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.2.3
setuptools       : 41.2.0
Cython           : None
pytest           : 5.3.5
hypothesis       : None
sphinx           : None
blosc            : None

</details>

Hints from issue discussion:
Oops, string bug is a duplicate and seems to be fixed, as per #31731. 
However, I can't find any mention of the boolean bug.
@jiannmeng Thanks for the report! 
The string one is indeed already fixed, the boolean not yet.
take

Commit message: Merge remote-tracking branch 'upstream/master' into fix_convert_dtypes_bool

Merge remote-tracking branch 'upstream/master' into fix_convert_dtypes_bool

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.2.rst b/doc/source/whatsnew/v1.0.2.rst
index 808e6ae709ce9..ebdf0c09e0b10 100644
--- a/doc/source/whatsnew/v1.0.2.rst
+++ b/doc/source/whatsnew/v1.0.2.rst
@@ -83,6 +83,7 @@ Bug fixes
 - Fixed bug in setting values using a slice indexer with string dtype (:issue:`31772`)
 - Fixed bug where :meth:`GroupBy.first` and :meth:`GroupBy.last` would raise a ``TypeError`` when groups contained ``pd.NA`` in a column of object dtype (:issue:`32123`)
 - Fix bug in :meth:`Series.convert_dtypes` for series with mix of integers and strings (:issue:`32117`)
+- Fixed bug in :meth:`DataFrame.convert_dtypes`, where ``BooleanDtype`` columns were converted to ``Int64`` (:issue:`32287`)
 
 .. ---------------------------------------------------------------------------
 
diff --git a/pandas/core/dtypes/cast.py b/pandas/core/dtypes/cast.py
index 7dac36b53fce5..7f2ecf470321b 100644
--- a/pandas/core/dtypes/cast.py
+++ b/pandas/core/dtypes/cast.py
@@ -70,6 +70,8 @@
 from pandas.core.dtypes.inference import is_list_like
 from pandas.core.dtypes.missing import isna, notna
 
+import pandas as pd
+
 _int8_max = np.iinfo(np.int8).max
 _int16_max = np.iinfo(np.int16).max
 _int32_max = np.iinfo(np.int32).max
@@ -1078,8 +1080,9 @@ def convert_dtypes(
                 inferred_dtype = input_array.dtype
 
         if convert_boolean:
-            if is_bool_dtype(input_array.dtype) and not is_extension_array_dtype(
-                input_array.dtype
+            if is_bool_dtype(input_array.dtype) and (
+                (not is_extension_array_dtype(input_array.dtype))
+                or (input_array.dtype == pd.BooleanDtype())
             ):
                 inferred_dtype = "boolean"
         else:
diff --git a/pandas/tests/series/methods/test_convert_dtypes.py b/pandas/tests/series/methods/test_convert_dtypes.py
index 17527a09f07a1..946b78184b395 100644
--- a/pandas/tests/series/methods/test_convert_dtypes.py
+++ b/pandas/tests/series/methods/test_convert_dtypes.py
@@ -279,3 +279,16 @@ def test_convert_string_dtype(self):
         )
         result = df.convert_dtypes()
         tm.assert_frame_equal(df, result)
+
+    def test_convert_bool_dtype(self):
+        # GH32287
+        df = pd.DataFrame([["abc", 123, True]])
+        exp_dtypes = pd.Series([np.object, np.int64, np.bool])
+        tm.assert_series_equal(df.dtypes, exp_dtypes)
+
+        df_1 = df.convert_dtypes()
+        exp_dtypes_1 = pd.Series([pd.StringDtype(), pd.Int64Dtype(), pd.BooleanDtype()])
+        tm.assert_series_equal(df_1.dtypes, exp_dtypes_1)
+
+        df_2 = df_1.convert_dtypes()
+        tm.assert_series_equal(df_2.dtypes, exp_dtypes_1)


Code review comments:
diff_hunk: @@ -1078,8 +1080,9 @@ def convert_dtypes(
                 inferred_dtype = input_array.dtype
 
         if convert_boolean:
-            if is_bool_dtype(input_array.dtype) and not is_extension_array_dtype(
-                input_array.dtype
+            if is_bool_dtype(input_array.dtype) and (
+                (not is_extension_array_dtype(input_array.dtype))
+                or (input_array.dtype == pd.BooleanDtype())
path: pandas/core/dtypes/cast.py
text: @user1:
Can this just be simplified to `is_bool_dtype`? 

@user2:
I think the issue is actually earlier. The condition at https://github.com/pandas-dev/pandas/pull/32490/files#diff-1df6c746d2a342e60cce09f774b566d7R1073-R1076 isn't wanted here.

Should this entire check be inside a big `if not is_extension_array_dtype(input_array.dtype)`? There aren't any cases where we want to convert from extension type to extension type, right? 

@user2:
And by "entire check" I mean immediately following the `if convert_string or convert_integer or convert_boolean` at https://github.com/pandas-dev/pandas/pull/32490/files#diff-1df6c746d2a342e60cce09f774b566d7R1054

diff_hunk: @@ -279,3 +279,16 @@ def test_convert_string_dtype(self):
         )
         result = df.convert_dtypes()
         tm.assert_frame_equal(df, result)
+
+    def test_convert_bool_dtype(self):
+        # GH32287
+        df = pd.DataFrame([["abc", 123, True]])
path: pandas/tests/series/methods/test_convert_dtypes.py
text: @user1:
A bit simpler test

```suggestion
        df = pd.DataFrame({"A": pd.array([True])})
        tm.assert_frame_equal(df, df.convert_dtypes())
```

And the rest can be removed.