Repository: pandas-dev/pandas
Pull request: REGR: Assigning label with registered EA dtype raises
- [x] closes #38386
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Previous behavior in `io.parsers` depended on `is_bool_dtype("boolean")` erroneously raising `AttributeError`.

Test currently iterates over the names:

> ['category', 'interval', 'boolean', 'Float32', 'Float64', 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64', 'string']

It skips any EA where the name is a property; I don't see a good way to iterate over these in the test.

I only added a note about `is_bool_dtype` in the whatsnew as the `DataFrame.__setitem__` issue does not occur on 1.1.x. It didn't seem appropriate for any section besides other.

Problem statement:
REGR: assigning column with name that is registered as extension dtype errors
```
In [34]: df = pd.DataFrame([0])

In [35]: df["Int64"] = [1]
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/scipy/pandas/pandas/core/generic.py in _set_item(self, key, value)
   3822         try:
-> 3823             loc = self._info_axis.get_loc(key)
   3824         except KeyError:

~/scipy/pandas/pandas/core/indexes/range.py in get_loc(self, key, method, tolerance)
    353                     raise KeyError(key) from err
--> 354             raise KeyError(key)
    355         return super().get_loc(key, method=method, tolerance=tolerance)

KeyError: 'Int64'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-35-26205ed43ee5> in <module>
----> 1 df["Int64"] = [1]

~/scipy/pandas/pandas/core/frame.py in __setitem__(self, key, value)
   3161         else:
   3162             # set column
-> 3163             self._set_item(key, value)
   3164 
   3165     def _setitem_slice(self, key: slice, value):

~/scipy/pandas/pandas/core/frame.py in _set_item(self, key, value)
   3238         self._ensure_valid_index(value)
   3239         value = self._sanitize_column(key, value)
-> 3240         NDFrame._set_item(self, key, value)
   3241 
   3242         # check if we are modifying a copy

~/scipy/pandas/pandas/core/generic.py in _set_item(self, key, value)
   3824         except KeyError:
   3825             # This item wasn't present, just insert at end
-> 3826             self._mgr.insert(len(self._info_axis), key, value)
   3827             return
   3828 

~/scipy/pandas/pandas/core/internals/managers.py in insert(self, loc, item, value, allow_duplicates)
   1195 
   1196         # insert to the axis; this could possibly raise a TypeError
-> 1197         new_axis = self.items.insert(loc, item)
   1198 
   1199         if value.ndim == self.ndim - 1 and not is_extension_array_dtype(value.dtype):

~/scipy/pandas/pandas/core/indexes/numeric.py in insert(self, loc, item)
    172     def insert(self, loc: int, item):
    173         try:
--> 174             item = self._validate_fill_value(item)
    175         except TypeError:
    176             return self.astype(object).insert(loc, item)

~/scipy/pandas/pandas/core/indexes/numeric.py in _validate_fill_value(self, value)
    117         Convert value to be insertable to ndarray.
    118         """
--> 119         if is_bool(value) or is_bool_dtype(value):
    120             # force conversion to object
    121             # so we don't lose the bools

~/scipy/pandas/pandas/core/dtypes/common.py in is_bool_dtype(arr_or_dtype)
   1398         return arr_or_dtype.is_object and arr_or_dtype.inferred_type == "boolean"
   1399     elif is_extension_array_dtype(arr_or_dtype):
-> 1400         return getattr(arr_or_dtype, "dtype", arr_or_dtype)._is_boolean
   1401 
   1402     return issubclass(dtype.type, np.bool_)

AttributeError: 'str' object has no attribute '_is_boolean'
```

This is something that started to fail recently. Getting this from the geopandas CI testing with pandas master.

Hints from issue discussion:
I suppose this is caused by #38030 (@jbrockmendel)

That PR introduced:

https://github.com/pandas-dev/pandas/blob/3aa84479e54612eb58dd12d2d1f02674acc321c4/pandas/core/indexes/numeric.py#L173-L176

but the problem here is that the validation function is not raising a TypeError, but a different error. This, in turn, seems to be caused by a bug in `is_bool_dtype`:

```
In [36]: pd.api.types.is_bool_dtype("Int64")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-36-e7e8bfa4bbdd> in <module>
----> 1 pd.api.types.is_bool_dtype("Int64")

~/scipy/pandas/pandas/core/dtypes/common.py in is_bool_dtype(arr_or_dtype)
   1398         return arr_or_dtype.is_object and arr_or_dtype.inferred_type == "boolean"
   1399     elif is_extension_array_dtype(arr_or_dtype):
-> 1400         return getattr(arr_or_dtype, "dtype", arr_or_dtype)._is_boolean
   1401 
   1402     return issubclass(dtype.type, np.bool_)

AttributeError: 'str' object has no attribute '_is_boolean'
```
Agreed we should fix this in is_bool_dtype

Commit message: whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v1.2.0.rst b/doc/source/whatsnew/v1.2.0.rst
index af9219bc25931..3eefd8c237d43 100644
--- a/doc/source/whatsnew/v1.2.0.rst
+++ b/doc/source/whatsnew/v1.2.0.rst
@@ -856,7 +856,7 @@ Other
 - Bug in :meth:`Index.drop` raising ``InvalidIndexError`` when index has duplicates (:issue:`38051`)
 - Bug in :meth:`RangeIndex.difference` returning :class:`Int64Index` in some cases where it should return :class:`RangeIndex` (:issue:`38028`)
 - Fixed bug in :func:`assert_series_equal` when comparing a datetime-like array with an equivalent non extension dtype array (:issue:`37609`)
-
+- Bug in :func:`.is_bool_dtype` would raise when passed a valid string such as ``"boolean"`` (:issue:`38386`).
 
 
 .. ---------------------------------------------------------------------------
diff --git a/pandas/core/dtypes/common.py b/pandas/core/dtypes/common.py
index b4f6d587c6642..d8b0ad739b056 100644
--- a/pandas/core/dtypes/common.py
+++ b/pandas/core/dtypes/common.py
@@ -1397,7 +1397,7 @@ def is_bool_dtype(arr_or_dtype) -> bool:
         # guess this
         return arr_or_dtype.is_object and arr_or_dtype.inferred_type == "boolean"
     elif is_extension_array_dtype(arr_or_dtype):
-        return getattr(arr_or_dtype, "dtype", arr_or_dtype)._is_boolean
+        return getattr(dtype, "_is_boolean", False)
 
     return issubclass(dtype.type, np.bool_)
 
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
index 5b623c360c3ef..7a366fdda9dfe 100644
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1689,9 +1689,8 @@ def _convert_to_ndarrays(
                     values, set(col_na_values) | col_na_fvalues, try_num_bool=False
                 )
             else:
-                is_str_or_ea_dtype = is_string_dtype(
-                    cast_type
-                ) or is_extension_array_dtype(cast_type)
+                is_ea = is_extension_array_dtype(cast_type)
+                is_str_or_ea_dtype = is_string_dtype(cast_type) or is_ea
                 # skip inference if specified dtype is object
                 # or casting to an EA
                 try_num_bool = not (cast_type and is_str_or_ea_dtype)
@@ -1707,11 +1706,7 @@ def _convert_to_ndarrays(
                     or is_extension_array_dtype(cast_type)
                 ):
                     try:
-                        if (
-                            is_bool_dtype(cast_type)
-                            and not is_categorical_dtype(cast_type)
-                            and na_count > 0
-                        ):
+                        if not is_ea and na_count > 0 and is_bool_dtype(cast_type):
                             raise ValueError(f"Bool column has NA values in column {c}")
                     except (AttributeError, TypeError):
                         # invalid input to is_bool_dtype
diff --git a/pandas/tests/dtypes/test_common.py b/pandas/tests/dtypes/test_common.py
index ce6737db44195..19d80b714a674 100644
--- a/pandas/tests/dtypes/test_common.py
+++ b/pandas/tests/dtypes/test_common.py
@@ -545,6 +545,7 @@ def test_is_bool_dtype():
     assert not com.is_bool_dtype(pd.Series([1, 2]))
     assert not com.is_bool_dtype(np.array(["a", "b"]))
     assert not com.is_bool_dtype(pd.Index(["a", "b"]))
+    assert not com.is_bool_dtype("Int64")
 
     assert com.is_bool_dtype(bool)
     assert com.is_bool_dtype(np.bool_)
@@ -553,6 +554,7 @@ def test_is_bool_dtype():
 
     assert com.is_bool_dtype(pd.BooleanDtype())
     assert com.is_bool_dtype(pd.array([True, False, None], dtype="boolean"))
+    assert com.is_bool_dtype("boolean")
 
 
 @pytest.mark.filterwarnings("ignore:'is_extension_type' is deprecated:FutureWarning")
diff --git a/pandas/tests/frame/indexing/test_setitem.py b/pandas/tests/frame/indexing/test_setitem.py
index 884cb6c20b77e..e7d922f308200 100644
--- a/pandas/tests/frame/indexing/test_setitem.py
+++ b/pandas/tests/frame/indexing/test_setitem.py
@@ -1,6 +1,7 @@
 import numpy as np
 import pytest
 
+from pandas.core.dtypes.base import registry as ea_registry
 from pandas.core.dtypes.dtypes import DatetimeTZDtype, IntervalDtype, PeriodDtype
 
 from pandas import (
@@ -197,6 +198,22 @@ def test_setitem_extension_types(self, obj, dtype):
 
         tm.assert_frame_equal(df, expected)
 
+    @pytest.mark.parametrize(
+        "ea_name",
+        # property would require instantiation
+        [
+            dtype.name
+            for dtype in ea_registry.dtypes
+            if not isinstance(dtype.name, property)
+        ],
+    )
+    def test_setitem_with_ea_name(self, ea_name):
+        # GH 38386
+        result = DataFrame([0])
+        result[ea_name] = [1]
+        expected = DataFrame({0: [0], ea_name: [1]})
+        tm.assert_frame_equal(result, expected)
+
     def test_setitem_dt64_ndarray_with_NaT_and_diff_time_units(self):
         # GH#7492
         data_ns = np.array([1, "nat"], dtype="datetime64[ns]")


Code review comments:
diff_hunk: @@ -1707,11 +1706,7 @@ def _convert_to_ndarrays(
                     or is_extension_array_dtype(cast_type)
                 ):
                     try:
-                        if (
-                            is_bool_dtype(cast_type)
-                            and not is_categorical_dtype(cast_type)
-                            and na_count > 0
-                        ):
+                        if not is_ea and na_count > 0 and is_bool_dtype(cast_type):
path: pandas/io/parsers.py
text: @user1:
do we have something that actually hits this. you changed the comparison and nothing broke here? (e.g. to check for ea and not categorical).

@author:
This block is hit by 77 tests across io.excel and io.parser, for which 33 have `is_categorical_dtype(cast_type)` being True, all within `io.parser.test_dtypes.test_categorical_*`.

Note that `is_ea` is True whenever `is_categorical_dtype(cast_type)` is True, so what's changing here is that EA types which are not categorical are no longer raising if they are Boolean and have null values. Behavior for categorical is unchanged. I think this is appropriate because EA types are to define their own logic via `_from_sequence_of_strings` which is called within `_cast_types` that occurs immediately after this try-except.

diff_hunk: @@ -197,6 +198,22 @@ def test_setitem_extension_types(self, obj, dtype):
 
         tm.assert_frame_equal(df, expected)
 
+    @pytest.mark.parametrize(
+        "ea_name",
+        # property would require instantiation
+        [
+            dtype.name
+            for dtype in ea_registry.dtypes
+            if not isinstance(dtype.name, property)
+        ],
path: pandas/tests/frame/indexing/test_setitem.py
text: @user1:
Can you maybe just add a few hardcoded ones that are not covered now because of being properties, like `+ ["datetime64[ns, UTC]", "period[D]"]`, to be complete?

@author:
Certainly, good idea. Done.