Repository: pandas-dev/pandas
Pull request: ENH: Implement _from_sequence_of_strings for BooleanArray
- [x] closes #31131
- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Problem statement:
ENH: Implement CSV reading for BooleanArray
#### Code Sample, a copy-pastable example if possible

In 1.0.0 rc0, `pd.read_csv` returns this error when reading NA with specifying dtype of 'boolean'

```python
# import pandas as pd
# import io

>>> txt = """
X1,X2,X3
1,a,True
2,b,False
NA,NA,NA
"""
>>> df1 = pd.read_csv(io.StringIO(txt), dtype={'X3': 'boolean'}) # `pd.BooleanDtype()` also fails
raceback (most recent call last):
  File "pandas/_libs/parsers.pyx", line 1191, in pandas._libs.parsers.TextReader._convert_with_dtype
  File "/usr/local/lib/python3.7/site-packages/pandas/core/arrays/base.py", line 232, in _from_sequence_of_strings
    raise AbstractMethodError(cls)
pandas.errors.AbstractMethodError: This method must be defined in the concrete class type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 454, in _read
    data = parser.read(nrows)
  File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 1133, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python3.7/site-packages/pandas/io/parsers.py", line 2037, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 859, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 951, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 1083, in pandas._libs.parsers.TextReader._convert_column_data
  File "pandas/_libs/parsers.pyx", line 1114, in pandas._libs.parsers.TextReader._convert_tokens
  File "pandas/_libs/parsers.pyx", line 1194, in pandas._libs.parsers.TextReader._convert_with_dtype
NotImplementedError: Extension Array: <class 'pandas.core.arrays.boolean.BooleanArray'> must implement _from_sequence_of_strings in order to be used in parser methods
```

while "Int64" and "string" with NA can be correctly recognized.

```python
>>> df2 = pd.read_csv(io.StringIO(txt), dtype={'X1': 'Int64', 'X2': 'string'})
>>> df2
     X1    X2     X3
0     1     a   True
1     2     b  False
2  <NA>  <NA>    NaN
>>> df2.dtypes
X1     Int64
X2    string
X3    object
dtype: object
```

#### Problem description

NA literal for boolean is not parsed by `pd.read_csv`

#### Expected Output

```python
df3 = pd.read_csv(io.StringIO(txt), dtype={'X3': 'boolean'})
>>> df3
    X1   X2     X3
0  1.0    a   True
1  2.0    b  False
2  NaN  NaN   <NA>
>>> df3.dtypes
X1    float64
X2     object
X3    boolean
dtype: object

# and
df4 = pd.read_csv(io.StringIO(txt), dtype={'X1': 'Int64', 'X2': 'string', 'X3': 'boolean'})
>>> df4
     X1    X2     X3
0     1     a   True
1     2     b  False
2  <NA>  <NA>   <NA>
>>> df4.dtypes
X1     Int64
X2    string
X3   boolean
dtype: object
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.184-linuxkit
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.0rc0
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 44.0.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : None
</details>

Hints from issue discussion:
@koizumihiroo  try this
`df = pd.read_csv(io.StringIO(txt), dtype={'X3': 'bool'})`
@koizumihiroo We need to implement `BooleanArray._from_sequence_of_strings`. Are you interested in doing that?

@sach99in bool and boolean are different dtypes.

Commit message: Blacken

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 3bd86bb02155f..c720070b96969 100755
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -144,7 +144,7 @@ type dedicated to boolean data that can hold missing values. The default
 ``bool`` data type based on a bool-dtype NumPy array, the column can only hold
 ``True`` or ``False``, and not missing values. This new :class:`~arrays.BooleanArray`
 can store missing values as well by keeping track of this in a separate mask.
-(:issue:`29555`, :issue:`30095`)
+(:issue:`29555`, :issue:`30095`, :issue:`31131`)
 
 .. ipython:: python
 
diff --git a/pandas/core/arrays/boolean.py b/pandas/core/arrays/boolean.py
index eaa17df1235d3..6b1621cf946ee 100644
--- a/pandas/core/arrays/boolean.py
+++ b/pandas/core/arrays/boolean.py
@@ -1,5 +1,5 @@
 import numbers
-from typing import TYPE_CHECKING, Any, Tuple, Type
+from typing import TYPE_CHECKING, Any, List, Tuple, Type
 import warnings
 
 import numpy as np
@@ -286,6 +286,23 @@ def _from_sequence(cls, scalars, dtype=None, copy: bool = False):
         values, mask = coerce_to_array(scalars, copy=copy)
         return BooleanArray(values, mask)
 
+    @classmethod
+    def _from_sequence_of_strings(
+        cls, strings: List[str], dtype=None, copy: bool = False
+    ):
+        def map_string(s):
+            if isna(s):
+                return s
+            elif s in ["True", "TRUE", "true"]:
+                return True
+            elif s in ["False", "FALSE", "false"]:
+                return False
+            else:
+                return s
+
+        scalars = [map_string(x) for x in strings]
+        return cls._from_sequence(scalars, dtype, copy)
+
     def _values_for_factorize(self) -> Tuple[np.ndarray, Any]:
         data = self._data.astype("int8")
         data[self._mask] = -1
diff --git a/pandas/tests/arrays/test_boolean.py b/pandas/tests/arrays/test_boolean.py
index cc8d0cdcb518d..c32757f3aac46 100644
--- a/pandas/tests/arrays/test_boolean.py
+++ b/pandas/tests/arrays/test_boolean.py
@@ -251,6 +251,13 @@ def test_coerce_to_numpy_array():
         np.array(arr, dtype="bool")
 
 
+def test_to_boolean_array_from_strings(na_value):
+    result = BooleanArray._from_sequence_of_strings(["True", "False"])
+    expected = BooleanArray(np.array([True, False]), np.array([False, False]))
+
+    tm.assert_extension_array_equal(result, expected)
+
+
 def test_repr():
     df = pd.DataFrame({"A": pd.array([True, False, None], dtype="boolean")})
     expected = "       A\n0   True\n1  False\n2   <NA>"
diff --git a/pandas/tests/io/parser/test_dtypes.py b/pandas/tests/io/parser/test_dtypes.py
index d08c86bf2ae75..11dcf7f04f76b 100644
--- a/pandas/tests/io/parser/test_dtypes.py
+++ b/pandas/tests/io/parser/test_dtypes.py
@@ -550,3 +550,35 @@ def test_numeric_dtype(all_parsers, dtype):
 
     result = parser.read_csv(StringIO(data), header=None, dtype=dtype)
     tm.assert_frame_equal(expected, result)
+
+
+def test_boolean_dtype(all_parsers):
+    parser = all_parsers
+    data = "\n".join(
+        [
+            "a",
+            "True",
+            "TRUE",
+            "true",
+            "False",
+            "FALSE",
+            "false",
+            "NaN",
+            "nan",
+            "NA",
+            "null",
+            "NULL",
+        ]
+    )
+
+    result = parser.read_csv(StringIO(data), dtype="boolean")
+    expected = pd.DataFrame(
+        {
+            "a": pd.array(
+                [True, True, True, False, False, False, None, None, None, None, None],
+                dtype="boolean",
+            )
+        }
+    )
+
+    tm.assert_frame_equal(result, expected)


Code review comments:
diff_hunk: @@ -251,6 +251,13 @@ def test_coerce_to_numpy_array():
         np.array(arr, dtype="bool")
 
 
+def test_to_boolean_array_from_strings(na_value):
+    result = BooleanArray._from_sequence_of_strings(["True", "False"])
path: pandas/tests/arrays/test_boolean.py
text: @user1:
```suggestion
    result = BooleanArray._from_sequence_of_strings(np.array(["True", "False", np.nan], dtype=object))
```

(that's the type of object that it receives from the parser)