Repository: pandas-dev/pandas

Issue:
Force boolean column to category while reading a csv
#### Problem description

Reading a csv file with dtypes specified in a dictionary produces NaNs for boolean columns read as category. It seems the output of dtypes changes from version 0.20 to 0.21 so that the below code produces NaNs for the second column. 

This might not really be a bug but it seems quite unintuitive to me as I would expect that output of df.dtypes should be useable to read it correctly again and I would expect the interpretation of booleans as a category as valid.

#### Code to reproduce the issue
```python
b = {'a':[5,4,3,2], 'b':[True, False, True, True], , 'c':['A', 'B', 'A', 'C']}
df = pd.DataFrame(b)

df['b'] = df['b'].astype('category')
df['c'] = df['c'].astype('category')
dtypes_dict = df.dtypes.to_dict()
df.to_csv("data.csv", index=False)

df2 = pd.read_csv("data.csv", dtype=dtypes_dict)
print(df2)
```

#### Produced Output
```
   a    b  c
0  5  NaN  A
1  4  NaN  B
2  3  NaN  A
3  2  NaN  C
```

#### Expected Output
```
   a    b  c
0  5  True  A
1  4  False  B
2  3  True  A
3  2  True  C
```
#### Output of df.dtypes version 0.22
```{'a': dtype('int64'), 'b': CategoricalDtype(categories=[False, True], ordered=False), 'c': CategoricalDtype(categories=['A', 'B', 'C'], ordered=False)}```

#### Output of df.dtypes version 0.20
```{'a': dtype('int64'), 'b': category, 'c': category}```

#### Version
Pands version 0.22
<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.8.6-040806-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_IE.UTF-8
LOCALE: en_IE.UTF-8

pandas: 0.22.0
pytest: None
pip: 9.0.2
setuptools: 39.0.1
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</details>



Discussion:
@svgsponer should the line `df['c'] = df['c'].astype('category')` be `df['c'] = df['b'].astype('category')`?
(changed a `c` to `b`)

Hmm perhaps not, because that won't get your expected output. Regardless, something seems to be going strange here.
Ahh, note the difference:

```python

In [15]: pd.read_csv(StringIO("A\nTrue\nFalse\nTrue"), dtype={"A": pd.api.types.CategoricalDtype(['True', 'False'])})
Out[15]:
       A
0   True
1  False
2   True

In [16]: pd.read_csv(StringIO("A\nTrue\nFalse\nTrue"), dtype={"A": pd.api.types.CategoricalDtype([True, False])})
Out[16]:
     A
0  NaN
1  NaN
2  NaN
```

We do some checking for numeric dtypes, I'm not sure about booleans.
The issue is likely in https://github.com/pandas-dev/pandas/blob/687cbe23b7e6c2129d1761f44da3d8acd923246b/pandas/core/arrays/categorical.py#L490-L508

Everything read by the CSV parser is a string. We do some checking for whether those strings should be converted to a specialized type (numeric, datetime-like). Pandas considers `Index([True, False])` to be object dtype, which is also used for strings, so it's skipped.

In https://github.com/pandas-dev/pandas/blob/687cbe23b7e6c2129d1761f44da3d8acd923246b/pandas/core/arrays/categorical.py#L516 we'll add a `if dtype.categories.is_boolean()`.

I'll get to this next week unless someone beats me to it.
Actually, I might just claim this issue to use for a talk I'm giving next week on how to contribute to pandas.
@TomAugspurger Sorry, forgot to add the 'c' column here after I edited my test code. It is updated now. `df['c'] = df['c'].astype('category')` should make more sense now and the code produces the provided example output.
Sorry, dropped the ball on this. Will try to resurrect my PR.

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -546,8 +546,9 @@ def base(self):
 
     @classmethod
     def _from_inferred_categories(cls, inferred_categories, inferred_codes,
-                                  dtype):
-        """Construct a Categorical from inferred values
+                                  dtype, true_values=None):
+        """
+        Construct a Categorical from inferred values.
 
         For inferred categories (`dtype` is None) the categories are sorted.
         For explicit `dtype`, the `inferred_categories` are cast to the
@@ -555,10 +556,12 @@ def _from_inferred_categories(cls, inferred_categories, inferred_codes,
 
         Parameters
         ----------
-
         inferred_categories : Index
         inferred_codes : Index
         dtype : CategoricalDtype or 'category'
+        true_values : list, optional
+            If none are provided, the default ones are
+            "True", "TRUE", and "true."
 
         Returns
         -------
@@ -567,27 +570,32 @@ def _from_inferred_categories(cls, inferred_categories, inferred_codes,
         from pandas import Index, to_numeric, to_datetime, to_timedelta
 
         cats = Index(inferred_categories)
-
         known_categories = (isinstance(dtype, CategoricalDtype) and
                             dtype.categories is not None)
 
         if known_categories:
-            # Convert to a specialzed type with `dtype` if specified
+            # Convert to a specialized type with `dtype` if specified.
             if dtype.categories.is_numeric():
-                cats = to_numeric(inferred_categories, errors='coerce')
+                cats = to_numeric(inferred_categories, errors="coerce")
             elif is_datetime64_dtype(dtype.categories):
-                cats = to_datetime(inferred_categories, errors='coerce')
+                cats = to_datetime(inferred_categories, errors="coerce")
             elif is_timedelta64_dtype(dtype.categories):
-                cats = to_timedelta(inferred_categories, errors='coerce')
+                cats = to_timedelta(inferred_categories, errors="coerce")
+            elif dtype.categories.is_boolean():
+                if true_values is None:
+                    true_values = ["True", "TRUE", "true"]
+
+                cats = cats.isin(true_values)
 
         if known_categories:
-            # recode from observation order to dtype.categories order
+            # Recode from observation order to dtype.categories order.
             categories = dtype.categories
             codes = _recode_for_categories(inferred_codes, cats, categories)
         elif not cats.is_monotonic_increasing:
-            # sort categories and recode for unknown categories
+            # Sort categories and recode for unknown categories.
             unsorted = cats.copy()
             categories = cats.sort_values()
+
             codes = _recode_for_categories(inferred_codes, unsorted,
                                            categories)
             dtype = CategoricalDtype(categories, ordered=False)
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1745,8 +1745,8 @@ def _cast_types(self, values, cast_type, column):
 
             cats = Index(values).unique().dropna()
             values = Categorical._from_inferred_categories(
-                cats, cats.get_indexer(values), cast_type
-            )
+                cats, cats.get_indexer(values), cast_type,
+                true_values=self.true_values)
 
         else:
             try:


Test patch:
diff --git a/pandas/tests/io/parser/test_dtypes.py b/pandas/tests/io/parser/test_dtypes.py
--- a/pandas/tests/io/parser/test_dtypes.py
+++ b/pandas/tests/io/parser/test_dtypes.py
@@ -324,6 +324,22 @@ def test_categorical_coerces_timedelta(all_parsers):
     tm.assert_frame_equal(result, expected)
 
 
+@pytest.mark.parametrize("data", [
+    "b\nTrue\nFalse\nNA\nFalse",
+    "b\ntrue\nfalse\nNA\nfalse",
+    "b\nTRUE\nFALSE\nNA\nFALSE",
+    "b\nTrue\nFalse\nNA\nFALSE",
+])
+def test_categorical_dtype_coerces_boolean(all_parsers, data):
+    # see gh-20498
+    parser = all_parsers
+    dtype = {"b": CategoricalDtype([False, True])}
+    expected = DataFrame({"b": Categorical([True, False, None, False])})
+
+    result = parser.read_csv(StringIO(data), dtype=dtype)
+    tm.assert_frame_equal(result, expected)
+
+
 def test_categorical_unexpected_categories(all_parsers):
     parser = all_parsers
     dtype = {"b": CategoricalDtype(["a", "b", "d", "e"])}