GH20591 read_csv raise ValueError for bool columns with missing values (C engine)
State: closed

- [X] closes #20591
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

As explained in the referenced issue, trying to cast missing values to bool dtype should result in a ValueError similar to when casting to int

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
Hello @JustinZhengBC! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/io/parsers.py`](https://github.com/JustinZhengBC/pandas/blob/a50c4b2fc2767c286e20f9d521d55d583c98d097/pandas/io/parsers.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/parser/test_na_values.py`](https://github.com/JustinZhengBC/pandas/blob/a50c4b2fc2767c286e20f9d521d55d583c98d097/pandas/tests/io/parser/test_na_values.py) !



##### Comment last updated on November 29, 2018 at 07:14 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=h1) Report
> Merging [#23968](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/43b2dabdcab50e0c83e37b0b8debcd579c539a9f?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23968/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23968      +/-   ##
==========================================
- Coverage   92.31%   92.31%   -0.01%     
==========================================
  Files         161      161              
  Lines       51513    51554      +41     
==========================================
+ Hits        47554    47591      +37     
- Misses       3959     3963       +4
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.71% <100%> (-0.01%)` | :arrow_down: |
| #single | `42.44% <0%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.38% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/arrays/timedeltas.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL3RpbWVkZWx0YXMucHk=) | `96% <0%> (-0.26%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.32% <0%> (-0.17%)` | :arrow_down: |
| [pandas/core/config.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29uZmlnLnB5) | `87.04% <0%> (-0.13%)` | :arrow_down: |
| [pandas/io/sas/sas\_xport.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNfeHBvcnQucHk=) | `90.14% <0%> (-0.1%)` | :arrow_down: |
| [pandas/io/formats/printing.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvcHJpbnRpbmcucHk=) | `93.01% <0%> (-0.08%)` | :arrow_down: |
| [pandas/plotting/\_core.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb3JlLnB5) | `83.58% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/computation/align.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vYWxpZ24ucHk=) | `97.84% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/reshape/melt.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZWx0LnB5) | `97.5% <0%> (-0.05%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `96.63% <0%> (-0.04%)` | :arrow_down: |
| ... and [15 more](https://codecov.io/gh/pandas-dev/pandas/pull/23968/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=footer). Last update [43b2dab...b8bca36](https://codecov.io/gh/pandas-dev/pandas/pull/23968?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Currently the python engine delegates type conversion to `np.ndarray.astype` in this case. Should it be overridden in this case for consistency with the C engine, or left as is so numpy users don't get unexpected behaviour?
> Currently the python engine delegates type conversion to np.ndarray.astype in this case. Should it be overridden in this case for consistency with the C engine, or left as is so numpy users don't get unexpected behaviour?

@gfyoung ?
> Currently the python engine delegates type conversion to np.ndarray.astype in this case. Should it be overridden in this case for consistency with the C engine, or left as is so numpy users don't get unexpected behaviour?

I would consider current behavior to be a bug.  Let's override so that we have consistent behavior.
@gfyoung I managed to do it in `_convert_data`. It requires iterating over the columns of `data` though (which happens again in `_convert_to_ndarrays`). I needed the try/catch block because `is_bool_dtype` throws ValueErrors on the line `dtype._is_boolean` when given inputs like "category" and "foo". Is that intended behaviour?
thanks @JustinZhengBC