BUG: read_csv with mixed bools and NaNs sometimes reads NaNs as 1.0 (#42808)
State: closed

When reading a column of mixed booleans and missing values into a specific floating dtype, for example `read_csv(..., dtype='float')`, missing values were being incorrectly converted to 1.0 rather than NaN. This patch fixes this issue.

- [x] closes #42808
- [x] closes #34120
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I've never been in the CSV parsing code before, so I've tried to put this change in the spot that makes the most sense and causes the least impact, but corrections are very welcome.

Why this issue was happening is that the [_try_bool_flex()](https://github.com/pandas-dev/pandas/blob/126a19d038b65493729e21ca969fbb58dab9a408/pandas/_libs/parsers.pyx#L1752) function parses a mixed column of bools and missing values into a uint8 array, storing False/True/missing as 0/1/255. It then returns a bool view of this array, "hiding" the missing values and making them just look like True. These missing values are meant to be later 'unhidden' by a call to [_maybe_upcast()](https://github.com/pandas-dev/pandas/blob/126a19d038b65493729e21ca969fbb58dab9a408/pandas/_libs/parsers.pyx#L1040), which views the array as uint8 again, and maps to a dtype=object array containing False/True/NaN.

However, when the user specifies an explicit float dtype, this upcast is skipped since the bool array gets converted to a float array before hitting _maybe_upcast(). This patch just replicates the upcast logic in this case.

Like I said above, this is the spot in the code that made the most sense to me to put the correction - I feel like a more robust fix would be to rewrite the `_convert_tokens()` function (and the methods it calls, and so on) to actually return the positions of the missing values, as well as the parsed values, which would be a large change.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv
name: Stale

Comments:
I've added a case pointed out in the issue for converting mixed-bools-and-nans to int, which was essentially the same issue as for float. It would be nice to actually have all of this type conversion logic in a single place in the parser, rather than spread out over various functions and conditionals. Is the type conversion logic (or rather, just should actually happen) written down anywhere? For instance one thing I found surprising is that when converting a CSV of `0\nNaN\nTrue\nFalse` to a dataframe with no dtype, we get back `([nan, True, False], dtype=object)`, but when actually specifying dtype=object in the call to `read_csv` we get back `([nan, 'True', 'False'], dtype=object)`, which is a different result.
cc @gfyoung 
@jreback comment(s) notwithstanding, these changes look good.
ping @joelgibson 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. Closing for now, but if interested in continuing please address the comments and merge conflict and we can reopen.