ENH: Add nullable dtypes to read_csv
State: closed

- [x] closes #36712 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
As a whole looks pretty good, though with the growing number of null-related parameters, I think we will at some point need to condense into one "super parameter" (e.g. `dict` of some sorts).
@lithomas1 Cool! Thanks for working on this.

We now also have a nullable float dtype nowadays. I know that this is less useful at this point (since float already supports missing values, in contrast to integer/boolean), but I think that a `use_nullable_dtypes` keyword should probably also result in nullable float.

EDIT: and actually also the nullable "string" dtype.

> As a whole looks pretty good, though with the growing number of null-related parameters, I think we will at some point need to condense into one "super parameter" (e.g. dict of some sorts).

@gfyoung I don't think this keyword is very related to the other read_csv null-related keywords? (or which ones were you thinking about?) And the `use_nullable_dtypes` keyword is also consistent with other IO methods (#29752, #31242)
> @gfyoung I don't think this keyword is very related to the other read_csv null-related keywords? (or which ones were you thinking about?) And the use_nullable_dtypes keyword is also consistent with other IO methods (#29752, #31242)

@jorisvandenbossche: They are related broadly speaking, but the consistency argument is fair as well.

I was just saying it's something to consider in the future, especially if we continue to add more arguments to an already long signature for `read_csv` (and the other I/O methods).
@jreback I think I will first try to get this feature-complete by adding in support for nullable float and string dtypes. Then I can spit out the inferencing changes. Marking this as draft in the meantime. 
Currently progress blocked on StringArray support by #40839
Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-05 02:36:55 UTC
@lithomas1 I think that the resulting dtype when using `use_nullable_dtypes=True` should not depend on whether NAs are present or not (IIUC the current state of the PR will eg only use nullable Int64 dtype if there is at least one missing value in the column, and otherwise return plain int64 dtype?) 
Having it depend on the presence if missing values introduces "value-dependent" behaviour, which we generally try to avoid. The nullable vs non-nullable dtype *can* behave somewhat differently, and so IMO it's better to return consistent dtypes. For example, assume you have a bunch of CSV files with the same structure (columns and types), then ideally you get the same inferred dtypes for each file (except for the corner case of all-missing data, of course).
the downside of that is it locks the user in to non-consolidated columns, with associated perf penalties.
@jorisvandenbossche That is correct, the nullable dtypes are not used unless null values appear in the data. 

Looking through the docs, when no NA values are present(for everything except StringArray/object), I don't think there really should be a difference between the nullable and non nullable dtypes.

So, given that dtype inference is inherently "value-dependent" and should return the "best dtypes", I still think that use_nullable_dtypes should only use the nullable dtypes when there are null values.

For StringArray vs object dtype for a column with all strings and no null values, I guess I'm fine with returning StringArray when this use_nullable_dtypes is True.
@jreback @gfyoung WDYT?

FYI, this is still blocked by #40839. I have a patch for that in this PR, but I will spit that out into its own precursor, given that it may be controversial.
@lithomas1 status of this?
Depends on #41412, but we need to discuss the correct behavior in #42973. 
@lithomas1 this might be easier these days?
Still stuck by at least #45168.
@lithomas1 status of this?
Thanks for the pull request, but it appears to have gone stale. Feel free to reopen when you have the time to revisit. Closing to clear out the queue.