ENH: add use_nullable_dtypes option in read_parquet
State: closed

xref #29752, #30929

Using some work I am doing in pyarrow (https://github.com/apache/arrow/pull/6189), we are able to provide an option in `read_parquet` to directly use new nullable dtypes instead of first using the default conversion (eg which gives floats for ints with nulls) and doing the conversion afterwards



Labels:
name: IO Parquet
description: parquet, feather

Comments:
whatever keyword is agreed here we should use in read_csv as well. Also should have a doc-string reference to convert_dtypes
> FWIW I would also prefer use_extension_dtypes

Can you then try to provide some (counter) arguments? (I was initially also not fond of `use_nullable_dtypes`, so happy to find a better alternative).

For me, the main reason to not use `use_extension_dtypes` is: 1) this option does not trigger to return extension dtypes in general. For example, it does not trigger to return categorical or datetimetz (as those are aready returned by default by pyarrow), and it does not trigger to return period or interval (those can be returned based on metadata saved in the parquet file / pyarrow exension types); in both cases, extension types will be returned even with `use_extension_dtypes=False`. In contrast, I find `use_nullable_dtypes` clearer in communicating the intent*.
In addition, and more semantically, "extension" types can give the idea of being about "external" extension types (but this is a problem in general with the term, so not that relevant here).

\*I think we are going to need *some* terminology to denote *"the dtypes that use `pd.NA` as missing value indicator"*. Also for our communication (and when discussing) about it, for in the docs, etc, it would be good to have a term for it that we can consistently use. I think "nullable dtypes" is an option for this (we already use "nullable integer dtype" for a while in the docs), although certainly not ideal, since strictly speaking other dtypes are also "nullable" (floats, object, datetime), just in a different way. 
Maybe having this more general discussion can help us find matching keyword names afterwards.
Sure as some quick counter arguments:

  - The semantics are unclear to an end user; I would think most consider np.float to be nullable which this wouldn't affect
  - Some of the arguments for its clarity are specific to parquet, but I think become more ambiguous if we reuse the same keyword for other parsers (which I hope we would)
  - If we added more extension types in the future that aren't just focused on NA handling then we have to add another keyword on top of this to parse, which just makes things more complicated

The third point would probably the one I think is most of an issue
> If we added more extension types in the future that aren't just focused on NA handling then we have to add another keyword on top of this to parse, which just makes things more complicated

This seems unlikely to me. The issue here is that we have multiple representations for the same data (int64, Int64, bool, boolean, etc.), and we want to make it easy to use the new representation, not the one the dataset was written with. Do you foresee any other cases where we would want to do this transformation? The only one I potentially see is if we for some reason added a floating-point extension type. 
Note: I moved the general discussion above to the original issue: https://github.com/pandas-dev/pandas/issues/29752
@jorisvandenbossche status of this?
Well, for me this is ready, but I am not sure everybody is agreeing on the keyword name (including you?) 
I also pinged the more general discussion in https://github.com/pandas-dev/pandas/issues/29752
@jorisvandenbossche can you rebase.
Let's first decide in #29752
Updated with latest master, and addressed some comments (versionadded, additional tests)
linting issue
More comments on this?
thanks @jorisvandenbossche comment for followon issue.
I think #29752 already covers that for now. We first need more IO supporting it anyhow, before a global option would be useful)