ENH: Implement convert_dtypes
State: closed

- [x] xref #29752
- [x] tests added / passed
    - pandas/tests/series/test_dtypes.py:test_convert_dtypes
    - pandas/tests/frame/test_dtypes.py:test_convert_dtypes
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This implements `DataFrame.convert_dtypes()` and `Series.convert_dtypes()`, which will make it much easier to use the new `pd.NA` functionality.

Added documentation in the section about the new `pd.NA` functionality.

I'm sure there will be comments about how I could have done this in a more/better/different way, and I'm open to resolving them so we get this into 1.0.

Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Hello @Dr-Irv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-24 02:29:52 UTC
> @Dr-Irv conceptually this is ok,
> very quick glance
> 
> but impl needs work and api needs discussion (name)
> 
> the api should be more similar to the options provided in infer_objects and of.to_numeric

So `infer_objects` has no parameters, so I don't see what options would apply there.

For `df.to_numeric`, options currently are `errors` and `downcast`.  I don't think `errors` applies in this case (we're not parsing anything), but I could see that using the `downcast` idea for values of `integer`, `signed` and `unsigned` would apply.  Is that what you are suggesting?

I'm open on what name to use.  There was some discussion between @jorisvandenbossche and me in #29752 and this was my last suggestion.


> but I could see that using the downcast idea for values of integer, signed and unsigned would apply. Is that what you are suggesting?

That might be an option to add, but I don't that is a priority to add now (users can first do `to_numeric` to downcast what they want, before calling this new method. Which is a bit verbose but perfectly possible right now, so I would first focus on getting the basics right / agreed.

> I'm open on what name to use. There was some discussion between @jorisvandenbossche and me in #29752 and this was my last suggestion.

Initially I had some reservation to use "nullable" in the name, but actually I think this is OK. String dtypes where already "nullable" before, but it's not using pd.NA, and we can maybe try to use the term "nullable dtype" consistently for those new dtypes that use pd.NA. Then that should be fine. 

I would maybe only use nullable_**d**types (with the "d"), since that's the term that is used elsewhere in APIs in pandas (eg `dtypes` property, `dtype=` keyword, etc).

---

Something else I was wondering: does this need to be a method? 
Yes, a method is certainly more discoverable. But to me this doesn't feel like a typical operation, it's also kind of a temporary thing to try out (in awaiting it to be the default at some point), so a top-level function `pd.as_nullable_dtypes(..)` might also be fine?



@jreback @jorisvandenbossche So I have this all green now.  More detailed review and comments are welcome.
cc @TomAugspurger 
@jorisvandenbossche @WillAyd you seem to disagree on whether this should be on the 1.0.0 milestone.....
@TomAugspurger all green.  Made your suggested changes.  Should be easier code to read now.
@jreback can we take a step back and first discuss what we are trying to achieve with this function? (because based on your comments, there is clearly either a misunderstanding or a disagreement on the purpose of the new function)

I think for @Dr-Irv and me, the goal is to make it easier to experiment with the new nullable dtypes: the dtypes that use `pd.NA` as missing value indicator (so yes, for us this is "about NA"), so at this point in time the string, integer and boolean nullable dtypes. 

So the goal is not just to convert to *any* extension type. For example, the goal is not to convert an object column with timestamp objects with a timezone to a datetimetz dtype. First, because that's already what `infer_objects()` is for, and second because datetimetz is not a nullable dtype (in the sense of using pd.NA or having the same behaviour as dtypes with NA). 

So therefore, this method is not meant to replace `infer_objects` (which is specifically to convert accidental object dtyped columns to their proper dtype), and so we didn't talk about deprecating that. And that is also the reason for the specific name (and not something general as `convert_dtypes`). 


@jorisvandenbossche 

this function is too narrowly focused. a user searches the docs and sees ``.infer_objects`` and ``.as_nullable_dtypes``. ok which one shall I use? when should I use it.

If the purpose is to provide a convenient way to 'infer_dtypes', then let's simply do that with a few simple options to it. It is SO confusing that I somehow have a 'object' dtype, so we have a function to 'fix' it. The same with nullables, we want to 'fix' this too.

So as I said I would be +1 on ``.infer_dtypes``, which by default does what ``infer_object`` does now (and deprecate that), along with ``keeep_integer=True`` (needs a more informative name, does this mean ``convert to nullable`` or don't convert my integers?

having 2 functions which do a very similiar thing under different namespaces is very confusing.


In itself, I am certainly fine with adding the capabilities as an option to an existing function. And it's true that `infer_objects` does similar things (returning the same dataframe but with inferred dtypes). 

But, for me, a downside of adding it to `infer_objects()` is that the name "infer objects" does not fully cover what we want to do here, as the current `as_nullable_dtypes` in this PR does *more* than only inferring object columns. It eg also checks float columns to see if they can be nullable integer. 

Renaming `infer_objects` to `infer_dtypes` as you propose can indeed be an option to overcome that naming issue. But I am not sure it is worth it to deprecate `infer_objects` for this. I want to note that this function is already a new version of the before-deprecated `convert_objects`. Putting users through a new deprecation cycle for the same functionality feels unneeded.

But I want to stress again that for me the two *use cases* are rather distinct. The current `infer_objects` tries to fix up object dtypes that should have been other dtypes (numeric, datetime), but are not for whatever reason (eg from reading excel files this sometimes happens, from constructing and enlarging a dataframe in steps, etc). The idea of the new function in this PR is not to "fix" dtypes, but to convert perfectly fine, properly inferred dtypes to dtypes that support `pd.NA`. 
> Renaming infer_objects to infer_dtypes as you propose can indeed be an option to overcome that naming issue. But I am not sure it is worth it to deprecate infer_objects for this. I want to note that this function is already a new version of the before-deprecated convert_objects. Putting users through a new deprecation cycle for the same functionality feels unneeded.

how about ``.infer_dtypes``, we *can* allow the option to ``convert_object=False`` (or not, don't care much).

to me its about the naming; having nullable in the description is wildly misleading, this changes the entire patter we have about *_dtypes.

if they are orthogonal but have the same 'purpose', e.g. conversion of dtypes, then they should be named similarly.
> to me its about the naming; having nullable in the description is wildly misleading,

What is it that you find confusing? As in the current PR, the *only purpose* of the new function is to change to *nullable* dtypes. 
Of course, in the above I am using "nullable dtypes" as "dtypes that use pd.NA". Is that the confusing part? (the same question will come up then for a keyword name)
@jorisvandenbossche @jreback I have done a new commit that (hopefully) addresses concerns of @jreback.  I have left the name of the method alone, as I agree with @jorisvandenbossche .  I did change the parameter from `keep_integer` to `convert_integer` meaning that if True (the default), we convert integers to the integer extension types.

@jorisvandenbossche all green again
I think there are two API discussions going on here, reflecting the two (well, three) behaviors of this method

1. Converting messy object-dtype stuff into a proper dtype
2. Converting a "properly" dtyped  array like int64 or bool to a nullable array.
3. Converting a float-dtype array containing all integers to a nullable integer array.

Of these, only 1 overlaps with `infer_objects`. IMO the use-case we're solving for here is making it easy to opt into these new EAs. I don't think this should come at the cost of current users of `infer_objects` having to go through a deprecation cycle. Given that, I'm +1 on the addition of a new method dedicated to that, and I think `.as_nullable_dtypes` is a fine name for that.
@TomAugspurger all green with your last suggested changes.
@jreback I understand you completely disagree with Tom, Irv and me. But you can still disagree in a bit more friendly way, eg there is no need to call this "a mess".
@pandas-dev/pandas-core more thoughts on the API? (we are a bit stuck in a disagreement on the API) 

This PR is adding a `as_nullable_dtypes()` method on the DataFrame/Series, which is meant to convert to dtypes that use `pd.NA` (so right now, this is nullable integer, and the new boolean and string dtypes). The main goal is to make it easier for people to start experimenting with those new dtypes.

To avoid having to read everything, I think Tom's comment above is a good summary: https://github.com/pandas-dev/pandas/pull/30929#issuecomment-574387728 (and Jeff's arguments against the API in the current PR: https://github.com/pandas-dev/pandas/pull/30929#issuecomment-574154805)


Yea I think I agree with Jeff's comments here. From a usage perspective, I don't think the functionality of `infer_objects` and `as_nullable_types` is immediately apparent; `infer_dtypes` would seem a more natural way to do what is prescribed by both of those

Is the conversation around `keep_integer` to prevent going from say `np.int64` to `Int64`? 
Question: does `infer_dtypes` imply an `object -> *` conversion? To me, "infer" means you've got something uncertain (object dtype) and you're going to something certain (e.g. int dtype).

In this case, it's possibly a bit different. You might have an `int64` and you want to go to `Int64`. There's no inference there right? You know everything ahead of time. Similarly for `bool` -> `boolean`.

`as_nullable_dtypes` *does* have inference in some cases though (mix of int & float, object-dtype mix of bools and NA, etc.), but it's not *limited* to that.
> `as_nullable_dtypes` _does_ have inference in some cases though (mix of int & float, object-dtype mix of bools and NA, etc.), but it's not _limited_ to that.

So maybe it should be then? i.e. we would force a user to do something like `df.infer_objects().as_nullable_types()` ? The naming there still leaves a bit to be desired but I think mixing inference and converting from non-nullable to nullable is confusing
> The naming there still leaves a bit to be desired but I think mixing inference and converting from non-nullable to nullable is confusing

What's our goal for this API? We should agree on that before getting into details :)

IMO, the most important thing is to make it easy to use the new EAs, and I *think* that's best served by `as_nullable_types` serving both roles (converting both object dtypes and non-nullable to nullable), though concerns about violating the single-purpose principle are noted.
> What's our goal for this API? We should agree on that before getting into details :)
> 
> IMO, the most important thing is to make it easy to use the new EAs, and I _think_ that's best served by `as_nullable_types` serving both roles (converting both object dtypes and non-nullable to nullable), though concerns about violating the single-purpose principle are noted.

Here was my thought process on this issue, i.e., why I created it (#29752) and then did the PR
1) I believe that missing values are generally introduced when you read in data, and not via other computations.  So I doubt people would do `df.infer_objects().as_nullable_dtypes()`, because generally the DataFrame with columns that has data you'd want to convert would come from a reader, and generally, the objects in such readers are strings.
2) Our readers currently don't _easily_ support using the new extension dtypes that use `pd.NA`.  By easily, one benefit to me of the readers is the simplicity of using `read_csv` on data with lots of columns, and let pandas infer the dypes during the reading process, as opposed to listing the dtype of each column in the arguments to `read_csv`.
3) To encourage use of the new integer types `Int64`, etc., `BooleanDtype` and `StringDtype`, which support `pd.NA`, the `as_nullable_dtypes()` method makes it easy to just do things like `pd.read_csv("mydata.csv").as_nullable_dtypes()` and then you're ready to use the new dtypes.
4) At some point, a decision will need to be made as to whether the readers automatically use the new dtypes, either via an option or by default.  To some extent, how that behavior will work in the future is related to the current debate.

I suggested the name `as_nullable_types` (now `as_nullable_dtypes`), but am not wedded to it.

The matrix of options you can serve by allowing this to infer and convert to nullable is far less and more confusing than if you treat those operations separately. e.g. if a user wanted keep an object column as is while converting another column from int64 to Int64 that would be very difficult to achieve if you blend both roles into this function
> we would force a user to do something like df.infer_objects().as_nullable_types() ? The naming there still leaves a bit to be desired but I think mixing inference and converting from non-nullable to nullable is confusing
> ...
> The matrix of options you can serve by allowing this to infer and convert to nullable is far less and more confusing than if you treat those operations separately.

@WillAyd There is currently really *not* a lot of overlap between `infer_objects` and `as_nullable_dtypes`.

`infer_objects` is meant to infer a proper dtype from "messy objects" (that can be created due to a few reasons). So it does:

* python ints or floats as objects -> int / float numpy dtype
* timestamps/datetime.datetime as objects -> datetime64 dtype
* timedeltas/datetime.timedelta as objects -> timedelta64 dtype

`as_nullable_dtypes` on the other hand is about converting "normal" dtypes (mostly already inferred) to "nullable" dtypes. So it does:

* strings as objects -> nullable StringDtype (note this is *not* a case of "messy objects", as strings have always been stored in object dtype)
* numpy int dtype -> nullable IntDtype
* numpy bool dtype -> nullable BooleanDtype
* ints as numpy floats (with nans) -> nullable IntDtype  (this has inference on whether it are actually ints)

The only overlapping case is that `as_nullable_dtypes` also does:

* ints as objects (potentially with nan/None) -> nullable IntDtype

I am happy to ditch this specific case if that can resolve the concern of overlapping functionality / confusion that it both infers and converts, if this is a way to unblock this discussion. 
Although pragmatically speaking, I think this small overlap is really not that bad (and note, it still has a different result), and would avoid the need to first do a `infer_objects()` for this specific case.

>  e.g. if a user wanted keep an object column as is while converting another column from int64 to Int64 that would be very difficult to achieve if you blend both roles into this function

A user can always still just use `astype` where you can very specifically give the exact target dtype for each column. I don't think `as_nullable_dtypes` necessarily needs to provide this fine-grained control, it's for ease of use of the general case to try out the nullable dtypes.
Can we keep this on the 1.0.0 milestone until a definitive decision is made?

> 
> @WillAyd There is currently really _not_ a lot of overlap between `infer_objects` and `as_nullable_dtypes`.
> 
> `infer_objects` is meant to infer a proper dtype from "messy objects" (that can be created due to a few reasons). So it does:
> 
> * python ints or floats as objects -> int / float numpy dtype
> * timestamps/datetime.datetime as objects -> datetime64 dtype
> * timedeltas/datetime.timedelta as objects -> timedelta64 dtype
> 
> `as_nullable_dtypes` on the other hand is about converting "normal" dtypes (mostly already inferred) to "nullable" dtypes. So it does:
> 
> * strings as objects -> nullable StringDtype (note this is _not_ a case of "messy objects", as strings have always been stored in object dtype)
> * numpy int dtype -> nullable IntDtype
> * numpy bool dtype -> nullable BooleanDtype
> * ints as numpy floats (with nans) -> nullable IntDtype  (this has inference on whether it are actually ints)
> 
> The only overlapping case is that `as_nullable_dtypes` also does:
> 
> * ints as objects (potentially with nan/None) -> nullable IntDtype
> 

So after reading the above, maybe we should combine them and do what @jreback suggests.  Create `infer_dtypes(create_nullable_dtypes=True)` with the following behavior 
1) Convert all `object` to dtypes as `infer_objects()` does now
2) If `create_nullable_dtypes==True`, then do the behavior that `as_nullable_dtypes()` does as listed above, otherwise leave it alone.

And we don't include a `convert_integer` argument.  If `create_nullable_dtypes==False`, everything is left alone.  If `create_nullable_dtypes==True` then anything that looks like integer is converted to the new `Int*` types.  

In addition, we deprecate `infer_objects()`
> And we don't include a `convert_integer` argument.  If `create_nullable_dtypes==False`, everything is left alone.  If `create_nullable_dtypes==True` then anything that looks like integer is converted to the new `Int*` types.  

The same initial arguments for including a `convert_integer` keyword to `as_nulllable_dtypes` would be valid here as well, though. 

Personally, by combining both concepts in a single method, I am not sure if the single method becomes less confusing than by keeping it separate in two methods .. But, I don't want to block it over this aspect, so in the end I am fine either way. 
But as I said before, I think something like an `infer_dtypes` method for this would be fine, but I *don't* think it is worth deprecating `infer_objects` for.


I am actually ok with leaving ``infer_objects`` completely as is (we can decide later to deprecate or not); I think ``infer_dtypes`` would be good name. 


For me, the downsides of adding a method named like `infer_dtypes` are:

* We are adding a method that by default does exactly the same thing as an already existing method (`infer_objects()`), while to get the functionality that we actually want to add in this PR, you will need to do something like `df.infer_dtypes(use_nullable_dtypes=True)` (or do we want to set `use_nullable_dtypes` as the default?)
* The method will do more than "inferring", it will also convert already properly inferred dtypes (eg int64) to another dtype