API: Infer extension types in array
State: closed

* string
* integer

Closes #29791 (though we'll want to add BooleanArray).

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
CI is passing now.
small comment about a typo, otherwise LGTM
Can you add BooleanArray as well?
note that in #29791 should refer to teaching ``infer_dtype`` about extension types. This PR doesn't actually do that, rather you teach ``pd.array`` to correctly infer given the results of ``infer_dtype``. I actually think we *should* push this down and remove the logic from ``pd.array`` itself, BUT, I suspect that actually would break things (e.g. we would need to process 'nullable-integer') for example. So even though this is a positive change, I think we need to open an issue about this (or revise this PR).
LGTM
> #29791 should refer to teaching infer_dtype about extension types. This PR doesn't actually do that, rather you teach pd.array to correctly infer given the results of infer_dtype. I actually think we should push this down and remove the logic from pd.array

@jreback what would be the advantage of doing that? I personally don't really see the benefit of `infer_dtype` to be able to return both "integer" or "integer-nullable" for the same integer data. In the end, if you want to use the new dtype or a numpy dtype does not necessarily depend on the data, but from the context where it is being used (eg the context of `pd.array` wanting to return the new dtypes, so `pd.array` can decide how to interpret "integer", as numpy int or nullable int).

> > #29791 should refer to teaching infer_dtype about extension types. This PR doesn't actually do that, rather you teach pd.array to correctly infer given the results of infer_dtype. I actually think we should push this down and remove the logic from pd.array
> 
> @jreback what would be the advantage of doing that? I personally don't really see the benefit of `infer_dtype` to be able to return both "integer" or "integer-nullable" for the same integer data. In the end, if you want to use the new dtype or a numpy dtype does not necessarily depend on the data, but from the context where it is being used (eg the context of `pd.array` wanting to return the new dtypes, so `pd.array` can decide how to interpret "integer", as numpy int or nullable int).

you are missing the point
this PR is just a temporary work around - infer_dtype is used practically everywhere 
this is a very specific case that you are trying to solve

Jeff, if you think I am missing your point, then try to explain it better (but I rather think we are just disagreeing). 
In any case, it's not a discussion for this PR I think, so maybe you can open a new issue for this?
Fixed the docstring & merging master. I can open an issue about deduplicating `array`, `maybe_convert_object` and `infer_dtype`. I think that can be a followup though, as this PR really is only changing ~10 LOC, despite the large diff :)
> Fixed the docstring & merging master. I can open an issue about deduplicating `array`, `maybe_convert_object` and `infer_dtype`. I think that can be a followup though, as this PR really is only changing ~10 LOC, despite the large diff :)

right not trying to cause an issue :->

just wanting to clarify the apis & implementation we have for internal (maybe_convert_*) and external (pd.array) that need to be cleanly separated and useful / documented etc.

opening an issue would be great.
All green. https://github.com/pandas-dev/pandas/issues/29973 for the duplicate functionality between `convert_objects` and `array`.
thanks @TomAugspurger 
Thanks!

On Mon, Dec 2, 2019 at 11:38 AM Jeff Reback <notifications@github.com>
wrote:

> thanks @TomAugspurger <https://github.com/TomAugspurger>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/29799?email_source=notifications&email_token=AAKAOIU6GHJ3GX4CKEBVFFTQWVB25A5CNFSM4JQVETG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFUI4IY#issuecomment-560500259>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIUPLDBTRWAHMV4JKXTQWVB25ANCNFSM4JQVETGQ>
> .
>