BUG: Handle IntegerArray in pd.cut
State: closed

xref https://github.com/pandas-dev/pandas/issues/30944.
I think this doesn't close it, since only the pd.cut compoment
is fixed.

cc @jorisvandenbossche @jreback. The changes here attempt to be extremely conservative, since we're backporting stuff. I'm trying to not change behavior for anything other than IntegerArray. In particular, I'm not trying to support arbitrary EAs in `pd.cut`. This leads to some code that's fairly ugly & specific to IntegerArray. I think we should attempt to clean that up in 1.1.

Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
> i know you are going for compatibility, but can this be simplified?

The answer to the rest of the inline comments is the same as one: Not especially, *if* we're trying to make this change for just IntegerArray and not other objects. Swapping in `len(x)` for `x.size` definitely works for both EAs and ndarrays. But we may have been unknowingly relying on the `.size` raising an AttributeError to do a conversion to an ndarray, and so changing this (might) change behavior.
> > i know you are going for compatibility, but can this be simplified?
> 
> The answer to the rest of the inline comments is the same as one: Not especially, _if_ we're trying to make this change for just IntegerArray and not other objects. Swapping in `len(x)` for `x.size` definitely works for both EAs and ndarrays. But we may have been unknowingly relying on the `.size` raising an AttributeError to do a conversion to an ndarray, and so changing this (might) change behavior.

on the size issue

> But we may have been unknowingly relying on the `.size` raising an AttributeError to do a conversion to an ndarray, and so changing this (might) change behavior.

I would simply change this, if it breaks better to do it now
@jreback here's a POC for my proposed followup: https://github.com/pandas-dev/pandas/compare/master...TomAugspurger:cut-2?expand=1, which is much cleaner. The remaining TODO is to draft an API / expectations for what `ExtensionArray._ndarray_values` is for 3rd party EAs. For now, I fall back to `_factorize()` which has the right semantics but does more work than necessary.

But I'm uncomfortable including that large of changes in 1.0.0.
Question: what was the reason that `cut` worked before for IntegerArray? Because it converted it to an object array with NaNs / or to a float array? Just asking, but if that was the way it worked, the short term hack can be to convert IntegerArray again to that format. And then indeed do a proper fix later. 
> Because it converted it to an object array with NaNs / or to a float array?

It converted to object dtype with NaN. So, I suppose the most minimal fix is to just restore that. I'll modify this PR (and we still have https://github.com/pandas-dev/pandas/compare/master...TomAugspurger:cut-2?expand=1 to do things properly).


great thanks @TomAugspurger . do we have an issue for updating this to a more general soln?
https://github.com/pandas-dev/pandas/issues/31389

On Mon, Jan 27, 2020 at 7:52 PM Jeff Reback <notifications@github.com>
wrote:

> great thanks @TomAugspurger <https://github.com/TomAugspurger> . do we
> have an issue for updating this to a more general soln?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/31290?email_source=notifications&email_token=AAKAOIVKM6QZ7FJ6FQEEKKTQ76FWFA5CNFSM4KLKECIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKBXVSI#issuecomment-579041993>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKAOIWRHBHJGXON7GK563DQ76FWFANCNFSM4KLKECIA>
> .
>