ENH: ExtensionArray support for objects with _can_hold_na=False and relational operators
State: closed

closes #20659
closes #20761
- [x] tests added / passed
  - New tests/extensions/relobject tests arrays that cannot hold NaN, and has relational operators defined
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry
  - Not added as this is part of the ExtensionArray support coming in the next release, which already has a whatsnew entry

NOTE: I expect that the pandas main developers will object to a lot of what is done here!  So let's be open on the discussion.

## Goals of this pull request:

1. If there is an `ExtensionArray`, allow the relational operators to delegate to the base type
2. Modify `ExtensionArray` tests to handle the cases of `_can_hold_na==False` as best as possible.
3. An eventual goal is that if I have a `RelObjectArray` that consists of objects that have operators such as `__le__` defined that return objects, and `a` and `b` are `Series` containing that type of array, then `a <= b` returns a `Series` of objects rather than booleans.

### Things I'm unsure of:

- In core/ops.py, I am using `inspect` to determine whether the user has implemented the relational operators for the subclassed `ExtensionArray` type as well as for the base type of elements in the array. I'm not sure if this is best practice or not.
- In my example that I'm separately working on, the relational operators (including `__eq__`) return objects as opposed to booleans, and some (intentionally) throw exceptions.  So I tried to catch these various cases.  Not having `__lt__` defined means that sorting is undefined, and not having `__eq__` return a boolean messes up some things related to `groupby`.  For my application, that is OK.


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
> I am using inspect to determine whether the user has implemented the relational operators for the subclassed ExtensionArray type as well as for the base type of elements in the array. I'm not sure if this is best practice or not.

One issue here is for `__eq__` and `__neq__`. All EAs will implement those by virtue of inheriting from object, even if the author doesn't implement it "correctly". We could have EA authors opt into this by defining a class attributes like `_equatable`, `_orderable`, etc.
this won't be for 0.23, you are basically defining the ops protocol for abstraction, which is needed, but also needs substantial testing in the current EA classes.
> this won't be for 0.23

Yeah, I think I agree given the deadline we set (RC was supposed to be yesterday). I could see this taking a bit of time to get right.

@Dr-Irv could you split the changes for `_can_hold_na` out to a separate PR? That part seems ready to go.l
@TomAugspurger Yes, I can split the changes out for the `_can_hold_na` part.  And I see that the operator stuff I tried to do is part of a bigger discussion.  Should I create a new test case for the `_can_hold_na` part?
Yeah, that'd be nice.

It can be a very simple example I think... Maybe make a subclass of
`DecimalArray` with `_can_hold_na=False` and see if the tests pass once you
apply your changes?

On Tue, Apr 24, 2018 at 10:09 AM, Dr. Irv <notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> Yes, I can split the
> changes out for the _can_hold_na part. And I see that the operator stuff
> I tried to do is part of a bigger discussion. Should I create a new test
> case for the _can_hold_na part?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/20801#issuecomment-383968362>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIsmL3_cFNRvq4fhiEo-UBQz90xGfks5tr0AWgaJpZM4Tge6g>
> .
>

Closing this as it the `_can_hold_na` functionality is in #20815 .  ops functionality for some other time.