ENH: add NDArrayBackedExtensionArray to public API
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
In the [`db-dtypes` package](https://github.com/googleapis/python-db-dtypes-pandas/issues/28) (time and date data types at the moment), we found `NDArrayBackedExtensionArray` to be quite useful. Unfortunately, this is a private API at the moment. Per https://github.com/googleapis/python-db-dtypes-pandas/issues/28#issuecomment-942752795, sending a PR to make this public.
no objections

cc @jbrockmendel @jorisvandenbossche 
I've been meaning to get around to this for a while.  Thanks @tswast for beating me to it!

One other thing we should ideally do is add a paragraph in the doc/source/development/extending.rst about this.  something to the effect of "if your EA is a thin wrapper around an ndarray, you can save some effort by using ..."
Added some tests and docs. Should be ready for review now.
I don't have any objections to this PR, but just to make it explicit: this means we will need to keep the `NDArrayBackedExtensionArray` API somewhat stable (while now it was internal we could change and refactor it however we wanted). 
(I don't know to what extent this has still changed the last year, though. Maybe it's already pretty stable)

The docs already mention `_box_func` and `_validate_scalar`, but there are also still a set of other additional methods compared to the base ExtensionArray class (`_from_backing_data`, `_simple_new`, and a few `_validate_.._value`-.

> (I don't know to what extent this has still changed the last year, though. Maybe it's already pretty stable)

The API doesn't appear to have changed much since pandas 0.24.2. We've been able to use that version and above in the db-dtypes package. https://github.com/googleapis/python-db-dtypes-pandas/blob/e9d41d17b5d6a7d83c46e2497feb8e314545adcb/setup.py#L33

> there are also still a set of other additional methods compared to the base ExtensionArray class (_from_backing_data, _simple_new, and a few _validate_.._value-.

I realized I do need `_from_backing_data` to support 2D arrays, so it's worth adding that to the docs in this PR.

I don't believe I've needed to use `_simple_new`, though perhaps I will as I work on getting db-dtypes passing against the compliance tests?

Not sure which `_validate` functions you're referring to? Perhaps these? https://github.com/googleapis/python-db-dtypes-pandas/blob/e9d41d17b5d6a7d83c46e2497feb8e314545adcb/db_dtypes/pandas_backports.py#L40-L43
@twast joris makes a good point.  ill spend some time this week going through the class and trying to collect/document which methods should be implemented by 3rd party authors
Summarizing some things that should be addressed before/while making this public

1) i think the methods that we need to document as needing overriding are
_box_func
_validate_scalar
_validate_setitem_value

2) Can we implement _validate_searchsorted_value in terms of the other methods?  If not, it needs to be added to 1).  The current `NDArrayBackedExtensionArray._validate_searchsorted_value` looks like it is correct for PandasArray but not the general case.
<b>update</b> Looks like we need _validate_searchsorted_value until 2.0, at which point we can use _validate_setitem_value.
<b>update 2</b> i dont think we can always reuse validate_setitem_value bc you could have e.g. a `PandasArray[int].searchsorted(ndarray[float])` which should work fine, but for which validate_setitem_value would raise.

3) ATM NDArrayBackedExtensionArray._quantile implicitly assumes that self._ndarray is reflects the ordering of the array where not NA.  This should either be documented as required or replaced with _values_for_argsort (which would itself require officializing the _values_for_argsort behavior to make this an allowable usage)

Thanks for looking into this @jbrockmendel ! We're not currently implementing `_validate_setitem_value` in db-dtypes, but perhaps when I get time to revisit https://github.com/googleapis/python-db-dtypes-pandas/pull/64 where I add the compliance tests, I will uncover the need for this.
Looks like `_internal_fill_value` needs to be documented as something to override, too.

https://github.com/pandas-dev/pandas/blob/e7a0cf3ba098c0074a4462522da5c7fdca804c78/pandas/core/arrays/_mixins.py#L109
am ok with merging this, but agree that we need a followup then to address https://github.com/pandas-dev/pandas/pull/45544#issuecomment-1052809232 and https://github.com/pandas-dev/pandas/pull/45544#issuecomment-1069349840
I lean towards trying to Get It Right in this PR rather than doing follow-ups, largely bc these days I'm having trouble keeping track of GH threads and the stati of follow-ups.
I'm okay with leaving this open until I figure out all that needs to be documented. I'm continuing to better my understanding as I go through the compliance test suite on the db-dtypes package.
Ready for another review.

I didn't quite understand the necessary action to address (3) in https://github.com/pandas-dev/pandas/pull/45544#issuecomment-1052809232 but I believe I've addressed everything else.
@jbrockmendel @jorisvandenbossche @TomAugspurger if comments.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.

Just pulled in the latest changes. Would love to get this documented, as it has been helpful in building the `dbtime` and `dbdate` dtypes in https://github.com/googleapis/python-db-dtypes-pandas
couple of comments, otherwise lgtm.  thanks for your patience
@tswast can you merge main.  some of the deprecations discussed above have been enforced
@jbrockmendel I've synced to main and addressed a remaining docs build warning.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
@tswast now that 2.2 is out the door, and now that Pandas 2.x in general has made huge strides in its ExtensionArray implementation(s), could we get this PR back to active status by merging in main and resubmitting?  If you don't have time for this, perhaps I can take a crack at it (with help from other Pint-Pandas people).  In that case, what is the git-friendly way for me to pick up where you left off?  We have an active use case that's ready to put this to the test.

CC: @andrewgsavage
@MichaelTiemannOSC I had a crack at it in my PR #56755. It's still active, just waiting on some help from pandas devs.
Thanks @MichaelTiemannOSC  and @andrewgsavage , I don't have time to revive this, so I appreciate your efforts.