API: Public data for Series and Index: .array and .to_numpy()
State: closed

Closes https://github.com/pandas-dev/pandas/issues/19954.

TODO:

- [ ] update references to `.values` in the docs to use `.array` or `.to_numpy()`
- [ ] cross ref between `.values` and the rest

Labels:
name: API Design

Comments:
Hello @TomAugspurger! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/base.py`](https://github.com/TomAugspurger/pandas/blob/888853f484561caf3408531e4759c2b5f774152f/pandas/core/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/base.py`](https://github.com/TomAugspurger/pandas/blob/888853f484561caf3408531e4759c2b5f774152f/pandas/core/indexes/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/multi.py`](https://github.com/TomAugspurger/pandas/blob/888853f484561caf3408531e4759c2b5f774152f/pandas/core/indexes/multi.py) !

 - There are no PEP8 issues in the file [`pandas/tests/test_base.py`](https://github.com/TomAugspurger/pandas/blob/888853f484561caf3408531e4759c2b5f774152f/pandas/tests/test_base.py) !


Not having read the diff, first thought is that I’d like to wait for DatetimeArray to see what we can simplify vis-a-vis values/_values
Can you expand on that? `.values` isn't changing (aside from recommending
against it in the docs).

`.array` is implemented in terms of `._values` right now, but we can adjust
remove that level of indirection if we want,
and that will have to wait till DatetimeArray is done.

On Sun, Nov 11, 2018 at 1:22 PM jbrockmendel <notifications@github.com>
wrote:

> Not having read the diff, first thought is that I’d like to wait for
> DatetimeArray to see what we can simplify vis-a-vis values/_values
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/23623#issuecomment-437696946>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIs7PBHbB1FrTd8xFy5ZmGStpU4KCks5uuHj8gaJpZM4YYert>
> .
>

Not at all a well thought out opinion, just gut reaction.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=h1) Report
> Merging [#23623](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/43b2dabdcab50e0c83e37b0b8debcd579c539a9f?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23623/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23623      +/-   ##
==========================================
+ Coverage   92.31%   92.31%   +<.01%     
==========================================
  Files         161      161              
  Lines       51513    51526      +13     
==========================================
+ Hits        47554    47567      +13     
  Misses       3959     3959
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.71% <100%> (ø)` | :arrow_up: |
| #single | `42.48% <50%> (+0.05%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.68% <ø> (ø)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.84% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `95.51% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `97.64% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.03% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23623/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.48% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=footer). Last update [43b2dab...f9eee65](https://codecov.io/gh/pandas-dev/pandas/pull/23623?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

lgtm small comment.
I suppose this is not WIP anymore ? 
(I can do a review of the docs tomorrow)
Yeah, sorry. No rush. I probably won't be around tomorrow.
@jorisvandenbossche if you have any  comments
@TomAugspurger There is still an error in doc build in basics.rst (using `.array` on a DataFrame; not sure what the best solution is in that case)
In addition, similar question as in `pd.array` PR: how do we see our backwards compatibility guarantees on the actual returned object?
Assume we add a StringArray and make it the default for strings at some point, ideally this should be returned by the `array` property, otherwise the guarantee of "this is a direct access to underlying data" would be dependent again on the dtype. But that also is a break in back-compat. 
If we decide that we want to keep `.array` close to its meaning (and thus updating the return value if there are changes), we mention it in the docs? 
> In addition, similar question as in pd.array PR: how do we see our backwards compatibility guarantees on the actual returned object?

Added a small section on that. I wasn't sure where was most appropriate, so I put it in the docstring.
lgtm. let's merge.
Lifting the datetime / datetimetz discussion out here:

What do people think the return types should be for `.array` and `.to_numpy()`?

We know that

- `Series[datetime64[ns, tz]].array` needs to be a DatetimeArray.
- `Series[datetime64[ns]].to_numpy()` is just the same NumPy array.

The questions are for

1. `Series[datetime64[ns]].array`: DatetimeArray or ndarray?
2. `Series[datetime64[ns, tz]].to_numpy`: object-dtype ndarray of Timestamps? (I think this one is clear)

---

and I'l update the docs, but since we don't actually have DatetimeArray yet, the implementation will have to wait a bit (hoping to have a WIP PR up today or tomorrow)
``Series[datetime64[ns]].array: DatetimeArray``
``Series[datetime64[ns, tz]].to_numpy: object-dtype ndarray of Timestamps? ``

so agree (and .array should *always* be an Array if we can (and here we can)
Do time-zone naive datetimes use extension arrays?

I'd love to have a clear rule for `.array`:
- It's an ndarray for builtin types supported by numpy.
- It's an extension array for pandas extension arrays.
> Do time-zone naive datetimes use extension arrays?

That's being worked on right now. It's unclear to me how things will end up, but it's possible there will be an ExtensionArray between Series and the actual ndarray for tz-naive data. This will certainly be the case for DatetimeIndex.

However, I don't think this fact needs to be exposed to the user. It's primarily for code reuse between datetime-tz, datetime, and timedelta. (We haven't really talked about `Series[timedelta64[ns]].array`, but I presume it should follow the behavior of `datetime64[ns]`).

> I'd love to have a clear rule for .array:

> - It's an ndarray for builtin types supported by numpy.
> - It's an extension array for pandas extension arrays.

I like this rule.
> However, I don't think this fact needs to be exposed to the user. It's primarily for code reuse between datetime-tz, datetime, and timedelta. (We haven't really talked about Series[timedelta64[ns]].array, but I presume it should follow the behavior of datetime64[ns]).

In that case I suppose it mostly comes down to:
1. which choice is most useful to users -- will they want to manipulate these extension array objects directly or would they rather have base numpy arrays?
2. which choice is most future proof -- are we going to be happy sticking with this choice for the long term? We definitely don't want upheaval with `.array` like we've had with `.values`.
> which choice is most useful to users -- will they want to manipulate these extension array objects directly or would they rather have base numpy arrays?

"Ideally", if a user wants a numpy array, they use `to_numpy` or `np.array(..)`, and they should only use `.array` if they don't really care about the distinction between both, but want to do a certain operation on the values (eg an operation without alignment and then put it back in a Series). 
But of course, it is difficult to control what users use it for .. (there will be people starting to use `.array` to get a numpy array, the only way to avoid this is to raise an error in such a case and only return ExtensionArrays, but that of course defeats the general usecase of the property).

> which choice is most future proof -- are we going to be happy sticking with this choice for the long term? We definitely don't want upheaval with .array like we've had with .values

I suppose that returning DatetimeArray instead of ndarray[datetime64[ns]] will be more future proof. But, that also relates to what I commented above (https://github.com/pandas-dev/pandas/pull/23623#issuecomment-442007629) related the back-compat guarantees we make. If we want to keep following the rule in the future, I have the feeling that we will need to change the return value at some point.

> I'd love to have a clear rule for .array:
>
>        It's an ndarray for builtin types supported by numpy.
>        It's an extension array for pandas extension arrays.

Some other possibilities that would go further than the above rule: 

* It's an extension array if the round-trip (eg Series -> array -> Series) with ndarray would not be completely information-preserving and completely cheap. 
  * In practice this might be the same as the above rule at the moment, but eg a possible future StringArray (wrapping an object array of strings but guaranteeing every element is a string) could be converted rather faithfully to an object array, but the roundtrip to ndarray would loose some information (the fact every element is a string). For DatetimeArray you could say it looses the `freq` attribute, but since this is an attribute on the Array, and not metadata of the dtype, I would say this is less of a problem.

* It's an extension array if a ndarray would not support the same operations as a Series holding the data
  * For eg DatetimeArray vs ndarray[datetime64], not all arithmetic and other operations will yield the same results, and one could expect that `Series[datetime]` and `Series[datetime].array` behave the same 

The rule might also whether it is implemented under the hood as an ExtensionArray or not. But this is course not necessarily clear to the user (it might be hidden somewhat), and @shoyer I assume that your proposal for a rule would be to have a clear expectation for the user? (so all the above rules that I mention would be more complicated).
Would it make sense to make to consider having `.array` *always* return an ExtensionArray object?

I imagine we could pretty quickly whip up an ExtensionArray for each NumPy dtype that simply defers to the underlying NumPy array for every operation. Internally, we could recognize these `NumpyExtensionArray` objects and use base numpy operations.

I guess we could also just clearly document: `.array` means you are going into pandas' internals and is not guaranteed to be stable. We may change the return value from `.array` in the future.
@shoyer I think the main problem is that would basically already get rid off the block-based internals?  And I think that was more a change we were contemplating for 2.0 instead of 1.0. (apart from the additional work it would mean on the short term)
(unless we use such a NumpyExtensionArray only for wrapping the array when returning it in `.array`)
> (unless we use such a NumpyExtensionArray only for wrapping the array when returning it in .array)

This is all I was thinking of.
Hmm, having `.array` always be an ExtensionArray is an interesting proposal... It kind of makes "`.array` is the actual array stored in the Series" a lie, but maybe users don't care about that? I assume they care more about things like zero-copy and inplace modification, than they do about how pandas chooses to handle a particular dtype.
What do people think about doing the remaining items as followup?

1. Determine the signature for `.to_numpy()` (copy=True is uncontroversial. the rest I'm not sure about, but we could figure it out and do here.)
2. Finalize DatetimeArray vs. ndarray for tz aware and naive
3. Explore `.array` always being an Array.

I think 2 and 3 will be easier to think about once we have a DatetimeArray.
> https://github.com/pandas-dev/pandas/pull/23623#issuecomment-442814670

certainly fine as a followup. I am not sure 3 is actually a blocker for 0.24.0 (though 2 is) and 1) as-is is fine for now
ping on green.
All green.
thanks @TomAugspurger very nice!