API: dispatch to EA.astype
State: closed

Closes https://github.com/pandas-dev/pandas/issues/21296

split from https://github.com/pandas-dev/pandas/pull/22325

Note that this is just a partial fix to get sparse working. We have some issues:

for better or worse, categorical is still special cased. `Series[EA].astype('category')` will not yet dispatch to `EA.astype`. That's good non-pandas EAs won't have to worry about specially handling categorical. It's bad, since it's a special case.

And that's the second issue, in general, how should we handle astyping from EA1 -> EA2? In principle, we always have `ndarray[object]` as the lowest common denominator. So something like `EA2(EA.astype(object))` may work. But that won't necessarily be efficient. I'm not sure what's best here.

cc @xhochy 

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=h1) Report
> Merging [#22343](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/cf70d11a25ec59a5b630921d86759c7fa49884db?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22343/graphs/tree.svg?width=650&src=pr&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22343      +/-   ##
==========================================
+ Coverage   92.05%   92.05%   +<.01%     
==========================================
  Files         169      169              
  Lines       50709    50712       +3     
==========================================
+ Hits        46679    46682       +3     
  Misses       4030     4030
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <100%> (ø)` | :arrow_up: |
| #single | `42.25% <72.72%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/22343/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `93.84% <100%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/22343/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `88.58% <100%> (ø)` | :arrow_up: |
| [pandas/core/arrays/integer.py](https://codecov.io/gh/pandas-dev/pandas/pull/22343/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVnZXIucHk=) | `94.69% <100%> (+0.02%)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/22343/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.24% <0%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/22343/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.72% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=footer). Last update [cf70d11...2606d02](https://codecov.io/gh/pandas-dev/pandas/pull/22343?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Code changes look good!

Regarding the discussion items you mention, I am wondering if we need some kind of "negotation protocol" between (EA) dtypes for this. Basically something like:

- consider ``Series(EA).astype(dtype)``
- first try ``EA.astype(dtype)`` -> if the EA knows how to handle the specific `dtype`, return a new EA/array; if it does not know how to convert itself to `dtype`, return NotImplemented
- if the above returned NotImplemented, ask `dtype` if it knows how to convert the `EA` values to its own dtype -> if it knows, return a new EA/array; otherwise return NotImplemented
- if both don't know (above return NotImplemented again), either raise an error or fall back to the base implementation (`np.array(self, dtype=dtype,`, which might also raise an error)

That would in principle enable `EA.astype(EA2)`, and might also mitigate the 'category' problem you mention.
Your negotiation seems sensible I think. I worry a bit about a divergence between `Series.astype(dtype)` and `Series.values.astype(dtype)`, but that's maybe unavoidable.

edit: though it would be strange (wrong) for a method like `ExtensionArray.astype` to return NotImplemented, so this would have to be a back-channel like `ExtensionArray._astype`, which is the same as astype, but can return NotImplemented? This seems messy.
> I worry a bit about a divergence between Series.astype(dtype) and Series.values.astype(dtype)

That's a good point, yeah, that would be strange. Such a negotiation does not really fit for a method, only for functions probably ..
> Such a negotiation does not really fit for a method, only for functions probably...

How about raising (and catching) an `UnsupportedConversionError` (but letting other errors pass through)?  That should easily be compatible with method calls.
Perhaps. We'll leave that for another issue though I think, if we discover a need for it in practice.
Fixed the referenced issue.
That's the same issue as for https://github.com/pandas-dev/pandas/pull/22345?
Ugh, too early. https://github.com/pandas-dev/pandas/issues/21296 is the one.
@jreback was your comment about the wrong issue number all that was left from your point of view? If so, this should be good to go.
thanks @TomAugspurger 

maybe need to modify text in pandas/arrays/base/ExtensionArray (the doc-string) to add astype)?