ENH: add ExtensionArray.to_numpy to have control over conversion to numpy array
State: closed

xref https://github.com/pandas-dev/pandas/issues/30038

Would still need to add this to the other arrays with NA (IntegerArray, StringArray), and need to pass through such option from Series.to_numpy. But already putting this up to check if we are OK with such interface and behaviour.

Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
cc @TomAugspurger 
Just a note: The signature here differs from Series/Index.to_numpy and DataFrame.to_numpy, since this includes an `na_value`.

Should we add `na_value` to the other versions with a default of `lib._no_default` which means "the NA value for that dtype"? I can see that being useful for Series. I'm not sure if it's useful for DataFrame, but perhaps.

Would that already be useful for any of the other implementations / dtypes? 

Right now I was planning to pass-through additional keywords to the EA version (if the Series is backed with EA and has a to_numpy method, or something). But of course adding the keyword specifically makes it clearer to document / find it.
Just passing through makes sense. I think we can actually add a default `ExtensionArray.to_numpy()` that just calls `np.asarray(self, dtype=dtype)`.
Shouldn't need to. I think we autodoc all the methods here.
I added the pass-through from Series
can you rebase & update to comments
I've pushed a somewhat large change that implements the discussion in https://github.com/pandas-dev/pandas/pull/30322#discussion_r360882152. It adds two main things

* ExtensionArray.to_numpy to the base ExtensionArray class
* `na_value` to Series/Index/DataFrame.to_numpy

A few things to note

1. ExtensionArray.to_numpy doesn't accept kwargs. I don't think it needs to. Subclasses can accept additional keyword arguments if they choose to (I added one to decimal as an example)
2. Subclasses will need to import `_no_default` from `pandas._libs`. We should make that public.
The CI failure is unrelated and is being fixed in https://github.com/pandas-dev/pandas/pull/30660
Thanks for the updates! 

Indeed, the `_no_default` then needs to be usable by EA authors, that's a bit unfortunate .. Put it in `pd.api.types` ? (I don't think there is a way to avoid using it if we want to add `na_value` as a keyword rather than pass it through in the kwargs)
This is passing again.
Would like to merge this in a few hours, and put up a PR creating a base class for BooleanArray & IntegerArray later today.
> and put up a PR creating a base class for BooleanArray & IntegerArray later today.

I already have a branch lying around with an initial take on this, in case you didn't start yet.
There is still a difference in the handling of NA in conversion to float between integer and boolean. 

For boolean, I implemented it as strict. So a float cannot hold NA, so if there are missing values, this errors, and you need to explicitly use `na_value=np.nan` (but it will work without if there are no missing values):

```
In [8]: a1 = pd.array([True, False, pd.NA]) 

In [9]: a1.to_numpy(float)   # (not the best error message though)
...
TypeError: float() argument must be a string or a number, not 'NAType'

In [10]: a1.to_numpy(float, na_value=np.nan) 
Out[10]: array([ 1.,  0., nan])
```

While for integer, you implemented this less strict (it's the base EA.to_numpy implementation that is getting this behaviour of `__array__`):

```
In [11]: a2 = pd.array([1, 2, pd.NA])  

In [12]: a2.to_numpy(float) 
Out[12]: array([ 1.,  2., nan])

In [13]: a2.to_numpy(float, na_value=np.nan) 
Out[13]: array([ 1.,  2., nan])

In [14]: a2.to_numpy(float, na_value=pd.NA)
...
TypeError: float() argument must be a string or a number, not 'NAType'
```

It would be nice to align this. This relates to the discussion in the issue https://github.com/pandas-dev/pandas/issues/30038
> I would do the rename of _no_default -> no_default though

Yes, +1. If we make it public to use for external EA authors, let's make that clear with the name you can use it  (and the actual object itself is already in the private `_lib`, so no need for the name itself to have a `_` there).
> It would be nice to align this. This relates to the discussion in the issue #30038

Rereading that again, it seems the conclusion was somewhat that we want to raise for this (so no automatic conversion of pd.NA to np.nan)
OK with doing the _no_default -> no_default in a followup?
FYI, I have a followup PR doing IntegerArray.to_numpy, which will fix the inconsistencies when converting to an ndarary (changing integer to match the behavior here). I think that needs to be its own PR, with a followup moving them to the base class.

OK to merge this?
> I think that needs to be its own PR, with a followup moving them to the base class.

You saw my comment above about this? (I have an initial branch for this, in case you didn't start it yet)

But yes, I think that is good as a follow-up.