ENH: Add Numpy Array Interface support to Pandas objects
State: closed

WIP


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
- pls run a perf check on this as well as its hits some critical paths.
- tests!

oops yeah, didn't mean to imply this was done. 

I actually threw this up real quick cuz I'm not sure the helper func belongs in common.py. afaik common.py shouldn't know about NDFrame, though maybe it's okay because the import happens within the function. 

I need a guard against everything that implements the array interface, but that we explicitly handle. Right now, that appears to be `NDFrame` and `ndarray`. 

So I found https://github.com/pydata/pandas/issues/5698. 

I think the confusion lies in that `__array__`, `__array_interface__`, `__array_struct__` all appear to make objects array-like. AFAIK, `__array__` is meant for subclasses and is not the array interface per se, it's just that `numpy` doesn't appear to check. 

I _think_ we're supposed to implement the `__array_interface__`. 

I know the [numpy docs](http://docs.scipy.org/doc/numpy/reference/arrays.classes.html) _say_ that these methods are for ndarray subclasses, but my experience writing a handful of ndarray-like objects (non-subclasses) is that numpy checks for these attributes (duck typing) instead of checking for actual ndarray subclasses. I would consider this a documentation bug for numpy.

`__array_interface__` appears to be design for low-level inoperability with numpy.ndarrays, and, as [the docs say](http://docs.scipy.org/doc/numpy/reference/arrays.interface.html) appears to be largely supplanted by the [revised buffer protocol](http://legacy.python.org/dev/peps/pep-3118/).

So, roughly speaking, these are two different ndarray interfaces that serve different needs. Are you need/want to be checking for the later? Maybe you could describe what exactly you are trying to do with this? :)

@shoyer +1 on the numpy docs being buggy! Yeah, looking through the numpy code it looks like checks for array-likeness in one big function through a series of `if` statements. I can use `__array_interface__` with `__array_wrap__` and everything is fine. I should've linked the original issue #8311. 

I put the check above our big `if` block since `DataFrame` has explicit handling for `ndarray`. If it was just a no-op like in `Series` I would've just put it in the `else`. If I put it in the `DataFrame` else, I'd have to copypasta replicate the `np.ndarray` handling. 

I think you should simply use `__array__`, see here: https://github.com/pydata/pandas/pull/7270

This would technicaly also allow Python Array classes too

the others are for 'c' checks IIRC (never clear on how this works)

@dalejung and would be nice to change the check in #7270 to integrate with yours (as a single function).

we have `is_list_like`, add `is_array_like`

@jreback yeah, for a long time I've only really known `__array__`. I only went through the other methods because of my tests breaking with `rpy2` compat. I don't think it's `rpy2` that's implementing things wrong, just the numpy api has features that overlap. 

I'll look through and see how we're using `__array__` for checks. Might be worthwhile to just create an `ABCNumpyArray` or something to replace checking for `ndarray` and its cousins. 

@jreback so I notice that there was an effort to start abstracting away our dependency on `numpy`. I'm in need of the `np.array` equivalent inside of `core/array.py`. It's just a simple pointer but what should it be called? Do we want a `pa.array` when we have `pa.Array`?

My 2 cents is that it's not worth much effort to abstract away our dependency on numpy until there are any even remotely plausible alternatives in sight. Until then it's very hard to say even exactly what we will need.

@dalejung their was _some_ work be @wesm to abstract the importing of numpy. That is mostly boilerplate (and should be removed, I'll make a separate issue for that).

Here's a gist with 3 vbench runs. https://gist.github.com/dalejung/cdf7e3f059f0f80f6486

nothing really sticks out to me, it kind of jumps around with different tests showing up as slow.

@dalejung I think to make things consistent, can you also `_is_sequence` -> `is_sequence_like` in the code base..thxs

if you happend to document this as well would be gr8!

@dalejung can you revisit. I like some of the things going on here, but this devolved a bit.

closing pls reopen if/when updated