INT: provide helpers for accessing the values of DataFrame columns
State: closed

Broken off from https://github.com/pandas-dev/pandas/pull/32867, also mentioned this in https://github.com/pandas-dev/pandas/pull/33229

In general, when doing certain things column-wise, we often just need the arrays (eg to calculate a reduction, to check the dtype, ..), and creating Series gives unnecessary overhead in that case. 

In this PR, I added therefore two helper functions for making it easier to do this (`_ixs_values` as variant on `_ixs` but returning the array instead of Series, and `_iter_arrays` that calls `_ixs_values` for each column iteratively as an additional helper function). 

I also used it in one place as illustration (in `_reduce`, what I was working on in https://github.com/pandas-dev/pandas/pull/32867, but it can of course be used more generally). 
In that example case, it is to replace an apply:

```
In [1]: df = pd.DataFrame(np.random.randint(1000, size=(10000,10))).astype("Int64").copy() 

In [2]: from pandas.core.dtypes.common import is_datetime64_any_dtype, is_period_dtype

In [3]: %timeit df.dtypes.apply(lambda x: is_datetime64_any_dtype(x) or is_period_dtype(x)) 
256 µs ± 2.66 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [5]: %timeit np.array([is_datetime64_any_dtype(values) or is_period_dtype(values) for values in df._iter_arrays()], dtype=bool) 
56.2 µs ± 282 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

(and this is only fo 10 columns)

@jbrockmendel 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
Do you have ideas generally about places this is applicable? One potential spot could be in groupby code:

https://github.com/pandas-dev/pandas/blob/66e45178c398e94305dfab3c6d88989c349fdb51/pandas/core/groupby/generic.py#L980

Which right now will yield a 2D object when there are duplicate labels but maybe shouldn't

Wondering if we should include the label as part of what is returned here
> Wondering if we should include the label as part of what is returned here

In principle you can still do that rather easily manually, by doing `for label, values in zip(df.columns, df._iter_arrays())`, if you also need to the corresponding labels. 
But yeah, it will depend a bit on where this would be used whether it makes sense to include this in the helper function or not.

The `_iterate_slices` you link to iterates through the columns as Series objects, while the function I am adding here specifically gives arrays. Now, I am not familiar enough with that specific groupby code to know if it actually needs the values as a Series.
> _iterate_slices

I've got a branch motivated by trying to optimize this, ended up making a `NDFrame._from_mgr` constructor for just when we are passing just a BlockManager/SingleBlockManager to DataFrame/Series.  Got a good deal of mileage out of that.

@jorisvandenbossche do you have a good read on what part of the Series construction is causing the overhead?
> do you have a good read on what part of the Series construction is causing the overhead?

I didn't check this time any more (looked at it previously), but here, when the only thing you need is an array, any overhead that the Series constructor gives is easily avoided overhead. It's not that this PR is adding complex code or so. I think it will be a useful helper function regularly (I added one use case here, and the two linked PRs can be two other use cases already).

I suppose that the Series constructor overhead can be decreased (which would certainly be useful anyway), but it might require some additional keywords (eg a fastpath that avoids `sanitize_array`) or separate private constructor like `_from_array` (similarly to how `DataFrame._from_arrays`).

I don't think tests are necessarily needed, since it's an internal helper function (most private methods don't have specific tests), but I can add a simple one if needed.
> I don't think tests are necessarily needed, since it's an internal helper function (most private methods don't have specific tests), but I can add a simple one if needed.

ok no big deal
some comments on annotations, otherwise LGTM
> "Union[np.ndarray, ExtensionArray]" -> "ArrayLike:"

Honestly, I am not yet much into the typing in pandas, but I find that a rather confusing name (as in human terms, array-like means something less strict to me)
> Honestly, I am not yet much into the typing in pandas, but I find that a rather confusing name (as in human terms, array-like means something less strict to me)

I understand where you're coming from, but this is one of those "if you want to change the policy, thats its own discussion" things

> do you have a preference for the name?

I'm fine with `_ixs_values`.  `_iter_arrays` might be clearer as `_iter_column_arrays`, in which case it might make sense for `_ixs_values` to become `_get_column_array`
> _iter_arrays might be clearer as _iter_column_arrays, in which case it might make sense for _ixs_values to become _get_column_array

+1 changed to that

And update the type annotations.
So I just noticed by trying to use this in https://github.com/pandas-dev/pandas/pull/32779 that `Block.iget` does not what I expected for Timedelta/DatetimeBlock (it returns numpy array instead of extension array). 

@jbrockmendel do you know if there is existing API to get one column of a 2D Timedelta/DatetimeBlock as EA ? `array_values` only works assuming you have a 2D block. I could in principle add an option to `Block.iget`, but that is also not really clean.
> do you know if there is existing API to get one column of a 2D Timedelta/DatetimeBlock as EA ?

There is not.  We already override `DatetimeLikeBlockMixin.iget`, what happens if we change that to

```
def iget(self, key):
    # TODO(EA2D): the reshape will be unnecessary with 2D EAs
    return self.array_values().reshape(self.shape)[key]
```

I changed `iget` earlier today to return TimedeltaArray, and there are quite some test failures then. I didn't really investigate, but at least we have some code assuming that iget returns a ndarray. I think many errors came from `_simple_new` asserting values being a ndarray

> I changed iget earlier today to return TimedeltaArray, and there are quite some test failures then. I didn't really investigate, but at least we have some code assuming that iget returns a ndarray. I think many errors came from _simple_new asserting values being a ndarray

It tentatively looks like this is coming from an issue with `DTA.__getitem__` that _may_ be fixed by #33290.

<b>Update</b> #33290 doesnt quite do it, but a related `DTA.__getitem__` bugfix (that ill push after 33290 goes through) does.
looks like needs a rebase
thanks @jorisvandenbossche