PERF: improve 2D array access / transpose() for masked dtypes
State: closed

xref https://github.com/pandas-dev/pandas/issues/52016

For `transpose()` or operations that do a transpose under the hood (eg typically operations with axis=1), this is generally expensive if you have extension dtypes. This explores the possibilities to add a fast path for the specific case of a DataFrame with all-masked dtypes.

Using the example dataframe from https://github.com/pandas-dev/pandas/issues/52016, this gives a speed-up 

```python
shape = 250_000, 100
mask = pd.DataFrame(np.random.randint(0, 1, size=shape))

np_mask = mask.astype(bool)
pd_mask = mask.astype(pd.BooleanDtype())

%timeit pd_mask.transpose()
# 9.13 s ± 347 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <-- main
# 689 ms ± 28.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- PR
```


Note there are some various different optimizations added together (that can be splitted, and still needs to be cleaned-up in general). 
The main speed-up comes from calling `BooleanArray(..)` instead of `BooleanArray._from_sequence(..)` (or in general with any other masked array class), because this generic `_from_sequence` has a lot of overhead (that adds up when calling it 250_000 times as in this example). To be able to use the main `__init__`, I added a version of `mgr.as_array()` to return separate data and mask arrays. 

Further, after the arrays are constructed, the BlockManager construction can be optimized by passing `verify_integrity=False` and by knowing that we have all 1D EAs (so no need to do block dtype grouping etc). 
Another tiny optimization is in the BooleanArray(..) to avoid creating a new dtype instance every time (this can easily be split of, we might want to do that generally for all primitive dtypes, and maybe rather in the init of the dtype)

Labels:
name: Performance
description: Memory or execution speed performance
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Mothballed
description: Temporarily-closed PR the author plans to return to
name: Stale

Comments:
Is the concern about transpose specifically or more about the axis=1 reductions?  If the latter, would a reduce_axis1 approach that avoids the transpose altogether be more robust?  It'd be nice to avoid special-casing our internal EAs.

Can any of this be extended eventually to arrow-backed cases?
> Is the concern about transpose specifically or more about the axis=1 reductions? If the latter, would a reduce_axis1 approach that avoids the transpose altogether be more robust?

For those ops where there is a simple online algorithm (e.g. mean, sum, stddev, variance, sem, prod, kurtosis), I do think this would be a good option to look into. However there will be ops for which this won't work (median, quantile, rank, mode, nunique).
Yes, I also mentioned that in https://github.com/pandas-dev/pandas/issues/52016#issuecomment-1476158450, but I think we can certainly try to avoid transpose altogether, but improving transpose itself as well might also be worth it? (edit: plus for methods where the transpose cannot easily be avoided, as Richard just commented)  
(the main new function here that I added, `as_array_masked`, could also be used for both, see my inline comment above at https://github.com/pandas-dev/pandas/pull/52083#discussion_r1142041025)

> Can any of this be extended eventually to arrow-backed cases?

Yes, in general the arrow dtypes will suffer the same problems for row based operations (as long as arrow doesn't add some specific functionality for that). I think it could be quite easy to expand the approach here to work for arrow dtypes as well (converting their bitmask to a bytemask)
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Closing to clear the queue, but feel free to reopen when you have time to revisit