PERF: Faster transposition of frames with masked arrays
State: closed

Faster transpose of dataframes with homogenous masked arrays. This also helps when doing reductions with `axis=1` as those currently transpose data before doing reductions.

Performance example:

```python
In [1]:         import pandas as pd, numpy as np
   ...:         values = np.random.randn(100000, 4)
   ...:         values = values.astype(int)
   ...:         df = pd.DataFrame(values).astype("Int64")
>>> %timeit df.transpose()
1.91 s ± 3.08 ms per loop  # main
563 ms ± 2.48 ms per loop  # this PR
```

Running `asv continuous -f 1.1 upstream/main HEAD -b reshape.ReshapeMaskedArrayDtype.time_transpose`:

```
       before           after         ratio
     [c3f0aac1]       [43162428]
     <master>         <cln_managers>
-     13.5±0.06ms      1.82±0.01ms     0.14  reshape.ReshapeMaskedArrayDtype.time_transpose('Float64')
-     13.8±0.03ms      1.83±0.01ms     0.13  reshape.ReshapeMaskedArrayDtype.time_transpose('Int64')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

There may be possible to improve performance when frames have a common masked dtype, I intend to look into that in a followup.

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
@topper-123 have you seen https://github.com/pandas-dev/pandas/pull/52083 and https://github.com/pandas-dev/pandas/pull/52689? 
I think it is generally the same idea (directly accessing _data and _mask and creating the transposed versions of those separately, and then faster reconstructing the MaskedArrays from that), but a bit differently organized.
Using the timings in the OP, I'm seeing

```
625 ms ± 8.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <-- This PR
733 ms ± 3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # <-- #52689
```

I haven't yet looked into the implementation to see where this difference is coming from. I'm happy to go with this implementation, would like to see if we can get this in 2.0.2.
I've merged the latest changes to the main branch into this in case this is he one we go with.
ping...
@topper-123 - have a conflict. @jbrockmendel - friendly ping.
Ping.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
The PR has gone quite stale, I think it's best to close this PR, unless it can be approved for merging.

cc @jbrockmendel , @rhshadrach .
I've updated.
Thanks @topper-123
Thanks.

In total, after the other related PRs have been merged, we have:

```python
n [1]:         import pandas as pd, numpy as np
   ...:         values = np.random.randn(100000, 4)
   ...:         values = values.astype(int)
   ...:         df = pd.DataFrame(values).astype("Int64")
>>> %timeit df.transpose()
1.91 s ± 3.08 ms per loop  # main at start
563 ms ± 2.48 ms per loop  # this PR, originally
375 ms ± 5.47 ms per loop  # this PR, now
```
An improvement, but very slow still...