PERF: homogeneous concat
State: closed

- [x] closes #50652 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Using the num_rows=100, num_cols=50_000, num_dfs=7 case from #50652, I'm getting 54.15s on main vs 1.21s on this branch.


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I ran this through the code in https://github.com/pandas-dev/pandas/issues/50652 and it looks good, see below.

But this only works for float64, so e.g. won't show a difference in ASV in `join_merge.ConcatDataFrames`, because that particular perf test is in float32.  

Can this not already be generalized all numpy floats and ints? What happens if you concatenate e.g. int8 and float32? 

EDT: Ok, I see that this depends on `take_2d_axis0_{dtype1}_{dtype2}` and there is no int8 -> float32 specifically and probably other conversions too. But if we (1) make all the `take_2d_axis0_{dtype1}_{dtype2}` functions and (2) find the common dtype of the concatenated dataframe, then should be easy? Though you mention "This will be simpler once JoinUnit.is_na behavior is deprecated", so maybe you have some followup, that will make that not necessary?

```
Testcase 1
NUM_ROWS: 100, NUM_COLS: 1000, NUM_DFS: 3
Pandas: 0.01
Manual: 0.01
True
...
Testcase 7
NUM_ROWS: 100, NUM_COLS: 10000, NUM_DFS: 7
Pandas: 0.14
Manual: 0.19
True
Testcase 8
NUM_ROWS: 100, NUM_COLS: 10000, NUM_DFS: 9
Pandas: 0.23
Manual: 0.30
True
...
Testcase 20
NUM_ROWS: 200, NUM_COLS: 10000, NUM_DFS: 9
Pandas: 0.40
Manual: 0.52
True
Testcase 21
NUM_ROWS: 200, NUM_COLS: 50000, NUM_DFS: 3
Pandas: 0.40
Manual: 0.43
True
Testcase 22
NUM_ROWS: 200, NUM_COLS: 50000, NUM_DFS: 5
Pandas: 0.91
Manual: 1.98
True
Testcase 23
NUM_ROWS: 200, NUM_COLS: 50000, NUM_DFS: 7
Pandas: 2.34
Manual: 7.30
True
Testcase 24
NUM_ROWS: 200, NUM_COLS: 50000, NUM_DFS: 9
Pandas: 10.25
Manual: 13.73
True
```
It would be easy to extend this to float32, others would take more effort
If we use `pd.core.array_algos.take._take_2d_axis0_dict` to match up different dtypes to the desired common dtype?
The trouble is determining what the desired common dtype is.
Yes, I can see it now, if the columns are not all the same it gets very complicated for ints.

I think this is good, but we should def. also get this performance boost for float32 IMO. 
Also would be good to have a whatsnew note
whatsnew added, float32 handled, + green
Thanks @jbrockmendel 
Hey, This PR caused a rather big slowdown on c-aligned ndarrays, see also discussion in #52786:

```python
>>> frame_c = pd.DataFrame(np.zeros((10000, 200), dtype=np.float32, order="C"))
>>> %timeit pd.concat([frame_c] * 20, axis=0, ignore_index=False)
45.1 ms ± 166 µs per loop  # after this PR
13.2 ms ± 126 µs per loop  # before this PR
```


I’m out of town this week, will take a look next week

On Wed, Apr 26, 2023 at 4:52 AM Terji Petersen ***@***.***>
wrote:

> Hey, This PR caused a rather big slowdown on c-aligned ndarrays, see also
> discussion in #52786 <https://github.com/pandas-dev/pandas/pull/52786>:
>
> >>> frame_c = pd.DataFrame(np.zeros((10000, 200), dtype=np.float32, order="C"))>>> %timeit pd.concat([frame_c] * 20, axis=0, ignore_index=False)45.1 ms ± 166 µs per loop  # after this PR132. ms ± 126 µs per loop  # before this PR
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/52685#issuecomment-1523287739>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6BTL6HJL5UKJPB6X6LXDED7XANCNFSM6AAAAAAW7S3GE4>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>

@phofl i have what i think should address this, but since i cant reproduce the slowdown also can't check if it works.  can you try adding the following at the top of _concat_homogeneous_fastpath

```
    if all(not indexers for _, indexers in mgrs_indexers):
        # https://github.com/pandas-dev/pandas/pull/52685#issuecomment-1523287739
        arrs = [mgr.blocks[0].values.T for mgr, _ in mgrs_indexers]
        arr = np.concatenate(arrs).T
        bp = libinternals.BlockPlacement(slice(shape[0]))
        nb = new_block_2d(arr, bp)
        return nb

```
Yep that works! Thx for looking into it.