Repository: modin-project/modin

Issue:
PERF: slow `concat`
Measured on master (https://github.com/modin-project/modin/commit/cfafbb254c221dd4f739a9cf5af17c9e8cdf13c3), Ray, 8 cores.

Problem: too much time is spent doing `concat`, it can be much faster. Pandas vs Modin: 0.89 sec vs 4,5 sec.

Possible solution: compute `new_widths` via `_column_widths_cache` where possible.

Script:
```python
import modin.pandas as pd
import numpy as np
from time import time

random_state = np.random.RandomState(seed=42)
array = random_state.rand(10**6, 35)

df1 = pd.DataFrame(array)
df2 = pd.DataFrame(array)

df1 = df1 - 1
df2 = df2 - 2

start = time()
df = pd.concat([df1, df2], axis=1, copy=False)
print(f"concat time: {time()-start}") <-- 4,5 sec
```


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -2581,6 +2581,18 @@ def concat(
             New Modin DataFrame.
         """
         axis = Axis(axis)
+        new_widths = None
+
+        def _compute_new_widths():
+            widths = None
+            if self._column_widths_cache is not None and all(
+                o._column_widths_cache is not None for o in others
+            ):
+                widths = self._column_widths_cache + [
+                    width for o in others for width in o._column_widths_cache
+                ]
+            return widths
+
         # Fast path for equivalent columns and partitioning
         if (
             axis == Axis.ROW_WISE
@@ -2600,6 +2612,8 @@ def concat(
             left_parts = self._partitions
             right_parts = [o._partitions for o in others]
             new_lengths = self._row_lengths_cache
+            # we can only do this for COL_WISE because `concat` might rebalance partitions for ROW_WISE
+            new_widths = _compute_new_widths()
         else:
             (
                 left_parts,
@@ -2611,6 +2625,7 @@ def concat(
             )
             if axis == Axis.COL_WISE:
                 new_lengths = partition_sizes_along_axis
+                new_widths = _compute_new_widths()
             else:
                 new_widths = partition_sizes_along_axis
         new_partitions = self._partition_mgr_cls.concat(
@@ -2646,15 +2661,15 @@ def concat(
             # frame. Typically, if we know the width for any partition in a
             # column, we know the width for the first partition in the column.
             # So just check the widths of the first row of partitions.
-            new_widths = []
-            if new_partitions.size > 0:
-                for part in new_partitions[0]:
-                    if part._width_cache is not None:
-                        new_widths.append(part.width())
-                    else:
-                        new_widths = None
-                        break
-
+            if not new_widths:
+                new_widths = []
+                if new_partitions.size > 0:
+                    for part in new_partitions[0]:
+                        if part._width_cache is not None:
+                            new_widths.append(part.width())
+                        else:
+                            new_widths = None
+                            break
         return self.__constructor__(
             new_partitions, new_index, new_columns, new_lengths, new_widths, new_dtypes
         )


Test patch: