Repository: modin-project/modin

Issue:
Do not split input dataframe along both axis at the same time in `.from_pandas()`
The recently introduced regression in pandas 2.1.0 (pandas-dev/pandas#55202) highlighted the problem that splitting across columns may be quite expensive and that we probably shouldn't do it in a loop as we do in our `.from_pandas()` implementation:
https://github.com/modin-project/modin/blob/a1a39a052919fe7ab6033d76a570f7ec3739ed4d/modin/core/dataframe/pandas/partitioning/partition_manager.py#L814-L823

Instead, it would be better to first chunk columns once (as it's quite expensive) and only then slice rows:
```python
col_parts = [
    df.iloc[:, i : i + col_chunksize]
    for i in range(0, len(df.columns), col_chunksize)
]
parts = [
    [
        col_part.iloc[i : i + row_chunksize]
        for col_part in col_parts
    ]
    for i in range(0, len(df), row_chunksize)
]
```


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/partitioning/partition_manager.py b/modin/core/dataframe/pandas/partitioning/partition_manager.py
--- a/modin/core/dataframe/pandas/partitioning/partition_manager.py
+++ b/modin/core/dataframe/pandas/partitioning/partition_manager.py
@@ -27,7 +27,7 @@
 import pandas
 from pandas._libs.lib import no_default
 
-from modin.config import BenchmarkMode, NPartitions, ProgressBar
+from modin.config import BenchmarkMode, Engine, NPartitions, ProgressBar
 from modin.core.dataframe.pandas.utils import concatenate
 from modin.core.storage_formats.pandas.utils import compute_chunksize
 from modin.error_message import ErrorMessage
@@ -811,13 +811,30 @@ def update_bar(pbar, f):
             )
         else:
             pbar = None
+
+        # even a full-axis slice can cost something (https://github.com/pandas-dev/pandas/issues/55202)
+        # so we try not to do it if unnecessary.
+        # FIXME: it appears that this optimization doesn't work for Unidist correctly as it
+        # doesn't explicitly copy the data when putting it into storage (as the rest engines do)
+        # causing it to eventially share memory with a pandas object that was provided by user.
+        # Everything works fine if we do this column slicing as pandas then would set some flags
+        # to perform in COW mode apparently (and so it wouldn't crash our tests).
+        # @YarShev promised that this will be eventially fixed on Unidist's side, but for now there's
+        # this hacky condition
+        if col_chunksize >= len(df.columns) and Engine.get() != "Unidist":
+            col_parts = [df]
+        else:
+            col_parts = [
+                df.iloc[:, i : i + col_chunksize]
+                for i in range(0, len(df.columns), col_chunksize)
+            ]
         parts = [
             [
                 update_bar(
                     pbar,
-                    put_func(df.iloc[i : i + row_chunksize, j : j + col_chunksize]),
+                    put_func(col_part.iloc[i : i + row_chunksize]),
                 )
-                for j in range(0, len(df.columns), col_chunksize)
+                for col_part in col_parts
             ]
             for i in range(0, len(df), row_chunksize)
         ]


Test patch: