Repository: modin-project/modin
Pull request: FEAT-#3338: Add additional parameters for `from_partitions`
Signed-off-by: Alexey Prutskov <alexey.prutskov@intel.com>

<!--
Thank you for your contribution!
Please review the contributing docs: https://modin.readthedocs.io/en/latest/CONTRIBUTING.html
if you have questions about contributing.
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] commit message follows format outlined [here](https://modin.readthedocs.io/en/latest/contributing.html)
- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #3338 <!-- issue must be created for each patch -->
- [x] tests added and passing
- [x] module layout described at `docs/developer/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->


Problem statement:
Expand parameters support of `modin.distributed.dataframe.pandas.from_partitions`
Currently, function `modin.distributed.dataframe.pandas.from_partitions` supports only `partitions` and `axis` parameters. We can avoid internal blocking calls by adding additional optional `index`, `columns`, `row_lengths` and `column_widths` parameters, which can be useful for user in case he has this data.

Hints from issue discussion:

Commit message: FEAT-#3338: Fix pydoc check

Signed-off-by: Alexey Prutskov <alexey.prutskov@intel.com>

Patch to review:
diff --git a/modin/distributed/dataframe/pandas/partitions.py b/modin/distributed/dataframe/pandas/partitions.py
index 1bd9ebbde9b..3a3d1db49e9 100644
--- a/modin/distributed/dataframe/pandas/partitions.py
+++ b/modin/distributed/dataframe/pandas/partitions.py
@@ -89,7 +89,9 @@ def _unwrap_partitions(oid):
         ]
 
 
-def from_partitions(partitions, axis):
+def from_partitions(
+    partitions, axis, index=None, columns=None, row_lengths=None, column_widths=None
+):
     """
     Create DataFrame from remote partitions.
 
@@ -106,11 +108,26 @@ def from_partitions(partitions, axis):
         * ``axis=0`` if you want to create DataFrame from row partitions
         * ``axis=1`` if you want to create DataFrame from column partitions
         * ``axis=None`` if you want to create DataFrame from 2D list of partitions
+    index : sequence, optional
+        The index for the DataFrame. Is computed if not provided.
+    columns : sequence, optional
+        The columns for the DataFrame. Is computed if not provided.
+    row_lengths : list, optional
+        The length of each partition in the rows. The "height" of
+        each of the block partitions. Is computed if not provided.
+    column_widths : list, optional
+        The width of each partition in the columns. The "width" of
+        each of the block partitions. Is computed if not provided.
 
     Returns
     -------
     modin.pandas.DataFrame
         DataFrame instance created from remote partitions.
+
+    Notes
+    -----
+    Pass `index`, `columns`, `row_lengths` and `column_widths` to avoid triggering
+    extra computations of the metadata.
     """
     from modin.data_management.factories.dispatcher import FactoryDispatcher
 
@@ -158,9 +175,20 @@ def from_partitions(partitions, axis):
         raise ValueError(
             f"Got unacceptable value of axis {axis}. Possible values are {0}, {1} or {None}."
         )
+    if index is None:
+        index = partition_mgr_class.get_indices(0, parts, lambda df: df.axes[0])
+
+    if columns is None:
+        columns = partition_mgr_class.get_indices(1, parts, lambda df: df.axes[1])
 
-    index = partition_mgr_class.get_indices(0, parts, lambda df: df.axes[0])
-    columns = partition_mgr_class.get_indices(1, parts, lambda df: df.axes[1])
     return DataFrame(
-        query_compiler=PandasQueryCompiler(partition_frame_class(parts, index, columns))
+        query_compiler=PandasQueryCompiler(
+            partition_frame_class(
+                parts,
+                index,
+                columns,
+                row_lengths=row_lengths,
+                column_widths=column_widths,
+            )
+        )
     )
diff --git a/modin/test/test_partition_api.py b/modin/test/test_partition_api.py
index 5debcea63f4..1d9a68bfe81 100644
--- a/modin/test/test_partition_api.py
+++ b/modin/test/test_partition_api.py
@@ -102,11 +102,35 @@ def test_unwrap_partitions(axis):
                 )
 
 
+@pytest.mark.parametrize("column_widths", [None, "column_widths"])
+@pytest.mark.parametrize("row_lengths", [None, "row_lengths"])
+@pytest.mark.parametrize("columns", [None, "columns"])
+@pytest.mark.parametrize("index", [None, "index"])
 @pytest.mark.parametrize("axis", [None, 0, 1])
-def test_from_partitions(axis):
-    data = np.random.randint(0, 100, size=(2 ** 16, 2 ** 8))
+def test_from_partitions(axis, index, columns, row_lengths, column_widths):
+    num_rows = 2 ** 16
+    num_cols = 2 ** 8
+    data = np.random.randint(0, 100, size=(num_rows, num_cols))
     df1, df2 = pandas.DataFrame(data), pandas.DataFrame(data)
     expected_df = pandas.concat([df1, df2], axis=1 if axis is None else axis)
+
+    index = expected_df.index if index == "index" else None
+    columns = expected_df.columns if columns == "columns" else None
+    row_lengths = (
+        None
+        if row_lengths is None
+        else [num_rows, num_rows]
+        if axis == 0
+        else [num_rows]
+    )
+    column_widths = (
+        None
+        if column_widths is None
+        else [num_cols]
+        if axis == 0
+        else [num_cols, num_cols]
+    )
+
     if Engine.get() == "Ray":
         if axis is None:
             futures = [[ray.put(df1), ray.put(df2)]]
@@ -118,7 +142,14 @@ def test_from_partitions(axis):
             futures = [client.scatter([df1, df2], hash=False)]
         else:
             futures = client.scatter([df1, df2], hash=False)
-    actual_df = from_partitions(futures, axis)
+    actual_df = from_partitions(
+        futures,
+        axis,
+        index=index,
+        columns=columns,
+        row_lengths=row_lengths,
+        column_widths=column_widths,
+    )
     df_equals(expected_df, actual_df)
 
 


Code review comments:
diff_hunk: @@ -158,9 +175,20 @@ def from_partitions(partitions, axis):
         raise ValueError(
             f"Got unacceptable value of axis {axis}. Possible values are {0}, {1} or {None}."
         )
+    if index is None:
+        index = partition_mgr_class.get_indices(0, parts, lambda df: df.axes[0])
+
+    if columns is None:
+        columns = partition_mgr_class.get_indices(1, parts, lambda df: df.axes[1])
 
-    index = partition_mgr_class.get_indices(0, parts, lambda df: df.axes[0])
-    columns = partition_mgr_class.get_indices(1, parts, lambda df: df.axes[1])
     return DataFrame(
-        query_compiler=PandasQueryCompiler(partition_frame_class(parts, index, columns))
+        query_compiler=PandasQueryCompiler(
+            partition_frame_class(
+                parts,
+                index,
+                columns,
+                row_lengths=row_lengths,
+                column_widths=column_widths,
path: modin/distributed/dataframe/pandas/partitions.py
text: @user1:
Let's also add `dtypes` parameter.

@user1:
Though, we can wait until it is really required. Please, fix small comment about the supplementation and we will merge the PR.

@author:
It's already added. The change doesn't look costly. Maybe let's leave this?

@user1:
We can leave it but in that case we need to add a note on `dtypes` must match underlying data of the partitions, i.e. having passed `dtypes` the DataFrame columns won't be cast to that types. Just to avoid confusing in such cases:
```python
import pandas
df = pandas.DataFrame([1, 2, 3], dtype="float")
df
     0
0  1.0
1  2.0
2  3.0
```
 Also, passing `dtypes` doesn't trigger computations when creating a DataFrame.

@author:
It that case let's postpone this until it's really required, as you proposed.

@user1:
Ok

@author:
Changes related to `dtypes` were removed.