Repository: modin-project/modin

Issue:
Modin DataFrame to_parquet Path Param Differences Between Dask and Ray
Thank you for the fantastic work on Modin! In some recent exploration of Modin capabilities, I noticed that `to_parquet` appears to seek slightly different path parameter depending on whether Dask or Ray are used as the engine. With Dask, path seems to be a file. With Ray, path seems to be a directory (by default, exporting potentially multiple files based on the dataset). Is the difference between Ray and Dask with Modin expected in this instance? If so, is there any documentation I may reference to learn more? Appreciate any guidance on this.


Discussion:
@d33bs thank you for your report. Unfortunately, while Modin on Ray supports distributed `to_parquet`, Modin on Dask does not yet. Modin parallelizes `to_parquet` by writing chunks of rows to separate files. You see multiple files with Ray because each file was written in parallel, while with Dask you should get warning `UserWarning: `to_parquet` defaulting to pandas implementation.` and end up with a single file.

For now, let's treat this issue as a feature request for `to_parquet` on Dask. 
@mvashishtha , thank you so much! Based on what you mention, could this be assisted in the meantime with a documentation update on the [pd.DataFrame supported APIs](https://modin.readthedocs.io/en/stable/supported_apis/dataframe_supported.html) page for `to_parquet`? For example, would it be correct to indicate in the notes what you mentioned here (that Ray provides parallelized file output into a dir and Dask does not)?
@d33bs Yes, we should document the partial support for `to_parquet`. I filed #4628 for that. If you would like to take that issue, your help would be very welcome. Someone from @modin-project/modin-core could help you get the PR submitted.

Note that as I mention in that issue, even Modin on Ray doesn't support `to_parquet` in all cases. We should document those cases as well.
Hi @mvashishtha - thank you again! I'd be more than happy to assist with #4628 , appreciate the opportunity to contribute.

Resolving patch:
diff --git a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
--- a/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
+++ b/modin/core/execution/dask/implementations/pandas_on_dask/io/io.py
@@ -13,6 +13,10 @@
 
 """Module houses class that implements ``BaseIO`` using Dask as an execution engine."""
 
+import os
+
+import pandas
+
 from modin.core.io import BaseIO
 from modin.core.storage_formats.pandas.query_compiler import PandasQueryCompiler
 from modin.core.execution.dask.implementations.pandas_on_dask.dataframe import (
@@ -67,3 +71,80 @@ class PandasOnDaskIO(BaseIO):
     read_excel = type(
         "", (DaskWrapper, PandasExcelParser, ExcelDispatcher), build_args
     ).read
+
+    @staticmethod
+    def _to_parquet_check_support(kwargs):
+        """
+        Check if parallel version of `to_parquet` could be used.
+
+        Parameters
+        ----------
+        kwargs : dict
+            Keyword arguments passed to `.to_parquet()`.
+
+        Returns
+        -------
+        bool
+            Whether parallel version of `to_parquet` is applicable.
+        """
+        path = kwargs["path"]
+        compression = kwargs["compression"]
+        if not isinstance(path, str):
+            return False
+        if any((path.endswith(ext) for ext in [".gz", ".bz2", ".zip", ".xz"])):
+            return False
+        if compression is None or not compression == "snappy":
+            return False
+        return True
+
+    @classmethod
+    def to_parquet(cls, qc, **kwargs):
+        """
+        Write a ``DataFrame`` to the binary parquet format.
+
+        Parameters
+        ----------
+        qc : BaseQueryCompiler
+            The query compiler of the Modin dataframe that we want to run `to_parquet` on.
+        **kwargs : dict
+            Parameters for `pandas.to_parquet(**kwargs)`.
+        """
+        if not cls._to_parquet_check_support(kwargs):
+            return BaseIO.to_parquet(qc, **kwargs)
+
+        output_path = kwargs["path"]
+        os.makedirs(output_path, exist_ok=True)
+
+        def func(df, **kw):
+            """
+            Dump a chunk of rows as parquet, then save them to target maintaining order.
+
+            Parameters
+            ----------
+            df : pandas.DataFrame
+                A chunk of rows to write to a parquet file.
+            **kw : dict
+                Arguments to pass to ``pandas.to_parquet(**kwargs)`` plus an extra argument
+                `partition_idx` serving as chunk index to maintain rows order.
+            """
+            compression = kwargs["compression"]
+            partition_idx = kw["partition_idx"]
+            kwargs[
+                "path"
+            ] = f"{output_path}/part-{partition_idx:04d}.{compression}.parquet"
+            df.to_parquet(**kwargs)
+            return pandas.DataFrame()
+
+        # Ensure that the metadata is synchronized
+        qc._modin_frame._propagate_index_objs(axis=None)
+        result = qc._modin_frame._partition_mgr_cls.map_axis_partitions(
+            axis=1,
+            partitions=qc._modin_frame._partitions,
+            map_func=func,
+            keep_partitioning=True,
+            lengths=None,
+            enumerate_partitions=True,
+        )
+        DaskWrapper.materialize(
+            [part.list_of_blocks[0] for row in result for part in row]
+        )


Test patch: