Repository: modin-project/modin

Issue:
BUG: read_parquet can't detect column partitioning in non-local filesystems
### System information
- **Modin version** (`modin.__version__`): 86d361078fbad03143bf02ce212ffa8af4d416b5

Parquet datasets are arbitrarily deep directories of data files that can include column partitioning (ref. [here](https://spark.apache.org/docs/2.4.0/sql-data-sources-parquet.html#partition-discovery)). In column partitioning, a directory like `col1=3` only has data where column 1 is equal to 3. Modin tries to default to pandas for datasets with column partitioning, but it [assumes](https://github.com/modin-project/modin/blob/5ca6d40f0ab9e6fede0dd5e77a415231a394fa90/modin/core/io/column_stores/parquet_dispatcher.py#L59) that column partitioning can only happen when the path to read is a **local** directory. The directory could, for example, be on s3.

I think that nothing is functionally wrong when we fail to detect column partitioning. Modin can still read the data correctly. It's just not defaulting to pandas even though we expect it to.


Discussion:
Maybe we should also consider **not** defaulting to pandas for cases like this. We can still parallelize the read across columns and might even do better than pandas in some cases.
> We can still parallelize the read across columns and might even do better than pandas in some cases.

Note that now that we parallelize across both rows and columns, I don't think we can read column-value-partitioned datasets so easily.
> I think that nothing is functionally wrong when we fail to detect column partitioning. Modin can still read the data correctly.

Per #5159, we now throw an error.
@gshimansky could you please copy the partitioned directory `s3://mahesh-vashishtha/modin_bug_5159_parquet/df.parquet` to somewhere in `s3://modin-datasets/` and make it publicly accessible so we can use that directory in a test case?
I created folder `s3://modin-datasets/modin-bugs` and copied `modin_bug_5159_parquet` into it. Please check that you have access, everything should be public.
@gshimansky thanks, I can read the `modin_bug_5159_parquet` directory with no credentials, and I can list the contents in `s3://modin-datasets/modin-bugs/`.

Resolving patch:
diff --git a/modin/core/io/column_stores/parquet_dispatcher.py b/modin/core/io/column_stores/parquet_dispatcher.py
--- a/modin/core/io/column_stores/parquet_dispatcher.py
+++ b/modin/core/io/column_stores/parquet_dispatcher.py
@@ -609,18 +609,28 @@ def _read(cls, path, engine, columns, **kwargs):
         )
         from modin.pandas.io import PQ_INDEX_REGEX
 
-        if isinstance(path, str) and os.path.isdir(path):
+        if isinstance(path, str):
+            if os.path.isdir(path):
+                path_generator = os.walk(path)
+            else:
+                storage_options = kwargs.get("storage_options")
+                if storage_options is not None:
+                    fs, fs_path = url_to_fs(path, **storage_options)
+                else:
+                    fs, fs_path = url_to_fs(path)
+                path_generator = fs.walk(fs_path)
             partitioned_columns = set()
             # We do a tree walk of the path directory because partitioned
             # parquet directories have a unique column at each directory level.
             # Thus, we can use os.walk(), which does a dfs search, to walk
             # through the different columns that the data is partitioned on
-            for (_, dir_names, files) in os.walk(path):
+            for (_, dir_names, files) in path_generator:
                 if dir_names:
                     partitioned_columns.add(dir_names[0].split("=")[0])
                 if files:
                     # Metadata files, git files, .DSStore
-                    if files[0][0] == ".":
+                    # TODO: fix conditional for column partitioning, see issue #4637
+                    if len(files[0]) > 0 and files[0][0] == ".":
                         continue
                     break
             partitioned_columns = list(partitioned_columns)
diff --git a/modin/core/storage_formats/pandas/parsers.py b/modin/core/storage_formats/pandas/parsers.py
--- a/modin/core/storage_formats/pandas/parsers.py
+++ b/modin/core/storage_formats/pandas/parsers.py
@@ -705,7 +705,7 @@ def _read_row_group_chunk(
     )
     def parse(files_for_parser, engine, **kwargs):
         columns = kwargs.get("columns", None)
-        storage_options = kwargs.pop("storage_options", {}) or {}
+        storage_options = kwargs.get("storage_options", {})
         chunks = []
         # `single_worker_read` just passes in a string path
         if isinstance(files_for_parser, str):


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -1728,6 +1728,21 @@ def test_read_parquet_2462(self, engine):
 
             df_equals(test_df, read_df)
 
+    @pytest.mark.skipif(
+        PandasCompatVersion.CURRENT == PandasCompatVersion.PY36,
+        reason="storage_options not supported for older pandas",
+    )
+    def test_read_parquet_s3_with_column_partitioning(self, engine):
+        # This test case comes from
+        # https://github.com/modin-project/modin/issues/4636
+        dataset_url = "s3://modin-datasets/modin-bugs/modin_bug_5159_parquet/df.parquet"
+        eval_io(
+            fn_name="read_parquet",
+            path=dataset_url,
+            engine=engine,
+            storage_options={"anon": True},
+        )
+
 
 class TestJson:
     @pytest.mark.parametrize("lines", [False, True])