Repository: modin-project/modin

Issue:
read_parquet reading only subset of multi-file Parquet directory
<!--
General questions should be asked on the mailing list modin-dev@googlegroups.com.

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: macOS 10.15.3
- **Modin installed from (source or binary)**: binary
- **Modin version**: 0.7.0
- **Python version**: 3.7.6
- **Exact command to reproduce**:

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
I have a set of Parquet files inside a directory "myparquet" on my local filesystem (copied from HDFS). Files have the following naming pattern:
```
part-00000-d50dbe9c-3b97-4f05-8355-6a01a98c8fef-c000.snappy.parquet
...
part-00071-d50dbe9c-3b97-4f05-8355-6a01a98c8fef-c000.snappy.parquet
```
Pandas by default reads all Parquet files correctly into a DataFrame, while it seems that the Modin implementation reads only a subset (maybe only a single file).

**Note**: I'm using Ray, not sure if this matters.

**Example code**:

```
import modin.pandas as pd
import pandas
import ray

ray.init()

if __name__ == '__main__':
    modin_df = pd.read_parquet('myparquet')
    pandas_df = pandas.read_parquet('myparquet')
    print(f'Modin DF len = {len(modin_df)}')
    print(f'Pandas DF len = {len(pandas_df)}')
```

The **output** will be something like the following:
```
Modin DF len = 2743
Pandas DF len = 132493
```

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be 
[myparquet.zip](https://github.com/modin-project/modin/files/4172095/myparquet.zip)
attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
Hi @guidoman thanks for posting!

This may be a regression introduced by #955. We test this use-case, but it looks like it's still not reading all of the part files in your case. Do you have an example file you can share? I'll try to reproduce this, but it would definitely help if you had a file that reproduced this. Thanks again for reporting!
[myparquet.zip](https://github.com/modin-project/modin/files/4172110/myparquet.zip)


Resolving patch:
diff --git a/modin/engines/base/io/column_stores/parquet_reader.py b/modin/engines/base/io/column_stores/parquet_reader.py
--- a/modin/engines/base/io/column_stores/parquet_reader.py
+++ b/modin/engines/base/io/column_stores/parquet_reader.py
@@ -27,7 +27,6 @@ def read(cls, path, engine, columns, **kwargs):
         if os.path.isdir(path):
             partitioned_columns = set()
             directory = True
-            original_path = path
             # We do a tree walk of the path directory because partitioned
             # parquet directories have a unique column at each directory level.
             # Thus, we can use os.walk(), which does a dfs search, to walk
@@ -39,17 +38,15 @@ def read(cls, path, engine, columns, **kwargs):
                     # Metadata files, git files, .DSStore
                     if files[0][0] == ".":
                         continue
-                    path = os.path.join(root, files[0])
                     break
             partitioned_columns = list(partitioned_columns)
             if len(partitioned_columns):
-                ErrorMessage.default_to_pandas("Partitioned Columns in Parquet")
+                ErrorMessage.default_to_pandas("Mixed Partitioning Columns in Parquet")
                 return cls.single_worker_read(
-                    original_path, engine=engine, columns=columns, **kwargs
+                    path, engine=engine, columns=columns, **kwargs
                 )
         else:
             directory = False
-
         if not columns:
             if directory:
                 # Path of the sample file that we will read to get the remaining columns


Test patch:
diff --git a/modin/pandas/test/data/hdfs.parquet/part-00000-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet b/modin/pandas/test/data/hdfs.parquet/part-00000-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet
new file mode 100644
Binary files /dev/null and b/modin/pandas/test/data/hdfs.parquet/part-00000-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet differ
diff --git a/modin/pandas/test/data/hdfs.parquet/part-00001-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet b/modin/pandas/test/data/hdfs.parquet/part-00001-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet
new file mode 100644
Binary files /dev/null and b/modin/pandas/test/data/hdfs.parquet/part-00001-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet differ
diff --git a/modin/pandas/test/data/hdfs.parquet/part-00002-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet b/modin/pandas/test/data/hdfs.parquet/part-00002-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet
new file mode 100644
Binary files /dev/null and b/modin/pandas/test/data/hdfs.parquet/part-00002-a7bff54c-2ff4-4654-9783-626542bd3a90-c000.snappy.parquet differ
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -433,6 +433,13 @@ def test_from_parquet_partitioned_columns_with_columns(make_parquet_file):
     df_equals(modin_df, pandas_df)
 
 
+def test_from_parquet_hdfs():
+    path = "modin/pandas/test/data/hdfs.parquet"
+    pandas_df = pandas.read_parquet(path)
+    modin_df = pd.read_parquet(path)
+    df_equals(modin_df, pandas_df)
+
+
 def test_from_json():
     setup_json_file(SMALL_ROW_SIZE)