Repository: modin-project/modin

Issue:
modin.pandas.read_parquet() files from HDFS working unexpectedly as compared to pandas.read_parquet()
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Modin version** (`modin.__version__`): 0.7.2
- **Python version**: 3.6.6


<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Hi I am trying to read parquet files from remote HDFS, which is working fine if I am using Pandas directly, but it is throwing ERROR when using  Modin -->

### Source code / logs
>>> import modin.pandas as pd1
^[[Aos.environ['JAVA_HOME'] = '/usr/jdk64/jdk1.8.0_112/jre'
KeyboardInterrupt
>>> os.environ["MODIN_ENGINE"] = "dask"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'os' is not defined
>>> import os
>>> os.environ["MODIN_ENGINE"] = "dask"
>>> import modin.pandas as pd1
>>> import pandas as pd
>>> import numpy as np
>>> import pyarrow
i>>> import os
s>>> os.environ['ARROW_LIBHDFS_DIR'] = '/usr/hdp/3.1.4.0-315/usr/lib'
>>> os.environ['JAVA_HOME'] = '/usr/jdk64/jdk1.8.0_112/jre'
>>>
>>>
>>> df = pd.read_parquet('hdfs://ip:port/abc.parquet’)
2020-03-17 00:35:29,552 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>> df1 = pd1.read_parquet('hdfs://ip:port/abc.parquet')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/username/.local/lib/python3.6/site-packages/modin/pandas/io.py", line 27, in read_parquet
    path=path, columns=columns, engine=engine, **kwargs
  File "/home/username/.local/lib/python3.6/site-packages/modin/data_management/factories.py", line 44, in read_parquet
    return cls._determine_engine()._read_parquet(**kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/modin/data_management/factories.py", line 48, in _read_parquet
    return cls.io_cls.read_parquet(**kwargs)
  File "/home/username/.local/lib/python3.6/site-packages/modin/engines/ray/generic/io.py", line 128, in read_parquet
    pf = ParquetFile(path)
  File "/home/username/.local/lib/python3.6/site-packages/ray/pyarrow_files/pyarrow/parquet.py", line 132, in __init__
    self.reader.open(source, use_memory_map=memory_map, metadata=metadata)
  File "pyarrow/_parquet.pyx", line 1002, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/io.pxi", line 1323, in pyarrow.lib.get_reader
  File "pyarrow/io.pxi", line 1312, in pyarrow.lib._get_native_file
  File "pyarrow/io.pxi", line 780, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 743, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 78, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Failed to open local file 'hdfs://ip:port/abc.parquet’, error: No such file or directory

<!--

-->


Discussion:
Hi @Nishant23 thanks for reporting!

We haven't implemented the HDFS support yet for parquet. The error is from Arrow because we have passed it a path that contains the address of the hdfs but it cannot handle that case.

Looking at this, it will be straightforward to implement and will be implemented this release. Looking through the other filetypes, it appears that we have HDFS support for those, but I will test this to be sure.
Hello,
I have been working on ray for 1 month. I can read txt file from local using Modin  but I want to read a parquet file on Hdfs.
Can you help me about how can i read a parquet file on hdfs with modin or ray  ?
Hi @kocmurat93!
We haven't implemented the HDFS support yet for `modin.pandas.read_parquet`. I am just working with this problem. We will notify you when it's done. Thank you!


Resolving patch:
diff --git a/modin/engines/base/io/column_stores/parquet_reader.py b/modin/engines/base/io/column_stores/parquet_reader.py
--- a/modin/engines/base/io/column_stores/parquet_reader.py
+++ b/modin/engines/base/io/column_stores/parquet_reader.py
@@ -21,18 +21,29 @@ class ParquetReader(ColumnStoreReader):
     @classmethod
     def _read(cls, path, engine, columns, **kwargs):
         """Load a parquet object from the file path, returning a Modin DataFrame.
-           Modin only supports pyarrow engine for now.
 
-        Args:
-            path: The filepath of the parquet file.
-                  We only support local files for now.
-            engine: Modin only supports pyarrow reader.
-                    This argument doesn't do anything for now.
-            kwargs: Pass into parquet's read_pandas function.
+        Modin only supports pyarrow engine for now.
 
-        Notes:
-            ParquetFile API is used. Please refer to the documentation here
-            https://arrow.apache.org/docs/python/parquet.html
+        Parameters
+        ----------
+        path: str
+            The filepath of the parquet file in local filesystem or hdfs.
+        engine: 'pyarrow'
+            Parquet library to use
+        columns: list or None
+            If not None, only these columns will be read from the file.
+        kwargs: dict
+            Keyword arguments.
+
+        Returns
+        -------
+        PandasQueryCompiler
+            A new Query Compiler.
+
+        Notes
+        -----
+        ParquetFile API is used. Please refer to the documentation here
+        https://arrow.apache.org/docs/python/parquet.html
         """
         from pyarrow.parquet import ParquetFile, ParquetDataset
         from modin.pandas.io import PQ_INDEX_REGEX
@@ -66,6 +77,13 @@ def _read(cls, path, engine, columns, **kwargs):
                 pd = ParquetDataset(path)
                 meta = pd.metadata
                 column_names = pd.schema.names
+            elif isinstance(path, str) and path.startswith("hdfs://"):
+                import fsspec.core
+
+                fs, path = fsspec.core.url_to_fs(path)
+                pd = ParquetDataset(path, filesystem=fs)
+                meta = pd.metadata
+                column_names = pd.schema.names
             else:
                 meta = ParquetFile(path).metadata
                 column_names = meta.schema.names


Test patch: