Repository: modin-project/modin

Issue:
BUG: Cannot read parquet file from s3 bucket when using Modin 0.16.1 on Ray
### Modin version checks

- [X] I have checked that this issue has not already been reported.

- [X] I have confirmed this bug exists on the latest released version of Modin.

- [X] I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow [this guide](https://modin.readthedocs.io/en/stable/getting_started/installation.html#installing-from-the-github-master-branch).)


### Reproducible Example

```python
import modin.pandas as pd

input_path = 's3://<bucket>/data/'
df = pd.read_parquet(input_path)
```


### Issue Description

When I call `pd.read_parquet('s3://<bucket>/data/')` to read parquet files from s3, it throws error `pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path '<bucket>/data/xxx.snappy.parquet', which is outside base dir 's3://<bucket>/data/'`

### Expected Behavior

When call `pd.read_parquet('s3://<bucket>/data/')`, it should successfully read the data from s3 bucket.

### Error Logs

<details>

```python-traceback

Traceback (most recent call last):
  File "modin_api_test.py", line 195, in <module>
    df = pd.read_parquet(input_path)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/_compat/pandas_api/latest/io.py", line 291, in read_parquet
    query_compiler=FactoryDispatcher.read_parquet(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/dispatcher.py", line 180, in read_parquet
    return cls.__factory._read_parquet(**kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/factories.py", line 207, in _read_parquet
    return cls.io_cls.read_parquet(**kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/io/file_dispatcher.py", line 154, in read
    query_compiler = cls._read(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 648, in _read
    return cls.build_query_compiler(dataset, columns, index_columns, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 560, in build_query_compiler
    index, sync_index = cls.build_index(dataset, partition_ids, index_columns)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/logging/logger_decorator.py", line 128, in run_and_log
    return obj(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 522, in build_index
    complete_index = dataset.to_pandas_dataframe(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/modin/core/io/column_stores/parquet_dispatcher.py", line 226, in to_pandas_dataframe
    return read_table(self.path, columns=columns, filesystem=self.fs).to_pandas()
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/parquet/__init__.py", line 2780, in read_table
    dataset = _ParquetDatasetV2(
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/parquet/__init__.py", line 2379, in __init__
    self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/dataset.py", line 749, in dataset
    return _filesystem_dataset(source, **kwargs)
  File "/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/dataset.py", line 449, in _filesystem_dataset
    factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
  File "pyarrow/_dataset.pyx", line 2059, in pyarrow._dataset.FileSystemDatasetFactory.__init__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path '<bucket>/data/xxx.snappy.parquet', which is outside base dir 's3://<bucket>/data/'
```

</details>


### Installed Versions

<details>

INSTALLED VERSIONS
------------------
commit           : 98a9694694584f50d03f6e27b516eb10d18d36ef
python           : 3.8.10.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.13.0-1025-aws
Version          : #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

Modin dependencies
------------------
modin            : 0.16.1
ray              : 2.0.0
dask             : None
distributed      : None
hdk              : None

pandas dependencies
-------------------
pandas           : 1.5.0
numpy            : 1.23.4
pytz             : 2022.4
dateutil         : 2.8.2
setuptools       : 45.2.0
pip              : 20.0.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : 8.5.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : None
brotli           : None
fastparquet      : 0.8.3
fsspec           : 2022.8.2
gcsfs            : None
matplotlib       : None
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 9.0.0
pyreadstat       : None
pyxlsb           : None
s3fs             : 2022.8.2
scipy            : None
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
zstandard        : None
tzdata           : None

</details>



Discussion:
This issue has been mentioned on **Modin Discuss**. There might be relevant details there:

https://discuss.modin.org/t/set-scheduling-strategy-to-spread-when-using-modin-on-ray/329/11

Thanks @mwjinjin for reporting this issue. I can't reproduce your error because I don't have access to your s3 directory, but I understand why you might not be able to share that. Some questions:

- Are you able to create a publicly available s3 directory to demonstrate the error? 
- Can pandas read the parquet directory? i.e. replace `modin.pandas` with `pandas` in the script you share. What happens?

Modin unit tests check that modin and pandas can read a publicly available s3 parquet file at `s3://modin-datasets/testing/test_data.parquet`: https://github.com/modin-project/modin/blob/f241ee0401a82df45afd7abfaef384093399364f/modin/pandas/test/test_io.py#L1588-L1589

However, it seems that neither pandas nor modin can read the directory `s3://modin-datasets/testing/`:

```python
import modin.pandas as pd
import pandas

# both of these work
print(pandas.read_parquet("s3://modin-datasets/testing/test_data.parquet", storage_options=dict(anon=True)))
print(pd.read_parquet("s3://modin-datasets/testing/test_data.parquet", storage_options=dict(anon=True)))

# both throw errors
print(pandas.read_parquet("s3://modin-datasets/testing/", storage_options=dict(anon=True)))
print(pd.read_parquet("s3://modin-datasets/testing/", storage_options=dict(anon=True)))
```

Both pandas and modin give me `FileNotFoundError: modin-datasets/testing/`.
Facing the same issue. It can be reproduced with a standard TPC-H dataset. And I have one of the tables attached.

And pandas can read the directory without an error:
```
In [1]: import pandas as pd

In [2]: df = pd.read_parquet("s3://my_bucket/part/")

In [3]: df
Out[3]:
        P_PARTKEY                                    P_NAME          P_MFGR  ... P_CONTAINER P_RETAILPRICE               P_COMMENT
0               1  goldenrod lavender spring chocolate lace  Manufacturer#1  ...   JUMBO PKG        901.00         ly. slyly ironi
1               2          blush thistle blue yellow saddle  Manufacturer#1  ...     LG CASE        902.00        lar accounts amo
2               3       spring green yellow purple cornsilk  Manufacturer#4  ...   WRAP CASE        903.00     egular deposits hag
3               4     cornflower chocolate smoke green pink  Manufacturer#3  ...    MED DRUM        904.00           p furiously r
4               5             forest brown coral puff cream  Manufacturer#3  ...      SM PKG        905.00         wake carefully
...           ...                                       ...             ...  ...         ...           ...                     ...
199995     199996           cream navajo saddle dodger navy  Manufacturer#4  ...    MED PACK       2095.99              silent dep
199996     199997          peru maroon snow grey chartreuse  Manufacturer#4  ...     SM DRUM       2096.99    furiously after the
199997     199998          pink wheat powder burlywood snow  Manufacturer#5  ...      LG BOX       2097.99  . special deposits hag
199998     199999          goldenrod drab brown salmon mint  Manufacturer#5  ...     LG CASE       2098.99       he quickly ironic
199999     200000            peach royal cornsilk sky sandy  Manufacturer#5  ...      LG CAN       1100.00         xes sleep quick

[200000 rows x 9 columns]
```

[part.zip](https://github.com/modin-project/modin/files/9783752/part.zip)

@UranusSeven thank you for your comment. I put your data in a publicly accessible bucket I can use to reproduce the issue:

```python
import pandas
import modin.pandas as pd

print(pandas.read_parquet("s3://mahesh-vashishtha/part"))
print('pandas succeeded')
print(pd.read_parquet("s3://mahesh-vashishtha/part"))
```

I'll see what we can do to fix this.

<details>

<summary>Stack trace</summary>

```python-traceback
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
Input In [14], in <cell line: 1>()
----> 1 pd.read_parquet("s3://mahesh-vashishtha/part")

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/_compat/pandas_api/latest/io.py:291, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
    287 Engine.subscribe(_update_engine)
    288 from modin.core.execution.dispatching.factories.dispatcher import FactoryDispatcher
    290 return DataFrame(
--> 291     query_compiler=FactoryDispatcher.read_parquet(
    292         path=path,
    293         engine=engine,
    294         columns=columns,
    295         storage_options=storage_options,
    296         use_nullable_dtypes=use_nullable_dtypes,
    297         **kwargs,
    298     )
    299 )

File ~/software_sources/modin/modin/core/execution/dispatching/factories/dispatcher.py:180, in FactoryDispatcher.read_parquet(cls, **kwargs)
    177 @classmethod
    178 @_inherit_docstrings(factories.BaseFactory._read_parquet)
    179 def read_parquet(cls, **kwargs):
--> 180     return cls.__factory._read_parquet(**kwargs)

File ~/software_sources/modin/modin/core/execution/dispatching/factories/factories.py:207, in BaseFactory._read_parquet(cls, **kwargs)
    199 @classmethod
    200 @doc(
    201     _doc_io_method_template,
   (...)
    205 )
    206 def _read_parquet(cls, **kwargs):
--> 207     return cls.io_cls.read_parquet(**kwargs)

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/io/file_dispatcher.py:154, in FileDispatcher.read(cls, *args, **kwargs)
    131 @classmethod
    132 def read(cls, *args, **kwargs):
    133     """
    134     Read data according passed `args` and `kwargs`.
    135
   (...)
    152     postprocessing work on the resulting query_compiler object.
    153     """
--> 154     query_compiler = cls._read(*args, **kwargs)
    155     # TODO (devin-petersohn): Make this section more general for non-pandas kernel
    156     # implementations.
    157     if StorageFormat.get() == "Pandas":

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/io/column_stores/parquet_dispatcher.py:648, in ParquetDispatcher._read(cls, path, engine, columns, **kwargs)
    641 column_names = columns if columns else dataset.columns
    642 columns = [
    643     c
    644     for c in column_names
    645     if c not in index_columns and not PQ_INDEX_REGEX.match(c)
    646 ]
--> 648 return cls.build_query_compiler(dataset, columns, index_columns, **kwargs)

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/io/column_stores/parquet_dispatcher.py:560, in ParquetDispatcher.build_query_compiler(cls, dataset, columns, index_columns, **kwargs)
    556 col_partitions, column_widths = cls.build_columns(columns)
    557 partition_ids = cls.call_deploy(
    558     dataset, col_partitions, storage_options, **kwargs
    559 )
--> 560 index, sync_index = cls.build_index(dataset, partition_ids, index_columns)
    561 remote_parts = cls.build_partition(partition_ids, column_widths)
    562 if len(partition_ids) > 0:

File ~/software_sources/modin/modin/logging/logger_decorator.py:128, in enable_logging.<locals>.decorator.<locals>.run_and_log(*args, **kwargs)
    113 """
    114 Compute function with logging if Modin logging is enabled.
    115
   (...)
    125 Any
    126 """
    127 if LogMode.get() == "disable":
--> 128     return obj(*args, **kwargs)
    130 logger = get_logger()
    131 logger_level = getattr(logger, log_level)

File ~/software_sources/modin/modin/core/io/column_stores/parquet_dispatcher.py:522, in ParquetDispatcher.build_index(cls, dataset, partition_ids, index_columns)
    519 # For the second check, let us consider the case where we have an empty dataframe,
    520 # that has a valid index.
    521 if range_index or (len(partition_ids) == 0 and len(column_names_to_read) != 0):
--> 522     complete_index = dataset.to_pandas_dataframe(
    523         columns=column_names_to_read
    524     ).index
    525 # Empty DataFrame case
    526 elif len(partition_ids) == 0:

File ~/software_sources/modin/modin/core/io/column_stores/parquet_dispatcher.py:226, in PyArrowDataset.to_pandas_dataframe(self, columns)
    220 def to_pandas_dataframe(
    221     self,
    222     columns,
    223 ):
    224     from pyarrow.parquet import read_table
--> 226     return read_table(self.path, columns=columns, filesystem=self.fs).to_pandas()

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/parquet/__init__.py:2737, in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties)
   2730     raise ValueError(
   2731         "The 'metadata' keyword is no longer supported with the new "
   2732         "datasets-based implementation. Specify "
   2733         "'use_legacy_dataset=True' to temporarily recover the old "
   2734         "behaviour."
   2735     )
   2736 try:
-> 2737     dataset = _ParquetDatasetV2(
   2738         source,
   2739         schema=schema,
   2740         filesystem=filesystem,
   2741         partitioning=partitioning,
   2742         memory_map=memory_map,
   2743         read_dictionary=read_dictionary,
   2744         buffer_size=buffer_size,
   2745         filters=filters,
   2746         ignore_prefixes=ignore_prefixes,
   2747         pre_buffer=pre_buffer,
   2748         coerce_int96_timestamp_unit=coerce_int96_timestamp_unit
   2749     )
   2750 except ImportError:
   2751     # fall back on ParquetFile for simple cases when pyarrow.dataset
   2752     # module is not available
   2753     if filters is not None:

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/parquet/__init__.py:2351, in _ParquetDatasetV2.__init__(self, path_or_paths, filesystem, filters, partitioning, read_dictionary, buffer_size, memory_map, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, schema, decryption_properties, **kwargs)
   2347 if partitioning == "hive":
   2348     partitioning = ds.HivePartitioning.discover(
   2349         infer_dictionary=True)
-> 2351 self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
   2352                            schema=schema, format=parquet_format,
   2353                            partitioning=partitioning,
   2354                            ignore_prefixes=ignore_prefixes)

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/dataset.py:694, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
    683 kwargs = dict(
    684     schema=schema,
    685     filesystem=filesystem,
   (...)
    690     selector_ignore_prefixes=ignore_prefixes
    691 )
    693 if _is_path_like(source):
--> 694     return _filesystem_dataset(source, **kwargs)
    695 elif isinstance(source, (tuple, list)):
    696     if all(_is_path_like(elem) for elem in source):

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/dataset.py:447, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
    439     fs, paths_or_selector = _ensure_single_source(source, filesystem)
    441 options = FileSystemFactoryOptions(
    442     partitioning=partitioning,
    443     partition_base_dir=partition_base_dir,
    444     exclude_invalid_files=exclude_invalid_files,
    445     selector_ignore_prefixes=selector_ignore_prefixes
    446 )
--> 447 factory = FileSystemDatasetFactory(fs, paths_or_selector, format, options)
    449 return factory.finish(schema)

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/_dataset.pyx:2031, in pyarrow._dataset.FileSystemDatasetFactory.__init__()

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/opt/anaconda3/envs/modin-dev/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: GetFileInfo() yielded path 'mahesh-vashishtha/part/part-099.pq', which is outside base dir 's3://mahesh-vashishtha/part'
```
</details>

Resolving patch:
diff --git a/modin/core/io/column_stores/parquet_dispatcher.py b/modin/core/io/column_stores/parquet_dispatcher.py
--- a/modin/core/io/column_stores/parquet_dispatcher.py
+++ b/modin/core/io/column_stores/parquet_dispatcher.py
@@ -223,7 +223,9 @@ def to_pandas_dataframe(
     ):
         from pyarrow.parquet import read_table
 
-        return read_table(self.path, columns=columns, filesystem=self.fs).to_pandas()
+        return read_table(
+            self._fs_path, columns=columns, filesystem=self.fs
+        ).to_pandas()
 
 
 @_inherit_docstrings(ColumnStoreDataset)


Test patch:
diff --git a/modin/pandas/test/test_io.py b/modin/pandas/test/test_io.py
--- a/modin/pandas/test/test_io.py
+++ b/modin/pandas/test/test_io.py
@@ -1579,7 +1579,20 @@ def test_read_parquet_hdfs(self, engine):
             engine=engine,
         )
 
-    @pytest.mark.parametrize("path_type", ["url", "object"])
+    @pytest.mark.parametrize(
+        "path_type",
+        [
+            "object",
+            "directory",
+            pytest.param(
+                "url",
+                marks=pytest.mark.xfail(
+                    PandasCompatVersion.CURRENT == PandasCompatVersion.PY36,
+                    reason="older pandas.read_parquet does not support storage_options",
+                ),
+            ),
+        ],
+    )
     @pytest.mark.parametrize("engine", ["pyarrow", "fastparquet"])
     @pytest.mark.xfail(
         condition="config.getoption('--simulate-cloud').lower() != 'off'",
@@ -1593,6 +1606,25 @@ def test_read_parquet_s3(self, path_type, engine):
             fs = s3fs.S3FileSystem(anon=True)
             with fs.open(dataset_url, "rb") as file_obj:
                 eval_io("read_parquet", path=file_obj, engine=engine)
+        elif path_type == "directory":
+            if PandasCompatVersion.CURRENT == PandasCompatVersion.PY36:
+                if engine == "pyarrow":
+                    pytest.xfail(
+                        reason="older pandas.read_parquet does not support storage_options"
+                    )
+                else:
+                    # older pandas.read_parquet does not support
+                    # storage_options, but modin and pandas both happen to
+                    # give different ValueErrors, so we cannot xfail.
+                    # n.b. conda doesn't seem to have pytest>6.2.5 for python
+                    # 3.6, and pytest 6.2.5 doesn't accept "skip" parameter.
+                    pytest.skip()
+            eval_io(
+                "read_parquet",
+                path="s3://modin-datasets/test_data_dir.parquet",
+                storage_options={"anon": True},
+                engine=engine,
+            )
         else:
             if PandasCompatVersion.CURRENT == PandasCompatVersion.PY36:
                 pytest.xfail(