Repository: modin-project/modin
Pull request: DOCS-#3766: update 'read_csv_glob' dispatcher, parser and docstring
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

<!--
Thank you for your contribution!
Please review the contributing docs: https://modin.readthedocs.io/en/latest/CONTRIBUTING.html
if you have questions about contributing.
-->

## What do these changes do?

1. Update docstring
2. An error with unpacking data is a consequence of the fact that the parser is not designed to work with a file name as a string. Problem fixed, test added.
3. Additionally added a warning if the filename does not contain `*` symbol.

<!-- Please give a short brief about these changes. -->

- [x] commit message follows format outlined [here](https://modin.readthedocs.io/en/latest/contributing.html)
- [x] passes `flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] passes `black --check modin/ asv_bench/benchmarks scripts/doc_checker.py`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #3766 <!-- issue must be created for each patch -->
- [x] tests added and passing
- [x] module layout described at `docs/developer/architecture.rst` is up-to-date <!-- if you have added, renamed or removed files or directories please update the documentation accordingly -->

Actual build doc is here: https://modin--3797.org.readthedocs.build/en/3797/advanced_usage/index.html

Problem statement:
Descriptive error message and usage documentation for read_csv_glob
### System information
- Linux Ubuntu 16.04 
- ray 1.8.0
- modin 0.12.0
- python 3.8

I'm seeing the below issue with `read_csv_glob` pointing to an S3 prefix that would results in multiple files being returned. I'm using the url `s3://nyc-tlc/trip data/yellow_tripdata_2020-` which returns several files. Note that this work when I point to an S3 url to a single `.csv` file. 

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_13805/755823039.py in <module>
      2 
      3 file_path = 's3://nyc-tlc/trip data/yellow_tripdata_2020-'
----> 4 modin_df = pd.read_csv_glob(file_path, parse_dates=["tpep_pickup_datetime", "tpep_dropoff_datetime"], quoting=3)
      5 
      6 

~/venv/lib/python3.8/site-packages/modin/experimental/pandas/io.py in parser_func(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision, storage_options)
    181 
    182         kwargs = {k: v for k, v in f_locals.items() if k in _pd_read_csv_signature}
--> 183         return _read(**kwargs)
    184 
    185     parser_func.__doc__ = _read.__doc__

~/venv/lib/python3.8/site-packages/modin/experimental/pandas/io.py in _read(**kwargs)
    206 
    207     try:
--> 208         pd_obj = FactoryDispatcher.read_csv_glob(**kwargs)
    209     except AttributeError:
    210         raise AttributeError("read_csv_glob() is only implemented for pandas on Ray.")

~/venv/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/dispatcher.py in read_csv_glob(cls, **kwargs)
    183     @_inherit_docstrings(factories.ExperimentalPandasOnRayFactory._read_csv_glob)
    184     def read_csv_glob(cls, **kwargs):
--> 185         return cls.__factory._read_csv_glob(**kwargs)
    186 
    187     @classmethod

~/venv/lib/python3.8/site-packages/modin/core/execution/dispatching/factories/factories.py in _read_csv_glob(cls, **kwargs)
    511     )
    512     def _read_csv_glob(cls, **kwargs):
--> 513         return cls.io_cls.read_csv_glob(**kwargs)
    514 
    515     @classmethod

~/venv/lib/python3.8/site-packages/modin/core/io/text/csv_glob_dispatcher.py in _read(cls, filepath_or_buffer, **kwargs)
     60         if isinstance(filepath_or_buffer, str):
     61             if not cls.file_exists(filepath_or_buffer):
---> 62                 return cls.single_worker_read(filepath_or_buffer, **kwargs)
     63             filepath_or_buffer = cls.get_path(filepath_or_buffer)
     64         elif not cls.pathlib_or_pypath(filepath_or_buffer):

~/venv/lib/python3.8/site-packages/modin/core/storage_formats/pandas/parsers.py in single_worker_read(cls, fname, **kwargs)
    267         ErrorMessage.default_to_pandas("Parameters provided")
    268         # Use default args for everything
--> 269         pandas_frame = cls.parse(fname, **kwargs)
    270         if isinstance(pandas_frame, pandas.io.parsers.TextFileReader):
    271             pd_read = pandas_frame.read

~/venv/lib/python3.8/site-packages/modin/core/storage_formats/pandas/parsers.py in parse(chunks, **kwargs)
    310 
    311         pandas_dfs = []
--> 312         for fname, start, end in chunks:
    313             if start is not None and end is not None:
    314                 # pop "compression" from kwargs because bio is uncompressed

ValueError: not enough values to unpack (expected 3, got 1)
```

Hints from issue discussion:
This issue has been mentioned on **Modin Discuss**. There might be relevant details there:

https://discuss.modin.org/t/read-csv-glob-doesnt-work-on-s3-prefixes/251/4

Hi @c3-cjazra!

I don't see `*` symbol in your URL, without which function `read_csv_glob` will not work.

Can you try `read_csv_glob("s3://nyc-tlc/trip data/yellow_tripdata_2020-*")`?
We should update the documentation for `read_csv_glob`, it works not only for local files now.
@anmyachev would it make sense to add a better error message? This error message does not inform the user, I think.
Also, `storage_options` support should be added for `read_csv_glob`. For now, only default s3 bucket with default opts will be supported for this func.
> @anmyachev would it make sense to add a better error message? This error message does not inform the user, I think.

I agree, the error message is not relevant now.
> Also, `storage_options` support should be added for `read_csv_glob`. For now, only default s3 bucket with default opts will be supported for this func.

Good point
@anmyachev `*` does work, sorry for missing that and thank you! Yes a better error message and updated documentation would certainly be desirable!
@anmyachev , are you going to do something as part of this issue or we can close it?
@YarShev this can be repurposed to add better error message and usage documentation.

Commit message: add the paragraph for 'Experimental APIs'

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

Patch to review:
diff --git a/docs/advanced_usage/index.rst b/docs/advanced_usage/index.rst
index bcacde10feb..3d8f9b380ed 100644
--- a/docs/advanced_usage/index.rst
+++ b/docs/advanced_usage/index.rst
@@ -61,6 +61,15 @@ Modin provides an implementation of distributed XGBoost machine learning algorit
 :doc:`Modin XGBoost architecture documentation </flow/modin/experimental/xgboost>` for information about implementation and
 internal execution flow.
 
+Experimental APIs
+-----------------
+Modin also supports these APIs on top of pandas that are under active experimental development.
+
+- :py:func:`~modin.experimental.pandas.read_csv_glob`
+- :py:func:`~modin.experimental.pandas.read_sql`
+- :py:func:`~modin.experimental.pandas.read_pickle_distributed`
+- :py:meth:`~modin.experimental.pandas.DataFrame.to_pickle_distributed`
+
 .. _`blog post`: https://medium.com/riselab/why-every-data-scientist-using-pandas-needs-modin-bringing-sql-to-dataframes-3b216b29a7c0
 .. _`Modin SQL documentation`: modin_sql.html
 .. _`Modin Spreadsheet API documentation`: spreadsheets_api.html
diff --git a/modin/core/io/text/csv_glob_dispatcher.py b/modin/core/io/text/csv_glob_dispatcher.py
index dd26e9aa853..c27f5b48c54 100644
--- a/modin/core/io/text/csv_glob_dispatcher.py
+++ b/modin/core/io/text/csv_glob_dispatcher.py
@@ -54,6 +54,12 @@ def _read(cls, filepath_or_buffer, **kwargs):
         # Ensures that the file is a string file path. Otherwise, default to pandas.
         filepath_or_buffer = cls.get_path_or_buffer(filepath_or_buffer)
         if isinstance(filepath_or_buffer, str):
+            is_folder = filepath_or_buffer.endswith("/")
+            if "*" not in filepath_or_buffer and not is_folder:
+                warnings.warn(
+                    "Shell-style wildcard '*' must be in the filename in order to read multiple "
+                    f"files at once; passed filename: '{filepath_or_buffer}'"
+                )
             if not cls.file_exists(filepath_or_buffer):
                 return cls.single_worker_read(filepath_or_buffer, **kwargs)
             filepath_or_buffer = cls.get_path(filepath_or_buffer)
diff --git a/modin/core/storage_formats/pandas/parsers.py b/modin/core/storage_formats/pandas/parsers.py
index 345eff28686..2f10306a5e8 100644
--- a/modin/core/storage_formats/pandas/parsers.py
+++ b/modin/core/storage_formats/pandas/parsers.py
@@ -308,6 +308,10 @@ def parse(chunks, **kwargs):
         num_splits = kwargs.pop("num_splits", None)
         index_col = kwargs.get("index_col", None)
 
+        # `single_worker_read` just pass filename via chunks; need check
+        if isinstance(chunks, str):
+            return pandas.read_csv(chunks, **kwargs)
+
         pandas_dfs = []
         for fname, start, end in chunks:
             if start is not None and end is not None:
diff --git a/modin/experimental/pandas/io.py b/modin/experimental/pandas/io.py
index 94843ca5977..a06ffd43d3f 100644
--- a/modin/experimental/pandas/io.py
+++ b/modin/experimental/pandas/io.py
@@ -191,7 +191,7 @@ def _read(**kwargs) -> DataFrame:
     General documentation is available in `modin.pandas.read_csv`.
 
     This experimental feature provides parallel reading from multiple csv files which are
-    defined by glob pattern. Works for local files only!
+    defined by glob pattern.
 
     Parameters
     ----------
@@ -201,6 +201,27 @@ def _read(**kwargs) -> DataFrame:
     Returns
     -------
     modin.DataFrame
+
+    Examples
+    --------
+    >>> import modin.experimental.pandas as pd
+    >>> df = pd.read_csv_glob("s3://nyc-tlc/trip data/yellow_tripdata_2020-1*")
+    UserWarning: `read_*` implementation has mismatches with pandas:
+    Data types of partitions are different! Please refer to the troubleshooting section of the Modin documentation to fix this issue.
+            VendorID tpep_pickup_datetime  ... total_amount  congestion_surcharge
+    0             1.0  2020-10-01 00:09:08  ...         4.30                   0.0
+    1             1.0  2020-10-01 00:09:19  ...        13.30                   2.5
+    2             1.0  2020-10-01 00:30:00  ...        15.36                   2.5
+    3             2.0  2020-10-01 00:56:46  ...        -3.80                   0.0
+    4             2.0  2020-10-01 00:56:46  ...         3.80                   0.0
+    ...           ...                  ...  ...          ...                   ...
+    4652008       NaN  2020-12-31 23:44:35  ...        43.95                   2.5
+    4652009       NaN  2020-12-31 23:41:36  ...        20.17                   2.5
+    4652010       NaN  2020-12-31 23:01:17  ...        78.98                   0.0
+    4652011       NaN  2020-12-31 23:31:29  ...        39.50                   0.0
+    4652012       NaN  2020-12-31 23:12:48  ...        20.64                   0.0
+
+    [4652013 rows x 18 columns]
     """
     Engine.subscribe(_update_engine)
 
diff --git a/modin/experimental/pandas/test/test_io_exp.py b/modin/experimental/pandas/test/test_io_exp.py
index a3eeb3f5331..fb27c8e1ef7 100644
--- a/modin/experimental/pandas/test/test_io_exp.py
+++ b/modin/experimental/pandas/test/test_io_exp.py
@@ -111,6 +111,11 @@ def test_read_csv_empty_frame(self):
 
         df_equals(modin_df, pandas_df)
 
+    def test_read_csv_without_glob(self):
+        with pytest.warns(UserWarning, match=r"Shell-style wildcard"):
+            with pytest.raises(FileNotFoundError):
+                pd.read_csv_glob("s3://nyc-tlc/trip data/yellow_tripdata_2020-")
+
 
 @pytest.mark.skipif(
     Engine.get() != "Ray", reason="Currently only support Ray engine for glob paths."


Code review comments:
diff_hunk: @@ -61,6 +61,15 @@ Modin provides an implementation of distributed XGBoost machine learning algorit
 :doc:`Modin XGBoost architecture documentation </flow/modin/experimental/xgboost>` for information about implementation and
 internal execution flow.
 
+Experimental APIs
+-----------------
+Modin also supports these APIs on top of pandas that are under active experimental development.
+
+- :py:func:`~modin.experimental.pandas.read_csv_glob`
+- :py:func:`~modin.experimental.pandas.read_sql`
path: docs/advanced_usage/index.rst
text: @user1:
```suggestion
- :py:func:`~modin.experimental.pandas.read_sql` -- add optional parameters for the database connection
```

diff_hunk: @@ -61,6 +61,15 @@ Modin provides an implementation of distributed XGBoost machine learning algorit
 :doc:`Modin XGBoost architecture documentation </flow/modin/experimental/xgboost>` for information about implementation and
 internal execution flow.
 
+Experimental APIs
+-----------------
+Modin also supports these APIs on top of pandas that are under active experimental development.
+
+- :py:func:`~modin.experimental.pandas.read_csv_glob`
+- :py:func:`~modin.experimental.pandas.read_sql`
+- :py:func:`~modin.experimental.pandas.read_pickle_distributed`
+- :py:meth:`~modin.experimental.pandas.DataFrame.to_pickle_distributed`
path: docs/advanced_usage/index.rst
text: @user1:
```suggestion
- :py:meth:`~modin.experimental.pandas.DataFrame.to_pickle_distributed` -- write to multiple files in a directory
```

diff_hunk: @@ -61,6 +61,15 @@ Modin provides an implementation of distributed XGBoost machine learning algorit
 :doc:`Modin XGBoost architecture documentation </flow/modin/experimental/xgboost>` for information about implementation and
 internal execution flow.
 
+Experimental APIs
+-----------------
+Modin also supports these APIs on top of pandas that are under active experimental development.
+
+- :py:func:`~modin.experimental.pandas.read_csv_glob`
path: docs/advanced_usage/index.rst
text: @user1:
```suggestion
- :py:func:`~modin.experimental.pandas.read_csv_glob` -- read multiple files in a directory
```

diff_hunk: @@ -61,6 +61,15 @@ Modin provides an implementation of distributed XGBoost machine learning algorit
 :doc:`Modin XGBoost architecture documentation </flow/modin/experimental/xgboost>` for information about implementation and
 internal execution flow.
 
+Experimental APIs
+-----------------
+Modin also supports these APIs on top of pandas that are under active experimental development.
+
+- :py:func:`~modin.experimental.pandas.read_csv_glob`
+- :py:func:`~modin.experimental.pandas.read_sql`
+- :py:func:`~modin.experimental.pandas.read_pickle_distributed`
path: docs/advanced_usage/index.rst
text: @user1:
```suggestion
- :py:func:`~modin.experimental.pandas.read_pickle_distributed`  -- read multiple files in a directory
```