Repository: dask/dask

Issue:
`require_extension` regression in `read_parquet`
This example requires adlfs and planetary-computer (`pip install adlfs planetary-computer`)

```python
>>> import dask.dataframe as dd, planetary_computer
>>> credential = planetary_computer.sas.get_token("ai4edataeuwest", "gbif").token
>>> dd.read_parquet('abfs://gbif/occurrence/2021-10-01/occurrence.parquet', storage_options=dict(account_name="ai4edataeuwest", credential=credential))
```

This raises with

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-3665ff1dc1a7> in <module>
      1 import dask.dataframe as dd, planetary_computer
      2 credential = planetary_computer.sas.get_token("ai4edataeuwest", "gbif").token
----> 3 dd.read_parquet('abfs://gbif/occurrence/2021-10-01/occurrence.parquet', storage_options=dict(account_name="ai4edataeuwest", credential=credential))

~/src/dask/dask/dask/dataframe/io/parquet/core.py in read_parquet(path, columns, filters, categories, index, storage_options, engine, gather_statistics, ignore_metadata_file, metadata_task_size, split_row_groups, chunksize, aggregate_files, **kwargs)
    324         gather_statistics = True
    325
--> 326     read_metadata_result = engine.read_metadata(
    327         fs,
    328         paths,

~/src/dask/dask/dask/dataframe/io/parquet/arrow.py in read_metadata(cls, fs, paths, categories, index, gather_statistics, filters, split_row_groups, chunksize, aggregate_files, ignore_metadata_file, metadata_task_size, **kwargs)
    328
    329         # Stage 1: Collect general dataset information
--> 330         dataset_info = cls._collect_dataset_info(
    331             paths,
    332             fs,

~/src/dask/dask/dask/dataframe/io/parquet/arrow.py in _collect_dataset_info(cls, paths, fs, categories, index, gather_statistics, filters, split_row_groups, chunksize, aggregate_files, ignore_metadata_file, metadata_task_size, require_extension, **dataset_kwargs)
    827                 ]
    828                 if len0 and paths == []:
--> 829                     raise ValueError(
    830                         "No files satisfy the `require_extension` criteria "
    831                         f"(files must end with {require_extension})."

ValueError: No files satisfy the `require_extension` criteria (files must end with ('.parq', '.parquet')).
```

Related to https://github.com/dask/dask/issues/8349. Is this a deliberate change and I'll need to update the code? Or is it unintentonal?

In case it helps, here's the directory structure:

```
In [5]: import adlfs

In [6]: fs = adlfs.AzureBlobFileSystem("ai4edataeuwest", credential=credential)
In [8]: fs.ls("gbif/occurrence/2021-10-01")
Out[8]:
['gbif/occurrence/2021-10-01/citation.txt',
 'gbif/occurrence/2021-10-01/occurrence.parquet']

In [9]: fs.ls("gbif/occurrence/2021-10-01/occurrence.parquet")[:10]
Out[9]:
['gbif/occurrence/2021-10-01/occurrence.parquet/000000',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000001',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000002',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000003',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000004',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000005',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000006',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000007',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000008',
 'gbif/occurrence/2021-10-01/occurrence.parquet/000009']
```

cc @rjzamora.


Discussion:
Reading through things a bit more closely, and looking at that file structure, I guess this might be the expected behavior?

I'm not seeing `require_extension` documented anywhere. IIUC it's passed through like `read_parquet(..., dataset={"required_extension": None})`. I don't see `dataset` documented either. Is that a pyarrow specific keyword, or can it be added as a top-level keyword argument?
> Reading through things a bit more closely, and looking at that file structure, I guess this might be the expected behavior?

Ah - Right. You will need to disable the required file extensions if you only specify a directory (without a _metadata) file. This behavior was added for Spark compatibility.

>I'm not seeing require_extension documented anywhere. IIUC it's passed through like read_parquet(..., dataset={"required_extension": None}). I don't see dataset documented either. Is that a pyarrow specific keyword, or can it be added as a top-level keyword argument?

I think the lack of `required_extension` documentation is a mistake, and the documentation for `dataset` ["exists"](https://github.com/dask/dask/blob/a86ff77a08809d7b5ab3712b468acea29b16f443/dask/dataframe/io/parquet/core.py#L241), but is pretty terrible. Note that #8339 improves the 'dataset' description slightly, but the real problem is that the `read=` and `dataset=` parqameters are described under `kwargs` (which doesn't really makes much sense at all).

Perhaps it would be best to add explict `dataset_options` and `read_options` arguments, with dedicated documentation? (For backward compatibility, we can redirect `read`/`dataset` arguments when they are still specified). 
Thanks. So to confirm, if I added a `_metadata` file then the additional keyword arguments wouldn't be necessary?

> Perhaps it would be best to add explict dataset_options and read_options arguments, with dedicated documentation? (For backward compatibility, we can redirect read/dataset arguments when they are still specified).

Yeah, I think that's a great idea. I can take that on if you want, though perhaps https://github.com/dask/dask/pull/8339/ should go in first.
> I can take that on if you want, though perhaps #8339 should go in first.

If you are motivated to do it soon, then don't worry too much about #8339 (I don't mind revising it if needed) :)

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -319,6 +319,7 @@ def read_metadata(
         aggregate_files=None,
         ignore_metadata_file=False,
         metadata_task_size=0,
+        parquet_file_extension=None,
         **kwargs,
     ):
 
@@ -335,6 +336,7 @@ def read_metadata(
             aggregate_files,
             ignore_metadata_file,
             metadata_task_size,
+            parquet_file_extension,
             kwargs,
         )
 
@@ -761,6 +763,7 @@ def _collect_dataset_info(
         aggregate_files,
         ignore_metadata_file,
         metadata_task_size,
+        parquet_file_extension,
         kwargs,
     ):
         """pyarrow.dataset version of _collect_dataset_info
@@ -790,11 +793,6 @@ def _collect_dataset_info(
             {"obj": pa_ds.HivePartitioning},
         )
 
-        # Set require_extension option
-        require_extension = _dataset_kwargs.pop(
-            "require_extension", (".parq", ".parquet")
-        )
-
         # Case-dependent pyarrow.dataset creation
         has_metadata_file = False
         if len(paths) == 1 and fs.isdir(paths[0]):
@@ -819,17 +817,19 @@ def _collect_dataset_info(
                 has_metadata_file = True
                 if gather_statistics is None:
                     gather_statistics = True
-            elif require_extension:
+            elif parquet_file_extension:
                 # Need to materialize all paths if we are missing the _metadata file
                 # Raise error if all files have been filtered by extension
                 len0 = len(paths)
                 paths = [
-                    path for path in fs.find(paths) if path.endswith(require_extension)
+                    path
+                    for path in fs.find(paths)
+                    if path.endswith(parquet_file_extension)
                 ]
                 if len0 and paths == []:
                     raise ValueError(
-                        "No files satisfy the `require_extension` criteria "
-                        f"(files must end with {require_extension})."
+                        "No files satisfy the `parquet_file_extension` criteria "
+                        f"(files must end with {parquet_file_extension})."
                     )
 
         elif len(paths) > 1:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -172,6 +172,7 @@ def read_parquet(
     split_row_groups=None,
     chunksize=None,
     aggregate_files=None,
+    parquet_file_extension=(".parq", ".parquet", ".pq"),
     **kwargs,
 ):
     """
@@ -300,6 +301,18 @@ def read_parquet(
                 └── └── 04.parquet
 
         Note that the default behavior of ``aggregate_files`` is False.
+    parquet_file_extension: str, tuple[str], or None, default (".parq", ".parquet", ".pq")
+        A file extension (or an iterable of extensions) to use when discovering
+        parquet files in a directory. Files that don't match these extensions
+        will be ignored. This argument only applies when ``paths`` corresponds
+        to a directory and no ``_metadata`` file is present (or
+        ``ignore_metadata_file=True``). Passing in ``parquet_file_extension=None``
+        will treat all files in the directory as parquet files.
+
+        The purpose of this argument is to ensure that the engine will ignore
+        unsupported metadata files (like Spark's '_SUCCESS' and 'crc' files).
+        It may be necessary to change this argument if the data files in your
+        parquet dataset do not end in ".parq", ".parquet", or ".pq".
     **kwargs: dict (of dicts)
         Passthrough key-word arguments for read backend.
         The top-level keys correspond to the appropriate operation type, and
@@ -331,6 +344,24 @@ def read_parquet(
             FutureWarning,
         )
 
+    # We support a top-level `parquet_file_extension` kwarg, but
+    # must check if the deprecated `require_extension` option is
+    # being passed to the engine. If `parquet_file_extension` is
+    # set to the default value, and `require_extension` was also
+    # specified, we will use `require_extension` but warn the user.
+    if (
+        "dataset" in kwargs
+        and "require_extension" in kwargs["dataset"]
+        and parquet_file_extension == (".parq", ".parquet", ".pq")
+    ):
+        parquet_file_extension = kwargs["dataset"].pop("require_extension")
+        warnings.warn(
+            "require_extension is deprecated, and will be removed from "
+            "read_parquet in a future release. Please use the top-level "
+            "parquet_file_extension argument instead.",
+            FutureWarning,
+        )
+
     # Store initial function arguments
     input_kwargs = {
         "columns": columns,
@@ -345,6 +376,7 @@ def read_parquet(
         "split_row_groups": split_row_groups,
         "chunksize=": chunksize,
         "aggregate_files": aggregate_files,
+        "parquet_file_extension": parquet_file_extension,
         **kwargs,
     }
 
@@ -398,6 +430,7 @@ def read_parquet(
         aggregate_files=aggregate_files,
         ignore_metadata_file=ignore_metadata_file,
         metadata_task_size=metadata_task_size,
+        parquet_file_extension=parquet_file_extension,
         **kwargs,
     )
 
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -372,6 +372,7 @@ def _collect_dataset_info(
         aggregate_files,
         ignore_metadata_file,
         metadata_task_size,
+        parquet_file_extension,
         kwargs,
     ):
 
@@ -391,9 +392,6 @@ def _collect_dataset_info(
 
         parts = []
         _metadata_exists = False
-        require_extension = dataset_kwargs.pop(
-            "require_extension", (".parq", ".parquet")
-        )
         if len(paths) == 1 and fs.isdir(paths[0]):
 
             # This is a directory.
@@ -430,14 +428,16 @@ def _collect_dataset_info(
                 # Use 0th file
                 # Note that "_common_metadata" can cause issues for
                 # partitioned datasets.
-                if require_extension:
+                if parquet_file_extension:
                     # Raise error if all files have been filtered by extension
                     len0 = len(paths)
-                    paths = [path for path in paths if path.endswith(require_extension)]
+                    paths = [
+                        path for path in paths if path.endswith(parquet_file_extension)
+                    ]
                     if len0 and paths == []:
                         raise ValueError(
-                            "No files satisfy the `require_extension` criteria "
-                            f"(files must end with {require_extension})."
+                            "No files satisfy the `parquet_file_extension` criteria "
+                            f"(files must end with {parquet_file_extension})."
                         )
                 pf = ParquetFile(
                     paths[:1], open_with=fs.open, root=base, **dataset_kwargs
@@ -855,6 +855,7 @@ def read_metadata(
         aggregate_files=None,
         ignore_metadata_file=False,
         metadata_task_size=None,
+        parquet_file_extension=None,
         **kwargs,
     ):
 
@@ -871,6 +872,7 @@ def read_metadata(
             aggregate_files,
             ignore_metadata_file,
             metadata_task_size,
+            parquet_file_extension,
             kwargs,
         )
 


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -3665,23 +3665,39 @@ def test_extra_file(tmpdir, engine):
     # expected file extension, or avoid checking file extensions
     # by passing False. Check here that this works:
 
+    def _parquet_file_extension(val, legacy=False):
+        # This function allows us to switch between "new"
+        # and "legacy" parquet_file_extension behavior
+        return (
+            {"dataset": {"require_extension": val}}
+            if legacy
+            else {"parquet_file_extension": val}
+        )
+
     # Should Work
-    out = dd.read_parquet(
-        tmpdir, engine=engine, dataset={"require_extension": ".parquet"}
-    )
+    out = dd.read_parquet(tmpdir, engine=engine, **_parquet_file_extension(".parquet"))
     assert_eq(out, df)
 
+    # Should Work (with FutureWarning)
+    with pytest.warns(FutureWarning, match="require_extension is deprecated"):
+        out = dd.read_parquet(
+            tmpdir,
+            engine=engine,
+            **_parquet_file_extension(".parquet", legacy=True),
+        )
+        assert_eq(out, df)
+
     # Should Fail (for not capturing the _SUCCESS and crc files)
     with pytest.raises((OSError, pa.lib.ArrowInvalid)):
         dd.read_parquet(
-            tmpdir, engine=engine, dataset={"require_extension": False}
+            tmpdir, engine=engine, **_parquet_file_extension(None)
         ).compute()
 
     # Should Fail (for filtering out all files)
     # (Related to: https://github.com/dask/dask/issues/8349)
     with pytest.raises(ValueError):
         dd.read_parquet(
-            tmpdir, engine=engine, dataset={"require_extension": ".foo"}
+            tmpdir, engine=engine, **_parquet_file_extension(".foo")
         ).compute()