Repository: dask/dask

Issue:
Inconsistency and errors between different parquet engines on Spark-generated parquet files.
Creation of two partitioned parquet files using Spark:
```python
import pandas as pd

data = [
    [0, 1, 2],
    [0, 2, 3]
]

(
    spark
    .createDataFrame(pd.DataFrame(data, columns = ['col1', 'col2', 'col3']))
    .write.mode('overwrite').partitionBy('col1', 'col2')
    .parquet('test.parquet')
)

data = [
    [0, 1, 2],
    [0, 1, 3]
]

(
    spark
    .createDataFrame(pd.DataFrame(data, columns = ['col1', 'col2', 'col3']))
    .write.mode('overwrite').partitionBy('col1', 'col2')
    .parquet('test2.parquet')
)
```

Content of the parquet files:
```
test.parquet
├── col1=0
│   ├── col2=1
│   │   └── part-00000-0787dc35-33b6-46a8-8fc9-a88b443877d3.c000.snappy.parquet
│   └── col2=2
│       └── part-00001-0787dc35-33b6-46a8-8fc9-a88b443877d3.c000.snappy.parquet
└── _SUCCESS
test2.parquet
├── col1=0
│   └── col2=1
│       ├── part-00000-c021737d-18be-4dde-a721-842afe197a42.c000.snappy.parquet
│       └── part-00001-c021737d-18be-4dde-a721-842afe197a42.c000.snappy.parquet
└── _SUCCESS
```


#### Experiment 1 (`pyarrow-legacy` engine)
```python
>>> ddf = dd.read_parquet('test.parquet', engine='pyarrow-legacy') 
>>> print(ddf)
Dask DataFrame Structure:
                col3             col1             col2
npartitions=2                                         
               int64  category[known]  category[known]
                 ...              ...              ...
                 ...              ...              ...
Dask Name: read-parquet, 2 tasks
>>> print(ddf.head(npartitions=-1))
   col3 col1 col2
0     2    0    1
0     3    0    2

>>> ddf2 = dd.read_parquet('test2.parquet', engine='pyarrow-legacy')
>>> print(ddf2.head(npartitions=-1))
   col3 col1 col2
0     2    0    1
0     3    0    1
```

All works as expected.

#### Experiment 2 (`pyarrow` engine)
```python
>>> ddf = dd.read_parquet('test.parquet', engine='pyarrow')  ## the same result with pyarrow-dataset engine
>>> print(ddf)
Dask DataFrame Structure:
                col3             col1             col2
npartitions=2                                         
               int64  category[known]  category[known]
                 ...              ...              ...
                 ...              ...              ...
Dask Name: read-parquet, 2 tasks
>>> print(ddf.head(npartitions=-1))
Exception: KeyError("['col1', 'col2'] not in index")
```
So, something goes wrong with hierarchy-inferred columns. The same exception with with `test2.parquet`.


#### Experiment 3 (`fastparquet` engine)
```python
>>> dd.read_parquet('test.parquet', engine='fastparquet')
Exception: OSError: [Errno 22] Invalid argument
```
Reason : `fastparquet` engine fails while trying to parse empty `_SUCCESS` file as parquet file

Ok, removing file (`rm test.parquet/_SUCCESS`) and start again:

```python
>>> ddf = dd.read_parquet('test.parquet', engine='fastparquet')
>>> print(ddf)
Dask DataFrame Structure:
                col3             col2
npartitions=2                        
               int64  category[known]
                 ...              ...
                 ...              ...
Dask Name: read-parquet, 2 tasks
>>> print(ddf.head(npartitions=-1))
   col3 col2
0     2    0
0     3    0

>>> ddf2 = dd.read_parquet('test2.parquet', engine='fastparquet')
>>> print(ddf2)
Dask DataFrame Structure:
                col3
npartitions=2       
               int64
                 ...
                 ...
Dask Name: read-parquet, 2 tasks
>>> print(ddf.head(npartitions=-1))
   col3
0     2
0     3
```

Note two big problems:
  1) If _hierarchy inferred_ columns have only one unique value, then that columns simply dropped.
  2) If _hierarchy inferred_ columns have more than one unique value, then all that values turns to zeros.

<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**Environment**:

- Dask version: 2021.8.1
- Python version: 3.7
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): conda
- Spark: 2.4.8



Discussion:

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -752,6 +752,7 @@ def _collect_dataset_info(
         aggregate_files,
         ignore_metadata_file,
         metadata_task_size,
+        require_extension=(".parq", ".parquet"),
         **dataset_kwargs,
     ):
         """pyarrow.dataset version of _collect_dataset_info
@@ -804,6 +805,11 @@ def _collect_dataset_info(
                 has_metadata_file = True
                 if gather_statistics is None:
                     gather_statistics = True
+            elif require_extension:
+                # Need to materialize all paths if we are missing the _metadata file
+                paths = [
+                    path for path in fs.find(paths) if path.endswith(require_extension)
+                ]
 
         elif len(paths) > 1:
             paths, base, fns = _sort_and_analyze_paths(paths, fs)
@@ -827,7 +833,13 @@ def _collect_dataset_info(
                 # Populate valid_paths, since the original path list
                 # must be used to filter the _metadata-based dataset
                 fns.remove("_metadata")
-                valid_paths = fns
+                valid_paths = (
+                    [fn for fn in fns if fn.endswith(require_extension)]
+                    if require_extension
+                    else fns
+                )
+            elif require_extension:
+                paths = [path for path in paths if path.endswith(require_extension)]
 
         # Final "catch-all" pyarrow.dataset call
         if ds is None:
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -368,6 +368,7 @@ def _collect_dataset_info(
         aggregate_files,
         ignore_metadata_file,
         metadata_task_size,
+        require_extension=(".parq", ".parquet"),
         **kwargs,
     ):
 
@@ -407,6 +408,8 @@ def _collect_dataset_info(
                     paths = [fs.sep.join([base, fn]) for fn in fns]
                 _metadata_exists = False
 
+            if require_extension:
+                paths = [path for path in paths if path.endswith(require_extension)]
             if _metadata_exists:
                 # Using _metadata file (best-case scenario)
                 pf = ParquetFile(
@@ -435,10 +438,12 @@ def _collect_dataset_info(
             _metadata_exists = "_metadata" in fns
             if _metadata_exists and ignore_metadata_file:
                 fns.remove("_metadata")
-                paths = [fs.sep.join([base, fn]) for fn in fns]
                 _metadata_exists = False
+            if require_extension:
+                fns = [fn for fn in fns if fn.endswith(require_extension)]
+            paths = [fs.sep.join([base, fn]) for fn in fns]
 
-            elif _metadata_exists:
+            if _metadata_exists:
                 # We have a _metadata file, lets use it
                 pf = ParquetFile(
                     fs.sep.join([base, "_metadata"]),
@@ -1063,7 +1068,7 @@ def initialize_write(
             try:
                 # to append to a dataset without _metadata, need to load
                 # _common_metadata or any data file here
-                pf = fastparquet.api.ParquetFile(path, open_with=fs.open, sep=fs.sep)
+                pf = fastparquet.api.ParquetFile(path, open_with=fs.open)
             except (OSError, ValueError):
                 # append for create
                 append = False


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -3719,3 +3719,34 @@ def test_metadata_task_size(tmpdir, engine, write_metadata_file, metadata_task_s
                 gather_statistics=True,
                 metadata_task_size=metadata_task_size,
             )
+
+
+def test_extra_file(tmpdir, engine):
+    # Check that read_parquet can handle spark output
+    # See: https://github.com/dask/dask/issues/8087
+    tmpdir = str(tmpdir)
+    df = pd.DataFrame({"a": range(100), "b": ["dog", "cat"] * 50})
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf.to_parquet(tmpdir, engine=engine)
+    open(os.path.join(tmpdir, "_SUCCESS"), "w").close()
+    open(os.path.join(tmpdir, "part.0.parquet.crc"), "w").close()
+    os.remove(os.path.join(tmpdir, "_metadata"))
+    out = dd.read_parquet(tmpdir, engine=engine)
+    assert_eq(out, df)
+
+    if engine != "pyarrow-legacy":
+        # For "fastparquet" and "pyarrow-dataset", we can pass the
+        # expected file extension, or avoid checking file extensions
+        # by passing False. Check here that this works:
+
+        # Should Work
+        out = dd.read_parquet(
+            tmpdir, engine=engine, dataset={"require_extension": ".parquet"}
+        )
+        assert_eq(out, df)
+
+        # Should Fail
+        with pytest.raises((OSError, pa.lib.ArrowInvalid)):
+            dd.read_parquet(
+                tmpdir, engine=engine, dataset={"require_extension": False}
+            ).compute()