Repository: dask/dask

Issue:
Change `to_parquet` default to `write_metadata_file=False`
*Some* parquet tools and libraries make use of a `_metadata` file. This file aggregates the parquet metadata from all files in the dataset, making it more efficient to access this metadata.

Doing some software history spelunking:

- This convention appears (without comment) in the `apache/parquet-mr` repo [~9 years ago](https://github.com/apache/parquet-mr/commit/12b99b1a78478999ef8da2c5617d65a5b0e687ac#diff-9f06fe5118ecba86a35b562b7ef0f5e8ee56f947668d27d367d4bc58fbe6b549R58).
- It's then deprecated (equally without comment) [~4 years ago](https://github.com/apache/parquet-mr/commit/8bfd9b4d8f4fb0a2b522c9328f67eb642066306b#diff-7f85b7a9ddab623b931615aee3ebbcc469e41c5bae3fd53c52fe79a85c3af6e0R752) as "not recommended"
- Spark (the main historical producer of such files) stopped writing these files [~6 years ago](https://issues.apache.org/jira/browse/SPARK-15719) due to scalability issues

The issue is that creation of a `_metadata` file requires collecting the metadata from all parquet files in a written dataset, merging this metadata, and then writing it out. The size of this metadata scales with `total_n_row_groups * n_columns` in the dataset - for large datasets (for varying definitions of "large") the size of this metadata can swamp a worker leading to execution failure. A rudimentary benchmark shows creating this file using `pyarrow` takes roughly 1 KiB per row group per column (this seems very high, but even if it was lower we'd still eventually hit a limit). There are several issues in the dask issue tracker documenting this:

- https://github.com/dask/dask/issues/4410
- https://github.com/dask/dask/issues/6600
- https://github.com/dask/dask/issues/3873

Note that this can also cause issues in the `read_parquet` side if the `_metadata` file is large, though the scaling is a bit different here.

Currently creation of the `_metadata` file can be disabled by passing `write_metadata_file=False` to `to_parquet`, which is the recommendation we give in #8899. I'd prefer not to have to mention this caveat though, and have our defaults match our recommendations. Given that other tools have stopped creating or using this file I'd argue that we should stop as well.

I'm not sure if there's a good way to smoothly deprecate this behavior. The best option I've come up with is warning if `write_metadata_file` isn't set explicitly, but that may be overly noisy for users who won't care about this change. Not sure.


Discussion:
cc @martindurant, @rjzamora, @ian-r-rose, @bryanwweber 
So two issues, one at a time:
- do we in fact want to change the default?
- if yes, how to implement this in a way that is obvious but not too annoying to users.

Note that we do allow an option to ignore the file on read, even if it is present, for better or worse. Also, we are able to create such a file *after* a successful write of all of the data files, but I doubt this happens often. Finally, we allow to aggregate an effective _metadata file from data fles when it is not present, for the purposes of allowing row-group filtering, and this can be done either by the client or dask workers.
Yes, that's an accurate summary. Do you have thoughts on these two questions?
It is not straight forward. A _metadata file can be a significant read-time optimisation:
- for opening a dataset (single read can be faster than even just a file listing)
- for avoiding unnecessary reads via filter pushdown
and is not a significant cost for 1-100 files.

Aside: it is unfortunate that people store ML models in parquet, using a tabular structure for what is an array; they should use something like zarr instead.

At read, *if* there is such a file, we can ignore it, and if *not*, we can scan all the datafiles anyway; so the default we choose needs to be considered in conjunction with these other defaults.

Since it is the number of row-groups * number of columns which matters, we can make a sensible default of writing only when we expect the metadata to be small enough. Unfortunately, there's no way to know at write  time whether the stats information might become useful for filtering purposes. Also, if there is directory partitioning, is becomes much less likely that the global statistics are useful. Note that, as least with fastparquet, the user can opt not to create statistics on write, which saves a little time and not insignificant space in the metadata - and this can be done per-column, so you might only do it for columns you intend to filter on. More options, more complications.

So I think I advocate write_metedata_file="auto" (or suitable) name, where we do write if nrowgroup*ncols < 100,000, or configurable value.

On read, we can similarly prefer to ignore the metadata file if it exists and it's size is above some threshold, for the case that the user doesn't explicitly want to get statistics.
I forgot to mention another side-issue: you would not recommend a _metadata file for datasets intended for multiple append; but at append time, you need to respect one if it's already there (removing it or leaving it inconsistent seem bad, and adding row groups is relatively cheap).
We've talked in person since the above comments, but rehashing for posterity:

> for opening a dataset (single read can be faster than even just a file listing)

Agreed, but in my experience the cost of a file listing is usually not that expensive. It might be slower, but not by much.

> for avoiding unnecessary reads via filter pushdown and is not a significant cost for 1-100 files.

Since most dask-dataframe operations scale linearly with the number of partitions (as opposed to dask-array, where quadratic+ isn't uncommon), I think I'm fine with creating partitions that will ultimately be empty at runtime. We can still filter at runtime to load less data from disk using the underlying parquet engine, the same as we do for columns. If users really need to reduce the number loaded files, they might make use of directory partitioning (which can be done efficiently) instead of relying on a `_metadata` file to allow row-group filtering.

> So I think I advocate write_metedata_file="auto" (or suitable) name, where we do write if nrowgroup*ncols < 100,000, or configurable value.

I'm not enthused about systems where the generated artifact changes form based on some characteristic of the data (in this case its size). Since the behavior of reading with a `_metadata` file is different than that without, having a magic switch would likely confuse users (they'd see one behavior on small data and a different behavior on large data).

I'm advocating that we should drop writing the `_metadata` file *by default*. Users can still ask to explicitly write it if they want, but it won't be the default.

*Aside: since other systems have moved to more complicated (but also more scalable) metadata files, we may also want to make it simpler for users to bring their own metadata (Rick's mentioned some related ideas in https://github.com/dask/dask/issues/8937#issuecomment-1101609249 and https://github.com/dask/dask/pull/8944). Users can already specify a list of file paths, making it easier to also include divisions/metadata/... might be an interesting problem.*

Since this change will be tricky to deprecate without annoying most users, I'll make the following proposal:

**Change the default to `write_metadata_file=None`. If `None`, a metadata file is only written when appending to an existing dataset that has a metadata file. Otherwise no metadata file is written. No deprecation period or warnings added, we'll just be sure to highlight this change in the release notes.**

I *think* this is unlikely to cause downstream issues, but it might. The other option is to warn unless `write_metadata_file` is explicitly set, which would definitely alert users but would also mean all usage would have to set this parameter for a while which feels unfriendly.

One issue with this proposal is that afaict the pyarrow engine only supports appending to datasets that currently have a `_metadata` file. The fastparquet engine doesn't appear to have this restriction (based on https://github.com/dask/dask/blob/76770db4285db43d9056d4fa9ce5b8e8006de7d1/dask/dataframe/io/parquet/fastparquet.py#L1165-L1171, cc @martindurant for clarification on the expected behavior here). This means that writing and appending to a dataset with the pyarrow engine would then break without explicitly passing in `write_metadata_file=True`, which we don't want. So this change would also require modifying the existing engines so `append` would still work if no metadata file is present.


> the cost of a file listing is usually not that expensive.

On a remote datastore with >=1000 files, it is indeed slow, because the listing must be paged. If we know something about the distribution of paths beforehand, we can run listing concurrently, in theory, but paging will be serial as things stand.

> for clarification on the expected behavior here

This simply ensures that append works even if the original location doesn't yet contain any data. 

> ould also require modifying the existing engines so append would still work if no metadata file is present

That should be the easier case - you just need to know the file numbering to go with to avoid any overwrites. However, we would need to decide what to do if a _metadata file does exist: do we still use it, remove it, error, or ...?
>I'm not enthused about systems where the generated artifact changes form based on some characteristic of the data (in this case its size). Since the behavior of reading with a _metadata file is different than that without, having a magic switch would likely confuse users (they'd see one behavior on small data and a different behavior on large data).

This is a conclusion that I have also recently reached. I think it makes a lot of sense to use "magic" switches to select internal algorithm defaults, but these switches should never affect the structure or content of the result.  In other words, I think its fine to use file count and/or _metadata presence to decide if we will do things like process metadata in parallel. However, these characteristics should not control partition sizes and/or metadata output. (This is obviously just my personal opinion, but I'm starting to feel more strongly about it by the day)
>However, we would need to decide what to do if a _metadata file does exist: do we still use it, remove it, error, or ...?

This is a good question. It certainly feels like we would definetly want to update the `_metadata` file on append if one is present. So, this would probably be the exception to the `write_metadata_file=False`-default rule. It **may** make sense to add an `ignore_metadata_file=False` to `to_parquet` in the off-chance that the user really does want to append to a dataset without updating the global metadata, but I have a hard time imagining a case where this isn't a terrible idea. So, throwing a clear error is probably the "correct" thing to do.
> It certainly feels like we would definetly want to update the _metadata file on append if one is present. 

Agreed, this is what I suggested above:

> **Change the default to write_metadata_file=None. If None, a metadata file is only written when appending to an existing dataset that has a metadata file. Otherwise no metadata file is written. No deprecation period or warnings added, we'll just be sure to highlight this change in the release notes.**

The issue is that appending behavior doesn't currently work without a `_metadata` file, and there are some inconsistencies between backends with regards to what happens. Assorted issues:

- `pyarrow` doesn't work with appending without a `_metadata` file. An error is raised suggesting setting `ignore_divisions=True` should make things work, but it doesn't
- `fastparquet` only works without a `_metadata` file if `ignore_divisions=True`, even if `df.known_divisions` is `False`.

I can fix these, but I'm curious about what the expected behavior is here.

- If `df` has `known_divisions` and no `_metadata` file, `fastparquet` will load the last file and attempt to see if the divisions are still sorted (and if not will error). Is this a restriction we want to keep?
- If `known_divisions` is `False` for `df`, should we skip this check (basically treat this as `ignore_divisions=True`). Is there a good way to tell from the last file if previous writes had known divisions even if the new data doesn't?
More broadly, what kind of changes to the backing engine classes are acceptable, given implementations live outside of dask. Can I change the signature of a backend method? Only add new methods (and don't call them if they're missing)? What's the best way to not break cudf here?
>Change the default to write_metadata_file=None. If None, a metadata file is only written when appending to an existing dataset that has a metadata file. Otherwise no metadata file is written. No deprecation period or warnings added, we'll just be sure to highlight this change in the release notes.

I completely agree with your proposal @jcrist

Also, it is good to know that there are existing bugs related to appending. Thanks for summarizing!

I don’t have any strong feelings on *exactly* how to resolve existing bugs, but my intuition is that we should probably assume the user doesn’t want to calculate/preserve divisions unless they explicitly ask for this extra work (note that one of my proposals below is to do this in `read_parquet` anyway, and I would assume reading and appending should have consistent defaults). 

>More broadly, what kind of changes to the backing engine classes are acceptable, given implementations live outside of dask. Can I change the signature of a backend method? Only add new methods (and don't call them if they're missing)? What's the best way to not break cudf here?

  I wouldn’t be too nervous about “breaking” dask_cudf on the write side. If a test breaks as a result of different default behavior, I can certainly find and “fix” it before the related Dask PR is even merged.  Regarding Engine-API changes: dask_cudf is just doing `to_parquet = partial(dd.to_parquet, engine=CudfEngine)`, and `CudfEngine` is just inheriting from `ArrowDatasetEngine` (and overriding the `write_partition` and `write_metadata` class methods). `CudfEngine` does not override `initialize_write` (where most appending logic currently lives), so fixing append bugs is probably pretty "safe."

Just to clarify some of the Parquet-related changes I am personally supportive of:

1. [Write] Change `write_metadata_file` default to `None` (becomes `False`, unless appending)

2. [Read] Change `split_row_groups` default to `False` (starts as `None` in the top-level signature so we can temporarily “warn” users when we hit the “old” code path where `split_row_groups` was originally switching to `True`). We would then submite a follow-up PR in ~2+ months to remove the warning (and set the `False` default in top-level function signature)

3.  [Read, but may relate to Write-with-append] Add `calculate_divisions` argument, and deprecate `gather_statistics`. The default for `calculate_divisions` will be `False`, but we will start with `None` in the top-level function signature (for now) so we can warn the user if they do not set anythin explicitly and there is a `_metadata` file present (because they will no longer get known divisions).  We could then submit a follow-up PR in ~2+ months to remove `gather_statistics` altogether and remove the warning (and set the `False` default in top-level function signature)

4. [Read][Explicit Plan TBD] Modify (and maybe deprecate?) `chunksize` and/or `aggregate_files`. I would like to use `from_dataset` to explore “good” defaults/options for this.

5. [Read][Explicit Plan TBD] Modify `categories` to include directory-partitioned columns. Converting a partitioned column to a pandas-category column has overhead, and not all users want/need a categorical dtype (e.g. Dask-sql). My opinion is that the user should need to “opt-in” to this extra work by including directory-partitioned columns in `categories` (right now this is actually not allowed).

^ would you mind annotating those 5 points for whether they affect reading or writing? We have so many options, I am not certain any more!

> fastparquet only works without a _metadata file if ignore_divisions=True, even if df.known_divisions is False.

I suppose an attempt to ensure that if there was a reliable (ordered) index, we don't break it. Not sure that's documented or tested, and I don't remember writing it. fastparquet itself doesn't care about this, it must be in dask's fp engine.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -493,35 +493,6 @@ def read_partition(
             df = df.set_index(index)
         return df
 
-    @classmethod
-    def _get_dataset_offset(cls, path, fs, append, ignore_divisions):
-        fmd = None
-        i_offset = 0
-        if append:
-            # Make sure there are existing file fragments.
-            # Otherwise there is no need to set `append=True`
-            i_offset = len(
-                list(
-                    pa_ds.dataset(path, filesystem=fs, format="parquet").get_fragments()
-                )
-            )
-            if i_offset == 0:
-                # No dataset to append to
-                return fmd, i_offset, False
-            try:
-                with fs.open(fs.sep.join([path, "_metadata"]), mode="rb") as fil:
-                    fmd = pq.read_metadata(fil)
-            except OSError:
-                # No _metadata file present - No appending allowed (for now)
-                if not ignore_divisions:
-                    # TODO: Be more flexible about existing metadata.
-                    raise NotImplementedError(
-                        "_metadata file needed to `append` "
-                        "with `engine='pyarrow-dataset'` "
-                        "unless `ignore_divisions` is `True`"
-                    )
-        return fmd, i_offset, append
-
     @classmethod
     def initialize_write(
         cls,
@@ -582,14 +553,34 @@ def initialize_write(
         if append and division_info is None:
             ignore_divisions = True
 
-        # Extract metadata and get file offset if appending
-        fmd, i_offset, append = cls._get_dataset_offset(
-            path, fs, append, ignore_divisions
-        )
-
-        # Inspect the intial metadata if appending
+        full_metadata = None  # metadata for the full dataset, from _metadata
+        tail_metadata = None  # metadata for at least the last file in the dataset
+        i_offset = 0
+        metadata_file_exists = False
         if append:
-            arrow_schema = fmd.schema.to_arrow_schema()
+            # Extract metadata and get file offset if appending
+            ds = pa_ds.dataset(path, filesystem=fs, format="parquet")
+            i_offset = len(ds.files)
+            if i_offset > 0:
+                try:
+                    with fs.open(fs.sep.join([path, "_metadata"]), mode="rb") as fil:
+                        full_metadata = pq.read_metadata(fil)
+                    tail_metadata = full_metadata
+                    metadata_file_exists = True
+                except OSError:
+                    try:
+                        with fs.open(
+                            sorted(ds.files, key=natural_sort_key)[-1], mode="rb"
+                        ) as fil:
+                            tail_metadata = pq.read_metadata(fil)
+                    except OSError:
+                        pass
+            else:
+                append = False  # No existing files, can skip the append logic
+
+        # If appending, validate against the initial metadata file (if present)
+        if append and tail_metadata is not None:
+            arrow_schema = tail_metadata.schema.to_arrow_schema()
             names = arrow_schema.names
             has_pandas_metadata = (
                 arrow_schema.metadata is not None and b"pandas" in arrow_schema.metadata
@@ -624,7 +615,10 @@ def initialize_write(
                 ignore_divisions = True
             if not ignore_divisions:
                 old_end = None
-                row_groups = [fmd.row_group(i) for i in range(fmd.num_row_groups)]
+                row_groups = (
+                    tail_metadata.row_group(i)
+                    for i in range(tail_metadata.num_row_groups)
+                )
                 for row_group in row_groups:
                     for i, name in enumerate(names):
                         if name != division_info["name"]:
@@ -645,7 +639,8 @@ def initialize_write(
                         "Previous: {} | New: {}".format(old_end, divisions[0])
                     )
 
-        return fmd, schema, i_offset
+        extra_write_kwargs = {"schema": schema, "index_cols": index_cols}
+        return i_offset, full_metadata, metadata_file_exists, extra_write_kwargs
 
     @classmethod
     def _pandas_to_arrow_table(
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -561,7 +561,7 @@ def to_parquet(
     partition_on=None,
     storage_options=None,
     custom_metadata=None,
-    write_metadata_file=True,
+    write_metadata_file=None,
     compute=True,
     compute_kwargs=None,
     schema=None,
@@ -616,8 +616,10 @@ def to_parquet(
         Custom key/value metadata to include in all footer metadata (and
         in the global "_metadata" file, if applicable).  Note that the custom
         metadata may not contain the reserved b"pandas" key.
-    write_metadata_file : bool, default True
-        Whether to write the special "_metadata" file.
+    write_metadata_file : bool or None, default None
+        Whether to write the special ``_metadata`` file. If ``None`` (the
+        default), a ``_metadata`` file will only be written if ``append=True``
+        and the dataset already has a ``_metadata`` file.
     compute : bool, default True
         If :obj:`True` (default) then the result is computed immediately. If  :obj:`False`
         then a ``dask.dataframe.Scalar`` object is returned for future computation.
@@ -721,6 +723,10 @@ def to_parquet(
             # (3) The path is not the current working directory
             fs.rm(path, recursive=True)
 
+    # Always skip divisions checks if divisions are unknown
+    if not df.known_divisions:
+        ignore_divisions = True
+
     # Save divisions and corresponding index name. This is necessary,
     # because we may be resetting the index to write the file
     division_info = {"divisions": df.divisions, "name": df.index.name}
@@ -770,31 +776,17 @@ def to_parquet(
         # Not writing index - might as well drop it
         df = df.reset_index(drop=True)
 
-    _to_parquet_kwargs = {
-        "engine",
-        "compression",
-        "write_index",
-        "append",
-        "ignore_divisions",
-        "partition_on",
-        "storage_options",
-        "write_metadata_file",
-        "compute",
-    }
-    kwargs_pass = {k: v for k, v in kwargs.items() if k not in _to_parquet_kwargs}
+    if custom_metadata and b"pandas" in custom_metadata.keys():
+        raise ValueError(
+            "User-defined key/value metadata (custom_metadata) can not "
+            "contain a b'pandas' key.  This key is reserved by Pandas, "
+            "and overwriting the corresponding value can render the "
+            "entire dataset unreadable."
+        )
 
     # Engine-specific initialization steps to write the dataset.
     # Possibly create parquet metadata, and load existing stuff if appending
-    if custom_metadata:
-        if b"pandas" in custom_metadata.keys():
-            raise ValueError(
-                "User-defined key/value metadata (custom_metadata) can not "
-                "contain a b'pandas' key.  This key is reserved by Pandas, "
-                "and overwriting the corresponding value can render the "
-                "entire dataset unreadable."
-            )
-        kwargs_pass["custom_metadata"] = custom_metadata
-    meta, schema, i_offset = engine.initialize_write(
+    i_offset, fmd, metadata_file_exists, extra_write_kwargs = engine.initialize_write(
         df,
         fs,
         path,
@@ -804,9 +796,15 @@ def to_parquet(
         division_info=division_info,
         index_cols=index_cols,
         schema=schema,
-        **kwargs_pass,
+        custom_metadata=custom_metadata,
+        **kwargs,
     )
 
+    # By default we only write a metadata file when appending if one already
+    # exists
+    if append and write_metadata_file is None:
+        write_metadata_file = metadata_file_exists
+
     # Check that custom name_function is valid,
     # and that it will produce unique names
     if name_function is not None:
@@ -817,10 +815,6 @@ def to_parquet(
             raise ValueError("``name_function`` must produce unique filenames.")
 
     # Create Blockwise layer for parquet-data write
-    kwargs_pass["fmd"] = meta
-    kwargs_pass["compression"] = compression
-    kwargs_pass["index_cols"] = index_cols
-    kwargs_pass["schema"] = schema
     data_write = df.map_partitions(
         ToParquetFunctionWrapper(
             engine,
@@ -830,7 +824,11 @@ def to_parquet(
             write_metadata_file,
             i_offset,
             name_function,
-            kwargs_pass,
+            toolz.merge(
+                kwargs,
+                {"compression": compression, "custom_metadata": custom_metadata},
+                extra_write_kwargs,
+            ),
         ),
         BlockIndex((df.npartitions,)),
         # Pass in the original metadata to avoid
@@ -853,7 +851,7 @@ def to_parquet(
                 engine.write_metadata,
                 [
                     data_write.__dask_keys__(),
-                    meta,
+                    fmd,
                     fs,
                     path,
                 ],
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -1161,11 +1161,13 @@ def initialize_write(
                 "because this required data in memory."
             )
 
+        metadata_file_exists = False
         if append:
             try:
                 # to append to a dataset without _metadata, need to load
                 # _common_metadata or any data file here
                 pf = fastparquet.api.ParquetFile(path, open_with=fs.open)
+                metadata_file_exists = fs.exists(fs.sep.join([path, "_metadata"]))
             except (OSError, ValueError):
                 # append for create
                 append = False
@@ -1224,8 +1226,8 @@ def initialize_write(
             )
             fmd.key_value_metadata = kvm
 
-        schema = None  # ArrowEngine compatibility
-        return (fmd, schema, i_offset)
+        extra_write_kwargs = {"fmd": fmd}
+        return i_offset, fmd, metadata_file_exists, extra_write_kwargs
 
     @classmethod
     def write_partition(


Test patch:
diff --git a/dask/bytes/tests/test_s3.py b/dask/bytes/tests/test_s3.py
--- a/dask/bytes/tests/test_s3.py
+++ b/dask/bytes/tests/test_s3.py
@@ -573,6 +573,7 @@ def test_parquet_append(s3, engine, s3so):
         engine=engine,
         storage_options=s3so,
         write_index=False,
+        write_metadata_file=True,
     )
     df.to_parquet(
         url,
@@ -613,7 +614,13 @@ def test_parquet_wstoragepars(s3, s3so, engine):
 
     data = pd.DataFrame({"i32": np.array([0, 5, 2, 5])})
     df = dd.from_pandas(data, chunksize=500)
-    df.to_parquet(url, engine=engine, write_index=False, storage_options=s3so)
+    df.to_parquet(
+        url,
+        engine=engine,
+        write_index=False,
+        storage_options=s3so,
+        write_metadata_file=True,
+    )
 
     dd.read_parquet(
         url,
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -173,7 +173,8 @@ def test_get_engine_auto_warning_if_both_installed():
 
 
 @write_read_engines()
-def test_local(tmpdir, write_engine, read_engine):
+@pytest.mark.parametrize("has_metadata", [False, True])
+def test_local(tmpdir, write_engine, read_engine, has_metadata):
     tmp = str(tmpdir)
     data = pd.DataFrame(
         {
@@ -187,11 +188,12 @@ def test_local(tmpdir, write_engine, read_engine):
     )
     df = dd.from_pandas(data, chunksize=500)
 
-    df.to_parquet(tmp, write_index=False, engine=write_engine)
+    kwargs = {"write_metadata_file": True} if has_metadata else {}
+    df.to_parquet(tmp, write_index=False, engine=write_engine, **kwargs)
 
     files = os.listdir(tmp)
-    assert "_common_metadata" in files
-    assert "_metadata" in files
+    assert ("_common_metadata" in files) == has_metadata
+    assert ("_metadata" in files) == has_metadata
     assert "part.0.parquet" in files
 
     df2 = dd.read_parquet(tmp, index=False, engine=read_engine)
@@ -213,7 +215,7 @@ def test_empty(tmpdir, write_engine, read_engine, index):
         df.set_index("a", inplace=True, drop=True)
     ddf = dd.from_pandas(df, npartitions=2)
 
-    ddf.to_parquet(fn, write_index=index, engine=write_engine)
+    ddf.to_parquet(fn, write_index=index, engine=write_engine, write_metadata_file=True)
     read_df = dd.read_parquet(fn, engine=read_engine)
     assert_eq(ddf, read_df)
 
@@ -228,8 +230,9 @@ def test_simple(tmpdir, write_engine, read_engine):
     df.set_index("a", inplace=True, drop=True)
     ddf = dd.from_pandas(df, npartitions=2)
     ddf.to_parquet(fn, engine=write_engine)
-    read_df = dd.read_parquet(fn, index=["a"], engine=read_engine)
-    assert_eq(ddf, read_df)
+    res = dd.read_parquet(fn, index=["a"], engine=read_engine)
+    sol = ddf.compute()
+    assert_eq(res, sol)
 
 
 @write_read_engines()
@@ -314,10 +317,12 @@ def test_columns_auto_index(tmpdir, write_engine, read_engine):
     fn = str(tmpdir)
     ddf.to_parquet(fn, engine=write_engine)
 
-    # XFAIL, auto index selection no longer supported (for simplicity)
     # ### Empty columns ###
     # With divisions if supported
-    assert_eq(dd.read_parquet(fn, columns=[], engine=read_engine), ddf[[]])
+    assert_eq(
+        dd.read_parquet(fn, columns=[], engine=read_engine, gather_statistics=True),
+        ddf[[]],
+    )
 
     # No divisions
     assert_eq(
@@ -328,7 +333,10 @@ def test_columns_auto_index(tmpdir, write_engine, read_engine):
 
     # ### Single column, auto select index ###
     # With divisions if supported
-    assert_eq(dd.read_parquet(fn, columns=["x"], engine=read_engine), ddf[["x"]])
+    assert_eq(
+        dd.read_parquet(fn, columns=["x"], engine=read_engine, gather_statistics=True),
+        ddf[["x"]],
+    )
 
     # No divisions
     assert_eq(
@@ -348,7 +356,10 @@ def test_columns_index(tmpdir, write_engine, read_engine):
     # ### Empty columns, specify index ###
     # With divisions if supported
     assert_eq(
-        dd.read_parquet(fn, columns=[], engine=read_engine, index="myindex"), ddf[[]]
+        dd.read_parquet(
+            fn, columns=[], engine=read_engine, index="myindex", gather_statistics=True
+        ),
+        ddf[[]],
     )
 
     # No divisions
@@ -363,7 +374,13 @@ def test_columns_index(tmpdir, write_engine, read_engine):
     # ### Single column, specify index ###
     # With divisions if supported
     assert_eq(
-        dd.read_parquet(fn, index="myindex", columns=["x"], engine=read_engine),
+        dd.read_parquet(
+            fn,
+            index="myindex",
+            columns=["x"],
+            engine=read_engine,
+            gather_statistics=True,
+        ),
         ddf[["x"]],
     )
 
@@ -383,7 +400,13 @@ def test_columns_index(tmpdir, write_engine, read_engine):
     # ### Two columns, specify index ###
     # With divisions if supported
     assert_eq(
-        dd.read_parquet(fn, index="myindex", columns=["x", "y"], engine=read_engine),
+        dd.read_parquet(
+            fn,
+            index="myindex",
+            columns=["x", "y"],
+            engine=read_engine,
+            gather_statistics=True,
+        ),
         ddf,
     )
 
@@ -540,11 +563,13 @@ def test_no_index(tmpdir, write_engine, read_engine):
 def test_read_series(tmpdir, engine):
     fn = str(tmpdir)
     ddf.to_parquet(fn, engine=engine)
+    sol = ddf.compute()
+
     ddf2 = dd.read_parquet(fn, columns=["x"], index="myindex", engine=engine)
-    assert_eq(ddf[["x"]], ddf2)
+    assert_eq(sol[["x"]], ddf2)
 
     ddf2 = dd.read_parquet(fn, columns="x", index="myindex", engine=engine)
-    assert_eq(ddf.x, ddf2)
+    assert_eq(sol.x, ddf2)
 
 
 def test_names(tmpdir, engine):
@@ -601,7 +626,8 @@ def test_categorical(tmpdir, write_engine, read_engine):
     assert (df.x == ddf2.x.compute()).all()
 
 
-def test_append(tmpdir, engine):
+@pytest.mark.parametrize("metadata_file", [False, True])
+def test_append(tmpdir, engine, metadata_file):
     """Test that appended parquet equal to the original one."""
     tmp = str(tmpdir)
     df = pd.DataFrame(
@@ -619,8 +645,15 @@ def test_append(tmpdir, engine):
     half = len(df) // 2
     ddf1 = dd.from_pandas(df.iloc[:half], chunksize=100)
     ddf2 = dd.from_pandas(df.iloc[half:], chunksize=100)
-    ddf1.to_parquet(tmp, engine=engine)
+    ddf1.to_parquet(tmp, engine=engine, write_metadata_file=metadata_file)
+    if metadata_file:
+        with open(str(tmpdir.join("_metadata")), "rb") as f:
+            metadata1 = f.read()
     ddf2.to_parquet(tmp, append=True, engine=engine)
+    if metadata_file:
+        with open(str(tmpdir.join("_metadata")), "rb") as f:
+            metadata2 = f.read()
+        assert metadata2 != metadata1  # 2nd write updated the metadata file
 
     ddf3 = dd.read_parquet(tmp, engine=engine)
     assert_eq(df, ddf3)
@@ -759,7 +792,8 @@ def test_partition_on_cats_2(tmpdir, engine):
     assert set(df.cat.categories) == {"x", "y", "z"}
 
 
-def test_append_wo_index(tmpdir, engine):
+@pytest.mark.parametrize("metadata_file", [False, True])
+def test_append_wo_index(tmpdir, engine, metadata_file):
     """Test append with write_index=False."""
     tmp = str(tmpdir.join("tmp1.parquet"))
     df = pd.DataFrame(
@@ -775,21 +809,24 @@ def test_append_wo_index(tmpdir, engine):
     half = len(df) // 2
     ddf1 = dd.from_pandas(df.iloc[:half], chunksize=100)
     ddf2 = dd.from_pandas(df.iloc[half:], chunksize=100)
-    ddf1.to_parquet(tmp, engine=engine)
+    ddf1.to_parquet(tmp, engine=engine, write_metadata_file=metadata_file)
 
     with pytest.raises(ValueError) as excinfo:
         ddf2.to_parquet(tmp, write_index=False, append=True, engine=engine)
     assert "Appended columns" in str(excinfo.value)
 
     tmp = str(tmpdir.join("tmp2.parquet"))
-    ddf1.to_parquet(tmp, write_index=False, engine=engine)
+    ddf1.to_parquet(
+        tmp, write_index=False, engine=engine, write_metadata_file=metadata_file
+    )
     ddf2.to_parquet(tmp, write_index=False, append=True, engine=engine)
 
     ddf3 = dd.read_parquet(tmp, index="f", engine=engine)
     assert_eq(df.set_index("f"), ddf3)
 
 
-def test_append_overlapping_divisions(tmpdir, engine):
+@pytest.mark.parametrize("metadata_file", [False, True])
+def test_append_overlapping_divisions(tmpdir, engine, metadata_file):
     """Test raising of error when divisions overlapping."""
     tmp = str(tmpdir)
     df = pd.DataFrame(
@@ -805,7 +842,7 @@ def test_append_overlapping_divisions(tmpdir, engine):
     half = len(df) // 2
     ddf1 = dd.from_pandas(df.iloc[:half], chunksize=100)
     ddf2 = dd.from_pandas(df.iloc[half - 10 :], chunksize=100)
-    ddf1.to_parquet(tmp, engine=engine)
+    ddf1.to_parquet(tmp, engine=engine, write_metadata_file=metadata_file)
 
     with pytest.raises(ValueError) as excinfo:
         ddf2.to_parquet(tmp, engine=engine, append=True)
@@ -814,7 +851,8 @@ def test_append_overlapping_divisions(tmpdir, engine):
     ddf2.to_parquet(tmp, engine=engine, append=True, ignore_divisions=True)
 
 
-def test_append_different_columns(tmpdir, engine):
+@pytest.mark.parametrize("metadata_file", [False, True])
+def test_append_different_columns(tmpdir, engine, metadata_file):
     """Test raising of error when non equal columns."""
     tmp = str(tmpdir)
     df1 = pd.DataFrame({"i32": np.arange(100, dtype=np.int32)})
@@ -825,7 +863,7 @@ def test_append_different_columns(tmpdir, engine):
     ddf2 = dd.from_pandas(df2, chunksize=2)
     ddf3 = dd.from_pandas(df3, chunksize=2)
 
-    ddf1.to_parquet(tmp, engine=engine)
+    ddf1.to_parquet(tmp, engine=engine, write_metadata_file=metadata_file)
 
     with pytest.raises(ValueError) as excinfo:
         ddf2.to_parquet(tmp, engine=engine, append=True)
@@ -893,7 +931,7 @@ def test_read_parquet_custom_columns(tmpdir, engine):
     df.to_parquet(tmp, engine=engine)
 
     df2 = dd.read_parquet(tmp, columns=["i32", "f"], engine=engine)
-    assert_eq(df[["i32", "f"]], df2, check_index=False)
+    assert_eq(df[["i32", "f"]], df2, check_index=False, check_divisions=False)
 
     fns = glob.glob(os.path.join(tmp, "*.parquet"))
     df2 = dd.read_parquet(fns, columns=["i32"], engine=engine).compute()
@@ -901,7 +939,7 @@ def test_read_parquet_custom_columns(tmpdir, engine):
     assert_eq(df[["i32"]], df2, check_index=False, check_divisions=False)
 
     df3 = dd.read_parquet(tmp, columns=["f", "i32"], engine=engine)
-    assert_eq(df[["f", "i32"]], df3, check_index=False)
+    assert_eq(df[["f", "i32"]], df3, check_index=False, check_divisions=False)
 
 
 @pytest.mark.parametrize(
@@ -974,9 +1012,9 @@ def test_roundtrip(tmpdir, df, write_kwargs, read_kwargs, engine):
     if str(ddf2.dtypes.get("x")) == "UInt16" and engine == "fastparquet":
         # fastparquet choooses to use masked type to be able to get true repr of
         # 16-bit int
-        assert_eq(ddf.astype("UInt16"), ddf2)
+        assert_eq(ddf.astype("UInt16"), ddf2, check_divisions=False)
     else:
-        assert_eq(ddf, ddf2)
+        assert_eq(ddf, ddf2, check_divisions=False)
 
 
 def test_categories(tmpdir, engine):
@@ -1035,7 +1073,6 @@ def test_empty_partition(tmpdir, engine):
     ddf2.to_parquet(fn, engine=engine)
 
     ddf3 = dd.read_parquet(fn, engine=engine)
-    assert ddf3.npartitions < 5
     sol = ddf2.compute()
     assert_eq(sol, ddf3, check_names=False, check_index=False)
 
@@ -1045,7 +1082,7 @@ def test_timestamp_index(tmpdir, engine):
     df = dd._compat.makeTimeDataFrame()
     df.index.name = "foo"
     ddf = dd.from_pandas(df, npartitions=5)
-    ddf.to_parquet(fn, engine=engine)
+    ddf.to_parquet(fn, engine=engine, write_metadata_file=True)
     ddf2 = dd.read_parquet(fn, engine=engine)
     assert_eq(ddf, ddf2)
 
@@ -1183,18 +1220,8 @@ def test_pyarrow_schema_inference(tmpdir, index, engine, schema):
         df = dd.from_pandas(df, npartitions=2)
 
     df.to_parquet(tmpdir, engine="pyarrow", schema=schema)
-    df_out = dd.read_parquet(tmpdir, engine=engine)
-    df_out.compute()
-
-    if index and engine == "fastparquet":
-        # Fastparquet fails to detect int64 from _metadata
-        df_out["amount"] = df_out["amount"].astype("int64")
-
-        # Fastparquet not handling divisions for
-        # pyarrow-written dataset with string index
-        assert_eq(df, df_out, check_divisions=False)
-    else:
-        assert_eq(df, df_out)
+    df_out = dd.read_parquet(tmpdir, engine=engine, gather_statistics=True)
+    assert_eq(df, df_out)
 
 
 def test_partition_on(tmpdir, engine):
@@ -1305,7 +1332,7 @@ def test_filters(tmpdir, write_engine, read_engine):
     ddf = dd.from_pandas(df, npartitions=5)
     assert ddf.npartitions == 5
 
-    ddf.to_parquet(tmp_path, engine=write_engine)
+    ddf.to_parquet(tmp_path, engine=write_engine, write_metadata_file=True)
 
     a = dd.read_parquet(tmp_path, engine=read_engine, filters=[("x", ">", 4)])
     assert a.npartitions == 3
@@ -1409,7 +1436,7 @@ def test_filtering_pyarrow_dataset(tmpdir, engine):
     fn = str(tmpdir)
     df = pd.DataFrame({"aa": range(100), "bb": ["cat", "dog"] * 50})
     ddf = dd.from_pandas(df, npartitions=10)
-    ddf.to_parquet(fn, write_index=False, engine=engine)
+    ddf.to_parquet(fn, write_index=False, engine=engine, write_metadata_file=True)
 
     # Filtered read
     aa_lim = 40
@@ -1581,7 +1608,7 @@ def test_to_parquet_lazy(tmpdir, scheduler, engine):
 
     ddf2 = dd.read_parquet(tmpdir, engine=engine)
 
-    assert_eq(ddf, ddf2)
+    assert_eq(ddf, ddf2, check_divisions=False)
 
 
 @FASTPARQUET_MARK
@@ -1691,7 +1718,9 @@ def test_explicit_compression_default_deprecated(tmpdir, engine):
     ddf = dd.from_pandas(df, npartitions=3)
 
     with pytest.warns(FutureWarning, match="compression='default'"):
-        ddf.to_parquet(fn, compression="default", engine=engine)
+        ddf.to_parquet(
+            fn, compression="default", engine=engine, write_metadata_file=True
+        )
     out = dd.read_parquet(fn, engine=engine)
     assert_eq(out, ddf)
     check_compression(engine, fn, "default")
@@ -1705,7 +1734,7 @@ def test_writing_parquet_with_compression(tmpdir, compression, engine):
     df.index.name = "index"
     ddf = dd.from_pandas(df, npartitions=3)
 
-    ddf.to_parquet(fn, compression=compression, engine=engine)
+    ddf.to_parquet(fn, compression=compression, engine=engine, write_metadata_file=True)
     out = dd.read_parquet(fn, engine=engine)
     assert_eq(out, ddf)
     check_compression(engine, fn, compression)
@@ -1719,7 +1748,13 @@ def test_writing_parquet_with_partition_on_and_compression(tmpdir, compression,
     df.index.name = "index"
     ddf = dd.from_pandas(df, npartitions=3)
 
-    ddf.to_parquet(fn, compression=compression, engine=engine, partition_on=["x"])
+    ddf.to_parquet(
+        fn,
+        compression=compression,
+        engine=engine,
+        partition_on=["x"],
+        write_metadata_file=True,
+    )
     check_compression(engine, fn, compression)
 
 
@@ -2008,7 +2043,9 @@ def test_writing_parquet_with_kwargs(tmpdir, engine):
         "fastparquet": {"compression": "snappy", "times": "int64", "fixed_text": None},
     }
 
-    ddf.to_parquet(path1, engine=engine, **engine_kwargs[engine])
+    ddf.to_parquet(
+        path1, engine=engine, write_metadata_file=True, **engine_kwargs[engine]
+    )
     out = dd.read_parquet(path1, engine=engine)
     assert_eq(out, ddf, check_index=(engine != "fastparquet"))
 
@@ -2178,7 +2215,7 @@ def test_datasets_timeseries(tmpdir, engine):
     ).persist()
     df.to_parquet(tmp_path, engine=engine)
 
-    df2 = dd.read_parquet(tmp_path, engine=engine)
+    df2 = dd.read_parquet(tmp_path, engine=engine, gather_statistics=True)
     assert_eq(df, df2)
 
 
@@ -2191,7 +2228,7 @@ def test_pathlib_path(tmpdir, engine):
     path = pathlib.Path(str(tmpdir))
     ddf.to_parquet(path, engine=engine)
     ddf2 = dd.read_parquet(path, engine=engine)
-    assert_eq(ddf, ddf2)
+    assert_eq(ddf, ddf2, check_divisions=False)
 
 
 @FASTPARQUET_MARK
@@ -2222,7 +2259,7 @@ def test_read_glob_no_meta(tmpdir, write_engine, read_engine):
 @write_read_engines()
 def test_read_glob_yes_meta(tmpdir, write_engine, read_engine):
     tmp_path = str(tmpdir)
-    ddf.to_parquet(tmp_path, engine=write_engine)
+    ddf.to_parquet(tmp_path, engine=write_engine, write_metadata_file=True)
     paths = glob.glob(os.path.join(tmp_path, "*.parquet"))
     paths.append(os.path.join(tmp_path, "_metadata"))
     ddf2 = dd.read_parquet(paths, engine=read_engine, gather_statistics=False)
@@ -2234,7 +2271,7 @@ def test_read_glob_yes_meta(tmpdir, write_engine, read_engine):
 @write_read_engines()
 def test_read_dir_nometa(tmpdir, write_engine, read_engine, statistics, remove_common):
     tmp_path = str(tmpdir)
-    ddf.to_parquet(tmp_path, engine=write_engine)
+    ddf.to_parquet(tmp_path, engine=write_engine, write_metadata_file=True)
     if os.path.exists(os.path.join(tmp_path, "_metadata")):
         os.unlink(os.path.join(tmp_path, "_metadata"))
     files = os.listdir(tmp_path)
@@ -2446,7 +2483,7 @@ def test_optimize_blockwise_parquet(tmpdir, engine):
     tmp = str(tmpdir)
     df = pd.DataFrame({"a": np.arange(size, dtype=np.int32)})
     expect = dd.from_pandas(df, npartitions=npartitions)
-    expect.to_parquet(tmp, engine=engine)
+    expect.to_parquet(tmp, engine=engine, write_metadata_file=True)
     ddf = dd.read_parquet(tmp, engine=engine)
 
     # `ddf` should now have ONE Blockwise layer
@@ -2661,7 +2698,7 @@ def test_optimize_and_not(tmpdir, engine):
 def test_chunksize_empty(tmpdir, write_engine, read_engine):
     df = pd.DataFrame({"a": pd.Series(dtype="int"), "b": pd.Series(dtype="float")})
     ddf1 = dd.from_pandas(df, npartitions=1)
-    ddf1.to_parquet(tmpdir, engine=write_engine)
+    ddf1.to_parquet(tmpdir, engine=write_engine, write_metadata_file=True)
     ddf2 = dd.read_parquet(tmpdir, engine=read_engine, chunksize="1MiB")
     assert_eq(ddf1, ddf2, check_index=False)
 
@@ -2914,7 +2951,7 @@ def test_pandas_metadata_nullable_pyarrow(tmpdir):
         ),
         npartitions=1,
     )
-    ddf1.to_parquet(tmpdir, engine="pyarrow")
+    ddf1.to_parquet(tmpdir, engine="pyarrow", write_metadata_file=True)
     ddf2 = dd.read_parquet(tmpdir, engine="pyarrow")
 
     assert_eq(ddf1, ddf2, check_index=False)
@@ -3170,8 +3207,7 @@ def test_pyarrow_dataset_simple(tmpdir, engine):
     df.set_index("a", inplace=True, drop=True)
     ddf = dd.from_pandas(df, npartitions=2)
     ddf.to_parquet(fn, engine=engine)
-    read_df = dd.read_parquet(fn, engine="pyarrow")
-    read_df.compute()
+    read_df = dd.read_parquet(fn, engine="pyarrow", gather_statistics=True)
     assert_eq(ddf, read_df)
 
 
@@ -3182,7 +3218,7 @@ def test_pyarrow_dataset_partitioned(tmpdir, engine, test_filter):
     df = pd.DataFrame({"a": [4, 5, 6], "b": ["a", "b", "b"]})
     df["b"] = df["b"].astype("category")
     ddf = dd.from_pandas(df, npartitions=2)
-    ddf.to_parquet(fn, engine=engine, partition_on="b")
+    ddf.to_parquet(fn, engine=engine, partition_on="b", write_metadata_file=True)
     read_df = dd.read_parquet(
         fn,
         engine="pyarrow",
@@ -3266,17 +3302,13 @@ def test_parquet_pyarrow_write_empty_metadata(tmpdir):
     )
 
     df = dd.from_delayed([df_a, df_b, df_c])
-
-    try:
-        df.to_parquet(
-            tmpdir,
-            engine="pyarrow",
-            partition_on=["x"],
-            append=False,
-        )
-
-    except AttributeError:
-        pytest.fail("Unexpected AttributeError")
+    df.to_parquet(
+        tmpdir,
+        engine="pyarrow",
+        partition_on=["x"],
+        append=False,
+        write_metadata_file=True,
+    )
 
     # Check that metadata files where written
     files = os.listdir(tmpdir)
@@ -3310,6 +3342,7 @@ def test_parquet_pyarrow_write_empty_metadata_append(tmpdir):
         engine="pyarrow",
         partition_on=["x"],
         append=False,
+        write_metadata_file=True,
     )
 
     df_c = dask.delayed(pd.DataFrame.from_dict)(
@@ -3326,6 +3359,7 @@ def test_parquet_pyarrow_write_empty_metadata_append(tmpdir):
         partition_on=["x"],
         append=True,
         ignore_divisions=True,
+        write_metadata_file=True,
     )
 
 
@@ -3543,6 +3577,7 @@ def test_custom_metadata(tmpdir, engine):
         path,
         engine=engine,
         custom_metadata=custom_metadata,
+        write_metadata_file=True,
     )
 
     # Check that data is correct
@@ -3654,7 +3689,7 @@ def test_extra_file(tmpdir, engine):
     tmpdir = str(tmpdir)
     df = pd.DataFrame({"a": range(100), "b": ["dog", "cat"] * 50})
     ddf = dd.from_pandas(df, npartitions=2)
-    ddf.to_parquet(tmpdir, engine=engine)
+    ddf.to_parquet(tmpdir, engine=engine, write_metadata_file=True)
     open(os.path.join(tmpdir, "_SUCCESS"), "w").close()
     open(os.path.join(tmpdir, "part.0.parquet.crc"), "w").close()
     os.remove(os.path.join(tmpdir, "_metadata"))
@@ -3717,7 +3752,12 @@ def test_unsupported_extension_dir(tmpdir, engine):
     # (Related to: https://github.com/dask/dask/issues/8349)
     path = str(tmpdir)
     ddf0 = dd.from_pandas(pd.DataFrame({"a": range(10)}), 1)
-    ddf0.to_parquet(path, engine=engine, name_function=lambda i: f"part.{i}.foo")
+    ddf0.to_parquet(
+        path,
+        engine=engine,
+        write_metadata_file=True,
+        name_function=lambda i: f"part.{i}.foo",
+    )
     assert_eq(ddf0, dd.read_parquet(path, engine=engine))
 
 
@@ -3727,7 +3767,12 @@ def test_custom_filename(tmpdir, engine):
         {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
     )
     df = dd.from_pandas(pdf, npartitions=2)
-    df.to_parquet(fn, name_function=lambda x: f"hi-{x}.parquet", engine=engine)
+    df.to_parquet(
+        fn,
+        write_metadata_file=True,
+        name_function=lambda x: f"hi-{x}.parquet",
+        engine=engine,
+    )
 
     files = os.listdir(fn)
     assert "_common_metadata" in files
@@ -3743,7 +3788,12 @@ def test_custom_filename_works_with_pyarrow_when_append_is_true(tmpdir, engine):
         {"num1": [1, 2, 3, 4], "num2": [7, 8, 9, 10]},
     )
     df = dd.from_pandas(pdf, npartitions=2)
-    df.to_parquet(fn, name_function=lambda x: f"hi-{x * 2}.parquet", engine=engine)
+    df.to_parquet(
+        fn,
+        write_metadata_file=True,
+        name_function=lambda x: f"hi-{x * 2}.parquet",
+        engine=engine,
+    )
 
     pdf = pd.DataFrame(
         {"num1": [33], "num2": [44]},