Repository: dask/dask

Issue:
Deprecate "pyarrow-legacy" engine in dask.dataframe.read_parquet
Given the pyarrow version requirements in Dask, we can now assume the `dataset` API will be supported if pyarrow is installed. We can also assume that deprecation warnings will be raised by the pyarrow backend if `dd.read_parquet(..., engine="pyarrow-legacy")` is used.  Therefore, I propose that we add an explicit deprecation warning for the "pyarrow-legacy" engine itself, and establish a timeline for its removal.

Deprecating and removing "pyarrow-legacy" should simplify`read_parquet` maintenance, and should have few (if any) downsides.  However, I welcome pushback if others expect this to cause pain or problems.



Discussion:
Thanks @rjzamora. This makes total sense from a maintenance perspective. What impact will this have on users? IIUC existing Parquet files that were written with the `pyarrow-legacy` engine will still be able to be read in using the `pyarrow-dataset` engine. 

cc @jorisvandenbossche @martindurant 
I would be surprised if there were still options/features available with the legacy backend not covered by dataset. I am not the expert on that, though.
There are still some options that the dataset reader does not yet support compared to the plain ParquetFile reader / the ParquetDataset constructor with `use_legacy_dataset=True`, but I suspect those are not of relevance for (most) dask users. 

- The legacy `ParquetDataset` has a `metadata` parameter for if you had obtained Parquet FileMetaData yourself (I suppose typically used when having a metadata-only file, but that use case is already covered by dask itself). 
- `split_row_groups` keyword: also already handled already by dask itself
- `metadata_nthreads` keyword: to discover the files of the dataset multithreaded. This would be potentially useful for dask I suppose if this happens on the client side to construct the graph (xref issue [#8058](https://github.com/dask/dask/issues/8058)), but something that could also be implemented for the new datasets interface. JIRA for this: https://issues.apache.org/jira/browse/ARROW-8137
- `validate_schema` keyword: in dask, you set this to False by default (if not specified by the user) to override the default of ParquetDataset. That corresponds to the default of the new dataset API (which will by default not check all files of a dataset in advance, but error on runtime in case of non-matching schemas).
- `schema` keyword, but in the legacy interface this is a Parquet schema, and not arrow schema, so that's not something you can construct yourself (and so I doubt many people are using this). In the new dataset interface, there is also a `schema` keyword which might be more useful, though (it's an arrow schema, and with this you can force to use a certain schema on how to read all the files)
> In the new dataset interface, there is also a `schema` keyword which might be more useful, though (it's an arrow schema, and with this you can force to use a certain schema on how to read all the files)

It seems currently no additional keywords are allowed to be passed through to the `pyarrow.dataset` constructor:

https://github.com/dask/dask/blob/ac01ddc9074365e40d888f80f5bcd955ba01e872/dask/dataframe/io/parquet/arrow.py#L780-L782

That's of course a separate question from the deprecation in this issue, but it might be useful to relax the above. 

Thanks for the excellent summary @jorisvandenbossche (Sorry for the delay - I know it was a while back). #8758 is taking the first step at addressing this issue (adding a deprecation warning).

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -1,5 +1,4 @@
 import json
-import warnings
 from collections import defaultdict
 from datetime import datetime
 
@@ -11,7 +10,6 @@
 
 from dask.base import tokenize
 from dask.core import flatten
-from dask.dataframe.io.parquet.core import create_metadata_file
 from dask.dataframe.io.parquet.utils import (
     Engine,
     _flatten_filters,
@@ -49,8 +47,6 @@
 def _append_row_groups(metadata, md):
     """Append row-group metadata and include a helpful
     error message if an inconsistent schema is detected.
-
-    Used by `ArrowDatasetEngine` and `ArrowLegacyEngine`.
     """
     try:
         metadata.append_row_groups(md)
@@ -83,9 +79,6 @@ def _write_partitioned(
     Logic copied from pyarrow.parquet.
     (arrow/python/pyarrow/parquet.py::write_to_dataset)
 
-    Used by `ArrowDatasetEngine` (and by `ArrowLegacyEngine`,
-    through inherited `write_partition` method).
-
     TODO: Remove this in favor of pyarrow's `write_to_dataset`
           once ARROW-8244 is addressed.
     """
@@ -135,9 +128,6 @@ def _write_partitioned(
 def _index_in_schema(index, schema):
     """Simple utility to check if all `index` columns are included
     in the known `schema`.
-
-    Used by `ArrowDatasetEngine` (and by `ArrowLegacyEngine`,
-    through inherited `write_partition` method).
     """
     if index and schema is not None:
         # Make sure all index columns are in user-defined schema
@@ -155,8 +145,6 @@ class PartitionObj:
     object (used in `ArrowLegacyEngine`). The larger purpose of this
     class is to allow the same `read_partition` definition to handle
     both Engine instances.
-
-    Used by `ArrowDatasetEngine` only.
     """
 
     def __init__(self, name, keys):
@@ -165,10 +153,7 @@ def __init__(self, name, keys):
 
 
 def _frag_subset(old_frag, row_groups):
-    """Create new fragment with row-group subset.
-
-    Used by `ArrowDatasetEngine` only.
-    """
+    """Create new fragment with row-group subset."""
     return old_frag.format.make_fragment(
         old_frag.path,
         old_frag.filesystem,
@@ -178,10 +163,7 @@ def _frag_subset(old_frag, row_groups):
 
 
 def _get_pandas_metadata(schema):
-    """Get pandas-specific metadata from schema.
-
-    Used by `ArrowDatasetEngine` and `ArrowLegacyEngine`.
-    """
+    """Get pandas-specific metadata from schema."""
 
     has_pandas_metadata = schema.metadata is not None and b"pandas" in schema.metadata
     if has_pandas_metadata:
@@ -204,7 +186,6 @@ def _read_table_from_path(
 ):
     """Read arrow table from file path.
 
-    Used in all cases by `ArrowLegacyEngine._read_table`.
     Used by `ArrowDatasetEngine._read_table` when no filters
     are specified (otherwise fragments are converted directly
     into tables).
@@ -410,10 +391,8 @@ def read_partition(
         schema=None,
         **kwargs,
     ):
-        """Read in a single output partition.
+        """Read in a single output partition"""
 
-        This method is also used by `ArrowLegacyEngine`.
-        """
         if isinstance(index, list):
             for level in index:
                 # unclear if we can use set ops here. I think the order matters.
@@ -807,8 +786,6 @@ def _collect_dataset_info(
         """pyarrow.dataset version of _collect_dataset_info
         Use pyarrow.dataset API to construct a dictionary of all
         general information needed to read the dataset.
-
-        This method is overriden by `ArrowLegacyEngine`.
         """
 
         # Use pyarrow.dataset API
@@ -999,6 +976,7 @@ def _collect_dataset_info(
         #          "duck type" a `ParquetPartitions` object, so that the
         #          same code path can be used for both legacy and
         #          pyarrow.dataset-based logic.
+        #          TODO: Reconsider now that the legacy API is deprecated
         #    - "partition_names" : (list)  This is a list containing the
         #          names of partitioned columns.
         #    - "partitioning" : (dict) The `partitioning` options
@@ -1034,8 +1012,6 @@ def _collect_dataset_info(
     def _create_dd_meta(cls, dataset_info):
         """Use parquet schema and hive-partition information
         (stored in dataset_info) to construct DataFrame metadata.
-
-        This method is used by both arrow engines.
         """
 
         # Collect necessary information from dataset_info
@@ -1161,8 +1137,6 @@ def _construct_collection_plan(cls, dataset_info):
         and ``common_metadata`` (which is a dictionary of kwargs
         that should be passed to the ``read_partition`` call for
         every output partition).
-
-        This method is overridden in `ArrowLegacyEngine`.
         """
 
         # Collect necessary dataset information from dataset_info
@@ -1508,11 +1482,7 @@ def _make_part(
         partition_obj=None,
         data_path=None,
     ):
-        """Generate a partition-specific element of `parts`.
-
-        This method is used by both `ArrowDatasetEngine`
-        and `ArrowLegacyEngine`.
-        """
+        """Generate a partition-specific element of `parts`."""
 
         # Get full path (empty strings should be ignored)
         full_path = fs.sep.join([p for p in [data_path, filename] if p != ""])
@@ -1535,10 +1505,7 @@ def _read_table(
         partition_keys,
         **kwargs,
     ):
-        """Read in a pyarrow table.
-
-        This method is overridden in `ArrowLegacyEngine`.
-        """
+        """Read in a pyarrow table"""
 
         if isinstance(path_or_frag, pa_ds.ParquetFileFragment):
             frag = path_or_frag
@@ -1681,733 +1648,3 @@ def aggregate_metadata(cls, meta_list, fs, out_path):
             return None
         else:
             return meta
-
-
-#
-#  PyArrow Legacy API [PyArrow<1.0.0]
-#
-
-
-def _get_dataset_object(paths, fs, filters, dataset_kwargs):
-    """Generate a ParquetDataset object"""
-    kwargs = dataset_kwargs.copy()
-    ignore_metadata_file = kwargs.pop("ignore_metadata_file", False)
-    if ignore_metadata_file:
-        raise ValueError("ignore_metadata_file not supported for ArrowLegacyEngine.")
-
-    if "validate_schema" not in kwargs:
-        kwargs["validate_schema"] = False
-    if len(paths) > 1:
-        # This is a list of files
-        paths, base, fns = _sort_and_analyze_paths(paths, fs)
-        proxy_metadata = None
-        if "_metadata" in fns:
-            # We have a _metadata file. PyArrow cannot handle
-            #  "_metadata" when `paths` is a list. So, we shuld
-            # open "_metadata" separately.
-            paths.remove(fs.sep.join([base, "_metadata"]))
-            fns.remove("_metadata")
-            with fs.open(fs.sep.join([base, "_metadata"]), mode="rb") as fil:
-                proxy_metadata = pq.ParquetFile(fil).metadata
-        # Create our dataset from the list of data files.
-        # Note #1: that this will not parse all the files (yet)
-        # Note #2: Cannot pass filters for legacy pyarrow API (see issue#6512).
-        #          We can handle partitions + filtering for list input after
-        #          adopting new pyarrow.dataset API.
-        dataset = pq.ParquetDataset(paths, filesystem=fs, **kwargs)
-        if proxy_metadata:
-            dataset.metadata = proxy_metadata
-    elif fs.isdir(paths[0]):
-        # This is a directory.  We can let pyarrow do its thing.
-        # Note: In the future, it may be best to avoid listing the
-        #       directory if we can get away with checking for the
-        #       existence of _metadata.  Listing may be much more
-        #       expensive in storage systems like S3.
-        allpaths = fs.glob(paths[0] + fs.sep + "*")
-        allpaths, base, fns = _sort_and_analyze_paths(allpaths, fs)
-        dataset = pq.ParquetDataset(paths[0], filesystem=fs, filters=filters, **kwargs)
-    else:
-        # This is a single file.  No danger in gathering statistics
-        # and/or splitting row-groups without a "_metadata" file
-        base = paths[0]
-        fns = [None]
-        dataset = pq.ParquetDataset(paths[0], filesystem=fs, **kwargs)
-
-    return dataset, base, fns
-
-
-class ArrowLegacyEngine(ArrowDatasetEngine):
-
-    #
-    # Private Class Methods
-    #
-
-    @classmethod
-    def _collect_dataset_info(
-        cls,
-        paths,
-        fs,
-        categories,
-        index,
-        gather_statistics,
-        filters,
-        split_row_groups,
-        chunksize,
-        aggregate_files,
-        ignore_metadata_file,
-        metadata_task_size,
-        kwargs,
-    ):
-        """pyarrow-legacy version of _collect_dataset_info
-        Use the ParquetDataset API to construct a dictionary of all
-        general information needed to read the dataset.
-
-        This method overrides `ArrowDatasetEngine._collect_dataset_info`.
-        """
-
-        if ignore_metadata_file:
-            raise ValueError("ignore_metadata_file not supported in ArrowLegacyEngine")
-
-        if metadata_task_size:
-            raise ValueError("metadata_task_size not supported in ArrowLegacyEngine")
-
-        # Extract "supported" key-word arguments from `kwargs`
-        (
-            dataset_kwargs,
-            read_kwargs,
-            user_kwargs,
-        ) = _split_user_options(**kwargs)
-
-        (
-            schema,
-            metadata,
-            base,
-            partition_info,
-            split_row_groups,
-            gather_statistics,
-        ) = cls._gather_metadata(
-            paths,
-            fs,
-            split_row_groups,
-            gather_statistics,
-            filters,
-            index,
-            dataset_kwargs,
-        )
-
-        # Check the `aggregate_files` setting
-        aggregation_depth = _get_aggregation_depth(
-            aggregate_files,
-            partition_info["partition_names"],
-        )
-
-        return {
-            "schema": schema,
-            "metadata": metadata,
-            "fs": fs,
-            "base_path": base,
-            "gather_statistics": gather_statistics,
-            "categories": categories,
-            "index": index,
-            "filters": filters,
-            "split_row_groups": split_row_groups,
-            "chunksize": chunksize,
-            "aggregate_files": aggregate_files,
-            "aggregation_depth": aggregation_depth,
-            "partition_keys": partition_info["partition_keys"],
-            "partition_names": partition_info["partition_names"],
-            "partitions": partition_info["partitions"],
-            "kwargs": {
-                "dataset": dataset_kwargs,
-                "read": read_kwargs,
-                **user_kwargs,
-            },
-        }
-
-    @classmethod
-    def _construct_collection_plan(cls, dataset_info):
-        """pyarrow-legacy version of _construct_collection_plan
-
-        This method overrides the `ArrowDatasetEngine` implementation.
-        """
-
-        # Wrap legacy `_construct_parts` implementation
-        return cls._construct_parts(
-            dataset_info["fs"],
-            dataset_info["metadata"],
-            dataset_info["schema"],
-            dataset_info["filters"],
-            dataset_info["index_cols"],
-            dataset_info["base_path"],
-            {
-                "partition_keys": dataset_info["partition_keys"],
-                "partition_names": dataset_info["partition_names"],
-                "partitions": dataset_info["partitions"],
-            },
-            dataset_info["categories"],
-            dataset_info["split_row_groups"],
-            dataset_info["gather_statistics"],
-            dataset_info["chunksize"],
-            dataset_info["aggregation_depth"],
-            dataset_info["kwargs"],
-        )
-
-    @classmethod
-    def _gather_metadata(
-        cls,
-        paths,
-        fs,
-        split_row_groups,
-        gather_statistics,
-        filters,
-        index,
-        dataset_kwargs,
-    ):
-        """Gather parquet metadata into a single data structure.
-
-        Use _metadata or aggregate footer metadata into a single
-        object.  Also, collect other information necessary for
-        parquet-to-ddf mapping (e.g. schema, partition_info).
-
-        This method overrides `ArrowDatasetEngine._gather_metadata`.
-        """
-
-        # Step 1: Create a ParquetDataset object
-        dataset, base, fns = _get_dataset_object(paths, fs, filters, dataset_kwargs)
-        if fns == [None]:
-            # This is a single file. No danger in gathering statistics
-            # and/or splitting row-groups without a "_metadata" file
-            if gather_statistics is None:
-                gather_statistics = True
-            if split_row_groups is None:
-                split_row_groups = True
-
-        # Step 2: Construct necessary (parquet) partitioning information
-        partition_info = {
-            "partitions": None,
-            "partition_keys": {},
-            "partition_names": [],
-        }
-        # The `partition_info` dict summarizes information needed to handle
-        # nested-directory (hive) partitioning.
-        #
-        #    - "partitions" : (ParquetPartitions) PyArrow-specific  object
-        #          needed to read in each partition correctly
-        #    - "partition_keys" : (dict) The keys and values correspond to
-        #          file paths and partition values, respectively. The partition
-        #          values (or partition "keys") will be represented as a list
-        #          of tuples. E.g. `[("year", 2020), ("state", "CA")]`
-        #    - "partition_names" : (list)  This is a list containing the names
-        #          of partitioned columns.  This list must be ordered correctly
-        #          by partition level.
-        fn_partitioned = False
-        if dataset.partitions is not None:
-            fn_partitioned = True
-            partition_info["partition_names"] = [
-                n.name for n in list(dataset.partitions) if n.name is not None
-            ]
-            partition_info["partitions"] = dataset.partitions
-            for piece in dataset.pieces:
-                partition_info["partition_keys"][piece.path] = piece.partition_keys
-
-        # Make sure gather_statistics allows filtering
-        # (if filters are desired)
-        if filters:
-            # Filters may require us to gather statistics
-            if gather_statistics is False and partition_info["partition_names"]:
-                warnings.warn(
-                    "Filtering with gather_statistics=False. "
-                    "Only partition columns will be filtered correctly."
-                )
-            elif gather_statistics is False:
-                raise ValueError("Cannot apply filters with gather_statistics=False")
-            elif not gather_statistics:
-                gather_statistics = True
-
-        # Step 3: Construct a single `metadata` object. We can
-        #         directly use dataset.metadata if it is available.
-        #         Otherwise, if `gather_statistics` or `split_row_groups`,
-        #         we need to gether the footer metadata manually
-        metadata = None
-        if dataset.metadata:
-            # We have a _metadata file.
-            # PyArrow already did the work for us
-            schema = dataset.metadata.schema.to_arrow_schema()
-            if gather_statistics is None:
-                gather_statistics = True
-            if split_row_groups is None:
-                split_row_groups = True
-            return (
-                schema,
-                dataset.metadata,
-                base,
-                partition_info,
-                split_row_groups,
-                gather_statistics,
-            )
-        else:
-            # No _metadata file.
-            # May need to collect footer metadata manually
-            if dataset.schema is not None:
-                schema = dataset.schema.to_arrow_schema()
-            else:
-                schema = None
-            if gather_statistics is None:
-                gather_statistics = False
-            if split_row_groups is None:
-                split_row_groups = False
-            metadata = None
-            if not (split_row_groups or gather_statistics):
-                # Don't need to construct real metadata if
-                # we are not gathering statistics or splitting
-                # by row-group
-                metadata = [p.path for p in dataset.pieces]
-                if schema is None:
-                    schema = dataset.pieces[0].get_metadata().schema.to_arrow_schema()
-                return (
-                    schema,
-                    metadata,
-                    base,
-                    partition_info,
-                    split_row_groups,
-                    gather_statistics,
-                )
-            # We have not detected a _metadata file, and the user has specified
-            # that they want to split by row-group and/or gather statistics.
-            # This is the only case where we MUST scan all files to collect
-            # metadata.
-            if len(dataset.pieces) > 1:
-                # Perform metadata collection in parallel.
-                metadata = create_metadata_file(
-                    [p.path for p in dataset.pieces],
-                    root_dir=base,
-                    engine=cls,
-                    out_dir=False,
-                    fs=fs,
-                )
-                if schema is None:
-                    schema = metadata.schema.to_arrow_schema()
-            else:
-                for piece, fn in zip(dataset.pieces, fns):
-                    md = piece.get_metadata()
-                    if schema is None:
-                        schema = md.schema.to_arrow_schema()
-                    if fn_partitioned:
-                        md.set_file_path(piece.path.replace(base + fs.sep, ""))
-                    elif fn:
-                        md.set_file_path(fn)
-                    if metadata:
-                        _append_row_groups(metadata, md)
-                    else:
-                        metadata = md
-
-            return (
-                schema,
-                metadata,
-                base,
-                partition_info,
-                split_row_groups,
-                gather_statistics,
-            )
-
-    @classmethod
-    def _construct_parts(
-        cls,
-        fs,
-        metadata,
-        schema,
-        filters,
-        index_cols,
-        data_path,
-        partition_info,
-        categories,
-        split_row_groups,
-        gather_statistics,
-        chunksize,
-        aggregation_depth,
-        kwargs,
-    ):
-        """Construct ``parts`` for ddf construction
-
-        Use metadata (along with other data) to define a tuple
-        for each ddf partition.  Also gather statistics if
-        ``gather_statistics=True``, and other criteria is met.
-
-        This method is only used by `ArrowLegacyEngine`.
-        """
-
-        partition_keys = partition_info["partition_keys"]
-        partition_obj = partition_info["partitions"]
-
-        # Check if `metadata` is just a list of paths
-        # (not splitting by row-group or collecting statistics)
-        if (
-            isinstance(metadata, list)
-            and len(metadata)
-            and isinstance(metadata[0], str)
-        ):
-            parts = []
-            stats = []
-            for full_path in metadata:
-                part = {"piece": (full_path, None, partition_keys.get(full_path, None))}
-                parts.append(part)
-            common_kwargs = {
-                "partitions": partition_obj,
-                "categories": categories,
-                **kwargs,
-            }
-            return parts, stats, common_kwargs
-
-        # Use final metadata info to update our options for
-        # `parts`/`stats` construnction
-        (
-            gather_statistics,
-            split_row_groups,
-            stat_col_indices,
-        ) = cls._update_metadata_options(
-            gather_statistics,
-            split_row_groups,
-            metadata,
-            schema,
-            index_cols,
-            filters,
-            partition_info,
-            chunksize,
-            aggregation_depth,
-        )
-
-        # Convert metadata into `parts` and `stats`
-        return cls._process_metadata(
-            metadata,
-            schema,
-            split_row_groups,
-            gather_statistics,
-            stat_col_indices,
-            filters,
-            categories,
-            partition_info,
-            data_path,
-            fs,
-            chunksize,
-            aggregation_depth,
-            kwargs,
-        )
-
-    @classmethod
-    def _update_metadata_options(
-        cls,
-        gather_statistics,
-        split_row_groups,
-        metadata,
-        schema,
-        index_cols,
-        filters,
-        partition_info,
-        chunksize,
-        aggregation_depth,
-    ):
-        """Update read_parquet options given up-to-data metadata.
-
-        The primary focus here is `gather_statistics`. We want to
-        avoid setting this option to `True` if it is unnecessary.
-
-        This method is only used by `ArrowLegacyEngine`.
-        """
-
-        # Cannot gather_statistics if our `metadata` is a list
-        # of paths, or if we are building a multiindex (for now).
-        # We also don't "need" to gather statistics if we don't
-        # want to apply any filters or calculate divisions. Note
-        # that the `ArrowDatasetEngine` doesn't even require
-        # `gather_statistics=True` for filtering.
-        if split_row_groups is None:
-            split_row_groups = False
-        _need_aggregation_stats = chunksize or (
-            int(split_row_groups) > 1 and aggregation_depth
-        )
-        if (
-            isinstance(metadata, list)
-            and len(metadata)
-            and isinstance(metadata[0], str)
-        ) or len(index_cols) > 1:
-            gather_statistics = False
-        elif not _need_aggregation_stats and filters is None and len(index_cols) == 0:
-            gather_statistics = False
-
-        # Determine which columns need statistics.
-        flat_filters = _flatten_filters(filters)
-        stat_col_indices = {}
-        for i, name in enumerate(schema.names):
-            if name in index_cols or name in flat_filters:
-                if name in partition_info["partition_names"]:
-                    # Partition columns wont have statistics
-                    continue
-                stat_col_indices[name] = i
-
-        # If the user has not specified `gather_statistics`,
-        # we will only do so if there are specific columns in
-        # need of statistics.
-        # NOTE: We cannot change `gather_statistics` from True
-        # to False (even if `stat_col_indices` is empty), in
-        # case a `chunksize` was specified, and the row-group
-        # statistics are needed for part aggregation.
-        if gather_statistics is None:
-            gather_statistics = bool(stat_col_indices)
-
-        return (
-            gather_statistics,
-            split_row_groups,
-            stat_col_indices,
-        )
-
-    @classmethod
-    def _organize_row_groups(
-        cls,
-        metadata,
-        split_row_groups,
-        gather_statistics,
-        stat_col_indices,
-        filters,
-        chunksize,
-        aggregation_depth,
-    ):
-        """Organize row-groups by file.
-
-        This method is used by ArrowLegacyEngine._process_metadata
-        """
-
-        sorted_row_group_indices = range(metadata.num_row_groups)
-        if aggregation_depth:
-            sorted_row_group_indices = sorted(
-                range(metadata.num_row_groups),
-                key=lambda x: metadata.row_group(x).column(0).file_path,
-            )
-
-        # Get the number of row groups per file
-        single_rg_parts = int(split_row_groups) == 1
-        file_row_groups = defaultdict(list)
-        file_row_group_stats = defaultdict(list)
-        file_row_group_column_stats = defaultdict(list)
-        cmax_last = {}
-        for rg in sorted_row_group_indices:
-            row_group = metadata.row_group(rg)
-
-            # NOTE: Here we assume that all column chunks are stored
-            # in the same file. This is not strictly required by the
-            # parquet spec.
-            fpath = row_group.column(0).file_path
-            if fpath is None:
-                raise ValueError(
-                    "Global metadata structure is missing a file_path string. "
-                    "If the dataset includes a _metadata file, that file may "
-                    "have one or more missing file_path fields."
-                )
-            if file_row_groups[fpath]:
-                file_row_groups[fpath].append(file_row_groups[fpath][-1] + 1)
-            else:
-                file_row_groups[fpath].append(0)
-            if gather_statistics:
-                if single_rg_parts:
-                    s = {
-                        "file_path_0": fpath,
-                        "num-rows": row_group.num_rows,
-                        "total_byte_size": row_group.total_byte_size,
-                        "columns": [],
-                    }
-                else:
-                    s = {
-                        "num-rows": row_group.num_rows,
-                        "total_byte_size": row_group.total_byte_size,
-                    }
-                cstats = []
-                for name, i in stat_col_indices.items():
-                    column = row_group.column(i)
-                    if column.statistics:
-                        cmin = column.statistics.min
-                        cmax = column.statistics.max
-                        last = cmax_last.get(name, None)
-                        if not (filters or chunksize or aggregation_depth):
-                            # Only think about bailing if we don't need
-                            # stats for filtering
-                            if cmin is None or (last and cmin < last):
-                                # We are collecting statistics for divisions
-                                # only (no filters) - Column isn't sorted, or
-                                # we have an all-null partition, so lets bail.
-                                #
-                                # Note: This assumes ascending order.
-                                #
-                                gather_statistics = False
-                                file_row_group_stats = {}
-                                file_row_group_column_stats = {}
-                                break
-
-                        if single_rg_parts:
-                            to_ts = column.statistics.logical_type.type == "TIMESTAMP"
-                            s["columns"].append(
-                                {
-                                    "name": name,
-                                    "min": cmin if not to_ts else pd.Timestamp(cmin),
-                                    "max": cmax if not to_ts else pd.Timestamp(cmax),
-                                }
-                            )
-                        else:
-                            cstats += [cmin, cmax]
-                        cmax_last[name] = cmax
-                    else:
-
-                        if (
-                            not (filters or chunksize or aggregation_depth)
-                            and column.num_values > 0
-                        ):
-                            # We are collecting statistics for divisions
-                            # only (no filters) - Lets bail.
-                            gather_statistics = False
-                            file_row_group_stats = {}
-                            file_row_group_column_stats = {}
-                            break
-
-                        if single_rg_parts:
-                            s["columns"].append({"name": name})
-                        else:
-                            cstats += [None, None, None]
-                if gather_statistics:
-                    file_row_group_stats[fpath].append(s)
-                    if not single_rg_parts:
-                        file_row_group_column_stats[fpath].append(tuple(cstats))
-
-        return (
-            file_row_groups,
-            file_row_group_stats,
-            file_row_group_column_stats,
-            gather_statistics,
-        )
-
-    @classmethod
-    def _process_metadata(
-        cls,
-        metadata,
-        schema,
-        split_row_groups,
-        gather_statistics,
-        stat_col_indices,
-        filters,
-        categories,
-        partition_info,
-        data_path,
-        fs,
-        chunksize,
-        aggregation_depth,
-        kwargs,
-    ):
-        """Process row-groups and statistics.
-
-        This method is only used by `ArrowLegacyEngine`.
-        """
-
-        # Organize row-groups by file
-        (
-            file_row_groups,
-            file_row_group_stats,
-            file_row_group_column_stats,
-            gather_statistics,
-        ) = cls._organize_row_groups(
-            metadata,
-            split_row_groups,
-            gather_statistics,
-            stat_col_indices,
-            filters,
-            chunksize,
-            aggregation_depth,
-        )
-
-        # Convert organized row-groups to parts
-        parts, stats = _row_groups_to_parts(
-            gather_statistics,
-            split_row_groups,
-            aggregation_depth,
-            file_row_groups,
-            file_row_group_stats,
-            file_row_group_column_stats,
-            stat_col_indices,
-            cls._make_part,
-            make_part_kwargs={
-                "fs": fs,
-                "partition_keys": partition_info.get("partition_keys", None),
-                "partition_obj": partition_info.get("partitions", None),
-                "data_path": data_path,
-            },
-        )
-
-        # Add common kwargs
-        common_kwargs = {
-            "partitions": partition_info["partitions"],
-            "categories": categories,
-            "filters": filters,
-            **kwargs,
-        }
-
-        return parts, stats, common_kwargs
-
-    @classmethod
-    def _read_table(
-        cls,
-        path,
-        fs,
-        row_groups,
-        columns,
-        schema,
-        filters,
-        partitions,
-        partition_keys,
-        **kwargs,
-    ):
-        """Read in a pyarrow table.
-
-        This method is overrides the `ArrowDatasetEngine` implementation.
-        """
-
-        return _read_table_from_path(
-            path,
-            fs,
-            row_groups,
-            columns,
-            schema,
-            filters,
-            partitions,
-            partition_keys,
-            cls._parquet_piece_as_arrow,
-            **kwargs,
-        )
-
-    @classmethod
-    def multi_support(cls):
-        return cls == ArrowLegacyEngine
-
-    @classmethod
-    def _get_dataset_offset(cls, path, fs, append, ignore_divisions):
-        dataset = fmd = None
-        i_offset = 0
-        if append:
-            try:
-                # Allow append if the dataset exists.
-                # Also need dataset.metadata object if
-                # ignore_divisions is False (to check divisions)
-                dataset = pq.ParquetDataset(path, filesystem=fs)
-                if not dataset.metadata and not ignore_divisions:
-                    # TODO: Be more flexible about existing metadata.
-                    raise NotImplementedError(
-                        "_metadata file needed to `append` "
-                        "with `engine='pyarrow-legacy'` "
-                        "unless `ignore_divisions` is `True`"
-                    )
-                fmd = dataset.metadata
-                i_offset = len(dataset.pieces)
-            except (OSError, ValueError, IndexError):
-                # Original dataset does not exist - cannot append
-                append = False
-        return fmd, i_offset, append
-
-
-# Compatibility access to legacy ArrowEngine
-# (now called `ArrowLegacyEngine`)
-ArrowEngine = ArrowLegacyEngine
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -238,7 +238,6 @@ def read_parquet(
         if fastparquet is installed (and ArrowDatasetEngine otherwise). If
         'pyarrow' is specified, ArrowDatasetEngine (which leverages the
         pyarrow.dataset API) will be used.
-        NOTE: The 'pyarrow-legacy' option (ArrowLegacyEngine) is deprecated.
     gather_statistics : bool, default None
         Gather the statistics for each dataset partition. By default,
         this will only be done if the _metadata file is available. Otherwise,
@@ -1018,15 +1017,15 @@ def get_engine(engine):
         _ENGINES["fastparquet"] = eng = FastParquetEngine
         return eng
 
-    elif engine in ("pyarrow", "arrow", "pyarrow-legacy", "pyarrow-dataset"):
+    elif engine in ("pyarrow", "arrow", "pyarrow-dataset"):
 
         pa = import_required("pyarrow", "`pyarrow` not installed")
         pa_version = parse_version(pa.__version__)
 
-        if engine in ("pyarrow", "arrow"):
-            engine = "pyarrow-dataset"
+        if engine in ("pyarrow-dataset", "arrow"):
+            engine = "pyarrow"
 
-        if engine == "pyarrow-dataset":
+        if engine == "pyarrow":
             if pa_version.major < 1:
                 raise ImportError(
                     f"pyarrow-{pa_version.major} does not support the "
@@ -1036,17 +1035,6 @@ def get_engine(engine):
             from dask.dataframe.io.parquet.arrow import ArrowDatasetEngine
 
             _ENGINES[engine] = eng = ArrowDatasetEngine
-        else:
-            from dask.dataframe.io.parquet.arrow import ArrowLegacyEngine
-
-            warnings.warn(
-                "`ArrowLegacyEngine` ('pyarrow-legacy') is deprecated "
-                "and will be removed in the near future. Please use "
-                "`engine='pyarrow'` instead.",
-                FutureWarning,
-                stacklevel=3,
-            )
-            _ENGINES[engine] = eng = ArrowLegacyEngine
         return eng
 
     else:


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -60,23 +60,6 @@
     SKIP_PYARROW_REASON = "pyarrow not found"
 PYARROW_MARK = pytest.mark.skipif(SKIP_PYARROW, reason=SKIP_PYARROW_REASON)
 
-# "Legacy" and "Dataset"-specific MARK definitions
-SKIP_PYARROW_LE = SKIP_PYARROW
-SKIP_PYARROW_LE_REASON = "pyarrow not found"
-SKIP_PYARROW_DS = SKIP_PYARROW
-SKIP_PYARROW_DS_REASON = "pyarrow not found"
-if not SKIP_PYARROW_LE:
-    # NOTE: We should use PYARROW_LE_MARK to skip
-    # pyarrow-legacy tests once pyarrow officially
-    # removes ParquetDataset support in the future.
-    PYARROW_LE_MARK = pytest.mark.filterwarnings(
-        "ignore::DeprecationWarning",
-        "ignore::FutureWarning",
-    )
-else:
-    PYARROW_LE_MARK = pytest.mark.skipif(SKIP_PYARROW_LE, reason=SKIP_PYARROW_LE_REASON)
-PYARROW_DS_MARK = pytest.mark.skipif(SKIP_PYARROW_DS, reason=SKIP_PYARROW_DS_REASON)
-
 ANY_ENGINE_MARK = pytest.mark.skipif(
     SKIP_FASTPARQUET and SKIP_PYARROW,
     reason="No parquet engine (fastparquet or pyarrow) found",
@@ -99,8 +82,7 @@
 @pytest.fixture(
     params=[
         pytest.param("fastparquet", marks=FASTPARQUET_MARK),
-        pytest.param("pyarrow-legacy", marks=PYARROW_LE_MARK),
-        pytest.param("pyarrow-dataset", marks=PYARROW_DS_MARK),
+        pytest.param("pyarrow", marks=PYARROW_MARK),
     ]
 )
 def engine(request):
@@ -112,13 +94,12 @@ def write_read_engines(**kwargs):
 
     To add custom marks, pass keyword of the form: `mark_writer_reader=reason`,
     or `mark_engine=reason` to apply to all parameters with that engine."""
-    backends = {"pyarrow-dataset", "pyarrow-legacy", "fastparquet"}
+    backends = {"pyarrow", "fastparquet"}
 
     # Skip if uninstalled
     skip_marks = {
         "fastparquet": FASTPARQUET_MARK,
-        "pyarrow-legacy": PYARROW_LE_MARK,
-        "pyarrow-dataset": PYARROW_DS_MARK,
+        "pyarrow": PYARROW_MARK,
     }
     marks = {(w, r): [skip_marks[w], skip_marks[r]] for w in backends for r in backends}
 
@@ -145,8 +126,7 @@ def write_read_engines(**kwargs):
 pyarrow_fastparquet_msg = "pyarrow schema and pandas metadata may disagree"
 write_read_engines_xfail = write_read_engines(
     **{
-        "xfail_pyarrow-dataset_fastparquet": pyarrow_fastparquet_msg,
-        "xfail_pyarrow-legacy_fastparquet": pyarrow_fastparquet_msg,
+        "xfail_pyarrow_fastparquet": pyarrow_fastparquet_msg,
     }
 )
 
@@ -162,11 +142,9 @@ def write_read_engines(**kwargs):
     fp_pandas_msg = "pandas with fastparquet engine does not preserve index"
     fp_pandas_xfail = write_read_engines(
         **{
-            "xfail_pyarrow-dataset_fastparquet": pyarrow_fastparquet_msg,
-            "xfail_pyarrow-legacy_fastparquet": pyarrow_fastparquet_msg,
+            "xfail_pyarrow_fastparquet": pyarrow_fastparquet_msg,
             "xfail_fastparquet_fastparquet": fp_pandas_msg,
-            "xfail_fastparquet_pyarrow-dataset": fp_pandas_msg,
-            "xfail_fastparquet_pyarrow-legacy": fp_pandas_msg,
+            "xfail_fastparquet_pyarrow": fp_pandas_msg,
         }
     )
 else:
@@ -184,10 +162,6 @@ def test_pyarrow_getengine():
     assert get_engine("pyarrow") == ArrowDatasetEngine
     assert get_engine("arrow") == ArrowDatasetEngine
 
-    if SKIP_PYARROW_LE:
-        with pytest.warns(FutureWarning):
-            get_engine("pyarrow-legacy")
-
 
 @write_read_engines()
 def test_local(tmpdir, write_engine, read_engine):
@@ -1080,45 +1054,6 @@ def test_to_parquet_default_writes_nulls(tmpdir):
     assert table[1].null_count == 2
 
 
-@PYARROW_LE_MARK
-def test_to_parquet_pyarrow_w_inconsistent_schema_by_partition_fails_by_default(tmpdir):
-    df = pd.DataFrame(
-        {"partition_column": [0, 0, 1, 1], "strings": ["a", "b", None, None]}
-    )
-
-    ddf = dd.from_pandas(df, npartitions=2)
-    # In order to allow pyarrow to write an inconsistent schema,
-    # we need to avoid writing the _metadata file (will fail >0.17.1)
-    # and need to avoid schema inference (i.e. use `schema=None`)
-    ddf.to_parquet(
-        str(tmpdir),
-        engine="pyarrow",
-        partition_on=["partition_column"],
-        write_metadata_file=False,
-        schema=None,
-    )
-
-    # Test that schema is not validated by default
-    # (shouldn't raise error with legacy dataset)
-    dd.read_parquet(
-        str(tmpdir),
-        engine="pyarrow-legacy",
-        gather_statistics=False,
-    ).compute()
-
-    # Test that read fails when validate_schema=True
-    # Note: This fails differently for pyarrow.dataset api
-    with pytest.raises(ValueError) as e_info:
-        dd.read_parquet(
-            str(tmpdir),
-            engine="pyarrow-legacy",
-            gather_statistics=False,
-            dataset={"validate_schema": True},
-        ).compute()
-        assert e_info.message.contains("ValueError: Schema in partition")
-        assert e_info.message.contains("was different")
-
-
 @PYARROW_MARK
 def test_to_parquet_pyarrow_w_inconsistent_schema_by_partition_succeeds_w_manual_schema(
     tmpdir,
@@ -1410,7 +1345,7 @@ def test_filters_v0(tmpdir, write_engine, read_engine):
 
     # Recent versions of pyarrow support full row-wise filtering
     # (fastparquet and older pyarrow versions do not)
-    pyarrow_row_filtering = read_engine == "pyarrow-dataset"
+    pyarrow_row_filtering = read_engine == "pyarrow"
 
     fn = str(tmpdir)
     df = pd.DataFrame({"at": ["ab", "aa", "ba", "da", "bb"]})
@@ -1471,7 +1406,7 @@ def test_filtering_pyarrow_dataset(tmpdir, engine):
     aa_lim = 40
     bb_val = "dog"
     filters = [[("aa", "<", aa_lim), ("bb", "==", bb_val)]]
-    ddf2 = dd.read_parquet(fn, index=False, engine="pyarrow-dataset", filters=filters)
+    ddf2 = dd.read_parquet(fn, index=False, engine="pyarrow", filters=filters)
 
     # Check that partitions are filetered for "aa" filter
     nonempty = 0
@@ -1528,7 +1463,7 @@ def test_pyarrow_filter_divisions(tmpdir):
     # when `split_row_groups=False`.
     ddf = dd.read_parquet(
         str(tmpdir),
-        engine="pyarrow-dataset",
+        engine="pyarrow",
         split_row_groups=False,
         gather_statistics=True,
         filters=[("a", "<=", 3)],
@@ -1537,7 +1472,7 @@ def test_pyarrow_filter_divisions(tmpdir):
 
     ddf = dd.read_parquet(
         str(tmpdir),
-        engine="pyarrow-dataset",
+        engine="pyarrow",
         split_row_groups=True,
         gather_statistics=True,
         filters=[("a", "<=", 3)],
@@ -2056,14 +1991,13 @@ def test_writing_parquet_with_kwargs(tmpdir, engine):
     ddf = dd.from_pandas(df, npartitions=3)
 
     engine_kwargs = {
-        "pyarrow-dataset": {
+        "pyarrow": {
             "compression": "snappy",
             "coerce_timestamps": None,
             "use_dictionary": True,
         },
         "fastparquet": {"compression": "snappy", "times": "int64", "fixed_text": None},
     }
-    engine_kwargs["pyarrow-legacy"] = engine_kwargs["pyarrow-dataset"]
 
     ddf.to_parquet(path1, engine=engine, **engine_kwargs[engine])
     out = dd.read_parquet(path1, engine=engine)
@@ -2252,18 +2186,6 @@ def test_pathlib_path(tmpdir, engine):
     assert_eq(ddf, ddf2)
 
 
-@PYARROW_LE_MARK
-def test_pyarrow_metadata_nthreads(tmpdir):
-    tmp_path = str(tmpdir)
-    df = pd.DataFrame({"x": [4, 5, 6, 1, 2, 3]})
-    df.index.name = "index"
-    ddf = dd.from_pandas(df, npartitions=2)
-    ddf.to_parquet(tmp_path, engine="pyarrow")
-    ops = {"dataset": {"metadata_nthreads": 2}}
-    ddf2 = dd.read_parquet(tmp_path, engine="pyarrow-legacy", **ops)
-    assert_eq(ddf, ddf2)
-
-
 @FASTPARQUET_MARK
 def test_categories_large(tmpdir, engine):
     # Issue #5112
@@ -2348,96 +2270,12 @@ def test_timeseries_nulls_in_schema(tmpdir, engine, schema):
     ddf2.name = ddf2.name.where(ddf2.timestamp == "2000-01-01", None)
 
     # Note: `append_row_groups` will fail with pyarrow>0.17.1 for _metadata write
-    dataset = {"validate_schema": False} if engine == "pyarrow-legacy" else {}
     ddf2.to_parquet(tmp_path, engine=engine, write_metadata_file=False, schema=schema)
-    ddf_read = dd.read_parquet(tmp_path, engine=engine, dataset=dataset)
+    ddf_read = dd.read_parquet(tmp_path, engine=engine)
 
     assert_eq(ddf_read, ddf2, check_divisions=False, check_index=False)
 
 
-@PYARROW_LE_MARK
-@pytest.mark.parametrize("numerical", [True, False])
-@pytest.mark.parametrize(
-    "timestamp", ["2000-01-01", "2000-01-02", "2000-01-03", "2000-01-04"]
-)
-def test_timeseries_nulls_in_schema_pyarrow(tmpdir, timestamp, numerical):
-    tmp_path = str(tmpdir)
-    ddf2 = dd.from_pandas(
-        pd.DataFrame(
-            {
-                "timestamp": [
-                    pd.Timestamp("2000-01-01"),
-                    pd.Timestamp("2000-01-02"),
-                    pd.Timestamp("2000-01-03"),
-                    pd.Timestamp("2000-01-04"),
-                ],
-                "id": np.arange(4, dtype="float64"),
-                "name": ["cat", "dog", "bird", "cow"],
-            }
-        ),
-        npartitions=2,
-    ).persist()
-    if numerical:
-        ddf2.id = ddf2.id.where(ddf2.timestamp == timestamp, None)
-        ddf2.id = ddf2.id.astype("float64")
-    else:
-        ddf2.name = ddf2.name.where(ddf2.timestamp == timestamp, None)
-
-    # There should be no schema error if you specify a schema on write
-    schema = pa.schema(
-        [("timestamp", pa.timestamp("ns")), ("id", pa.float64()), ("name", pa.string())]
-    )
-    ddf2.to_parquet(tmp_path, schema=schema, write_index=False, engine="pyarrow")
-    assert_eq(
-        dd.read_parquet(
-            tmp_path,
-            dataset={"validate_schema": True},
-            index=False,
-            engine="pyarrow-legacy",
-        ),
-        ddf2,
-        check_divisions=False,
-        check_index=False,
-    )
-
-
-@PYARROW_LE_MARK
-def test_read_inconsistent_schema_pyarrow(tmpdir):
-    # Note: This is a proxy test for a cudf-related issue fix
-    # (see cudf#5062 github issue).  The cause of that issue is
-    # schema inconsistencies that do not actually correspond to
-    # different types, but whether or not the file/column contains
-    # null values.
-
-    df1 = pd.DataFrame({"id": [0, 1], "val": [10, 20]})
-    df2 = pd.DataFrame({"id": [2, 3], "val": [30, 40]})
-
-    desired_type = "int64"
-    other_type = "int32"
-    df1.val = df1.val.astype(desired_type)
-    df2.val = df2.val.astype(other_type)
-
-    df_expect = pd.concat([df1, df2], ignore_index=True)
-    df_expect["val"] = df_expect.val.astype(desired_type)
-
-    df1.to_parquet(os.path.join(tmpdir, "0.parquet"), engine="pyarrow")
-    df2.to_parquet(os.path.join(tmpdir, "1.parquet"), engine="pyarrow")
-
-    # Read Directory
-    check = dd.read_parquet(
-        str(tmpdir), dataset={"validate_schema": False}, engine="pyarrow-legacy"
-    )
-    assert_eq(check.compute(), df_expect, check_index=False)
-
-    # Read List
-    check = dd.read_parquet(
-        os.path.join(tmpdir, "*.parquet"),
-        dataset={"validate_schema": False},
-        engine="pyarrow-legacy",
-    )
-    assert_eq(check.compute(), df_expect, check_index=False)
-
-
 def test_graph_size_pyarrow(tmpdir, engine):
     import pickle
 
@@ -3096,10 +2934,7 @@ def test_pandas_timestamp_overflow_pyarrow(tmpdir):
         dd.read_parquet(str(tmpdir), engine="pyarrow").compute()
     assert "out of bounds" in str(e.value)
 
-    if pa_version >= parse_version("5.0.0"):
-        from dask.dataframe.io.parquet.arrow import ArrowDatasetEngine as ArrowEngine
-    else:
-        from dask.dataframe.io.parquet.arrow import ArrowEngine
+    from dask.dataframe.io.parquet.arrow import ArrowDatasetEngine as ArrowEngine
 
     class ArrowEngineWithTimestampClamp(ArrowEngine):
         @classmethod
@@ -3331,7 +3166,7 @@ def test_pyarrow_dataset_simple(tmpdir, engine):
     df.set_index("a", inplace=True, drop=True)
     ddf = dd.from_pandas(df, npartitions=2)
     ddf.to_parquet(fn, engine=engine)
-    read_df = dd.read_parquet(fn, engine="pyarrow-dataset")
+    read_df = dd.read_parquet(fn, engine="pyarrow")
     read_df.compute()
     assert_eq(ddf, read_df)
 
@@ -3400,7 +3235,7 @@ def test_pyarrow_dataset_filter_partitioned(tmpdir, split_row_groups):
     # Filter on a a non-partition column
     read_df = dd.read_parquet(
         fn,
-        engine="pyarrow-dataset",
+        engine="pyarrow",
         split_row_groups=split_row_groups,
         filters=[("a", "==", 5)],
     )
@@ -3757,28 +3592,19 @@ def test_ignore_metadata_file(tmpdir, engine, gather_statistics):
 
     # Read back the datasets with `ignore_metadata_file=True`, and
     # test that the results are the same
-    if engine != "pyarrow-legacy":
-        ddf2a = dd.read_parquet(
-            dataset_with_bad_metadata,
-            engine=engine,
-            ignore_metadata_file=True,
-            gather_statistics=gather_statistics,
-        )
-        ddf2b = dd.read_parquet(
-            dataset_without_metadata,
-            engine=engine,
-            ignore_metadata_file=True,
-            gather_statistics=gather_statistics,
-        )
-        assert_eq(ddf2a, ddf2b)
-    else:
-        # Check that "pyarrow-legacy" raises a ValueError
-        with pytest.raises(ValueError):
-            dd.read_parquet(
-                dataset_with_bad_metadata,
-                engine=engine,
-                ignore_metadata_file=True,
-            )
+    ddf2a = dd.read_parquet(
+        dataset_with_bad_metadata,
+        engine=engine,
+        ignore_metadata_file=True,
+        gather_statistics=gather_statistics,
+    )
+    ddf2b = dd.read_parquet(
+        dataset_without_metadata,
+        engine=engine,
+        ignore_metadata_file=True,
+        gather_statistics=gather_statistics,
+    )
+    assert_eq(ddf2a, ddf2b)
 
 
 @pytest.mark.parametrize("write_metadata_file", [True, False])
@@ -3794,39 +3620,28 @@ def test_metadata_task_size(tmpdir, engine, write_metadata_file, metadata_task_s
     )
 
     # Read back
-    if engine != "pyarrow-legacy" or not metadata_task_size:
-        ddf2a = dd.read_parquet(
-            str(tmpdir),
-            engine=engine,
-            gather_statistics=True,
-        )
-        ddf2b = dd.read_parquet(
+    ddf2a = dd.read_parquet(
+        str(tmpdir),
+        engine=engine,
+        gather_statistics=True,
+    )
+    ddf2b = dd.read_parquet(
+        str(tmpdir),
+        engine=engine,
+        gather_statistics=True,
+        metadata_task_size=metadata_task_size,
+    )
+    assert_eq(ddf2a, ddf2b)
+
+    with dask.config.set(
+        {"dataframe.parquet.metadata-task-size-local": metadata_task_size}
+    ):
+        ddf2c = dd.read_parquet(
             str(tmpdir),
             engine=engine,
             gather_statistics=True,
-            metadata_task_size=metadata_task_size,
         )
-        assert_eq(ddf2a, ddf2b)
-
-        with dask.config.set(
-            {"dataframe.parquet.metadata-task-size-local": metadata_task_size}
-        ):
-            ddf2c = dd.read_parquet(
-                str(tmpdir),
-                engine=engine,
-                gather_statistics=True,
-            )
-        assert_eq(ddf2b, ddf2c)
-
-    else:
-        # Check that other engines raise a ValueError
-        with pytest.raises(ValueError):
-            dd.read_parquet(
-                str(tmpdir),
-                engine=engine,
-                gather_statistics=True,
-                metadata_task_size=metadata_task_size,
-            )
+    assert_eq(ddf2b, ddf2c)
 
 
 def test_extra_file(tmpdir, engine):
@@ -3842,29 +3657,28 @@ def test_extra_file(tmpdir, engine):
     out = dd.read_parquet(tmpdir, engine=engine)
     assert_eq(out, df)
 
-    if engine != "pyarrow-legacy":
-        # For "fastparquet" and "pyarrow-dataset", we can pass the
-        # expected file extension, or avoid checking file extensions
-        # by passing False. Check here that this works:
+    # For "fastparquet" and "pyarrow", we can pass the
+    # expected file extension, or avoid checking file extensions
+    # by passing False. Check here that this works:
 
-        # Should Work
-        out = dd.read_parquet(
-            tmpdir, engine=engine, dataset={"require_extension": ".parquet"}
-        )
-        assert_eq(out, df)
+    # Should Work
+    out = dd.read_parquet(
+        tmpdir, engine=engine, dataset={"require_extension": ".parquet"}
+    )
+    assert_eq(out, df)
 
-        # Should Fail (for not capturing the _SUCCESS and crc files)
-        with pytest.raises((OSError, pa.lib.ArrowInvalid)):
-            dd.read_parquet(
-                tmpdir, engine=engine, dataset={"require_extension": False}
-            ).compute()
+    # Should Fail (for not capturing the _SUCCESS and crc files)
+    with pytest.raises((OSError, pa.lib.ArrowInvalid)):
+        dd.read_parquet(
+            tmpdir, engine=engine, dataset={"require_extension": False}
+        ).compute()
 
-        # Should Fail (for filtering out all files)
-        # (Related to: https://github.com/dask/dask/issues/8349)
-        with pytest.raises(ValueError):
-            dd.read_parquet(
-                tmpdir, engine=engine, dataset={"require_extension": ".foo"}
-            ).compute()
+    # Should Fail (for filtering out all files)
+    # (Related to: https://github.com/dask/dask/issues/8349)
+    with pytest.raises(ValueError):
+        dd.read_parquet(
+            tmpdir, engine=engine, dataset={"require_extension": ".foo"}
+        ).compute()
 
 
 def test_unsupported_extension_file(tmpdir, engine):
@@ -3873,7 +3687,7 @@ def test_unsupported_extension_file(tmpdir, engine):
     # (See: https://github.com/dask/dask/issues/8349)
     fn = os.path.join(str(tmpdir), "multi.foo")
     df0 = pd.DataFrame({"a": range(10)})
-    df0.to_parquet(fn, engine=engine.split("-")[0])
+    df0.to_parquet(fn, engine=engine)
     assert_eq(df0, dd.read_parquet(fn, engine=engine, index=False))