Repository: dask/dask

Issue:
Make use of Arrow Dataset API for dask.dataframe parquet reader?
**Long background**

<details>

In the Apache Arrow C++ project, we have been working the last moths on a Dataset API (original design document: https://docs.google.com/document/d/1bVhzifD38qDypnSjtf8exvpP3sSB5x_Kw9m-n66FB2c/edit, current python docs: https://arrow.apache.org/docs/python/dataset.html). 
A first iteration with python bindings is available in pyarrow 0.17 (but it's not yet considered stable). 

The dataset API has the goal to handle (scan/materialize) data sources larger than memory, and specifically provide:

- a unified interface for different sources (different file systems (local, cloud), different file formats (Parquet, CSV, JSON, feather, ..), and potentially also other sources like odbc connection, but this is not yet implemented)
- discovery of sources (crawling directories, handle directory-based partitioned datasets, basic schema normalization, ..)
- optimized reading with pedicate pushdown (filtering rows), projection (selecting columns), parallel reading or ability to manage scan tasks yourself (eg with dask)

Think: the current pyarrow.parquet.ParquetDataset functionality, but then not specific to parquet (currently also Feather ands CSV are supported), not tied to python (so for example the R bindings of arrow can also use this), and with more features (better schema normalization, more partitioning schemes, predicate pushdown, etc).

A small illustrative example of how it can look like in python using the new generic API:

```python
import pyarrow.dataset as ds
dataset = ds.dataset("path/to/partitioned/dataset",
                     format="parquet", partitioning="hive", filesystem=..)
table = dataset.to_table(columns=['col1', 'col2'],
                         filters=(ds.field('key2') == 1) & (ds.field('key2'== 2))
```

In addition, we also support using it from the existing pyarrow.parquet API with a keyword:

```python
import pyarrow.parquet as pq
pq.ParquetDataset("path/to/partitioned/dataset", use_legacy_dataset=False)
```

This `ParquetDataset` "shim", however, does not support the full existing API. So for example, it does not support the `.pieces` and ParquetDatasetPiece etc, which are specifically APIs that dask is using.

Long term, we would like to have this new Datasets implementation replace the python implementation of `pyarrow.parquet.ParquetDataset` (the basic `pyarrow.parquet.read/write_table` and `pyarrow.parquet.ParquetFile` are normally there to stay, but the Python ParquetDataset implementation hopefully not, long term, as it is duplicating the new Datasets implementation). 

</details>

Given that `pyarrow.ParquetDataset` eventually might go away (in its current form, we probably want to keep something similar, possibly with the same name, but probably not with an exactly identical "pieces" API), such a change would certainly have an impact on dask, given how dask currently makes use of the python ParquetDataset APIs. 

---

**Actual discussion items**

Long term, I think there are basically two options for dask: 1) take over the maintenance of the python implementation of ParquetDataset (it's also not that much, basically 800 lines of python code), or 2) rewrite dask's `read_parquet` arrow engine to use the new datasets API.

And, obviously, we (pyarrow) would love that dask can use the Arrow Datasets API, i.e. option 2 (because we are building to have it being used, and because we also think it will give a better implementation than the current python one). 
To that end, I have been experimenting with a **POC of a dask.dataframe reader** that uses the Arrow Datasets API under the hood. I opened https://github.com/dask/dask/pull/6172 with this branch for illustration purposes. And here is a demo notebook that uses it on a subset of the NYC Taxi dataset: https://gist.github.com/jorisvandenbossche/db8822b59b3641f12785aff0757e1e11 
To try it out locally, it requires pyarrow 0.17 (just released) or pyarrow master.

<sub> I presented this at the [Dask Summit](https://blog.dask.org/2020/04/28/dask-summit) as well ([slides](https://docs.google.com/presentation/d/1dBTFoe816ifUOX39LtduUq_wD5qOljpOOLTsC9sd710/edit#slide=id.g70124570a5_0_20)). In the meantime, some of the remaining issues have been solved (eg serializability to use with distributed, and the ability to map a single dask task to a single row group instead of full parquet file).
<sub>

Some general notes about this POC:

- The basics seem to be working (reading a partitioned dataset, selecting columns, filtering, etc). But it's certainly not yet a fully complete replacement of the current `read_parquet` (e.g. I didn't implement any inferring of potential index columns based on statistics).
- It does not yet support reading the manifest from `_metadata` / `_common_metadata` files, but this is actively being worked on ([ARROW-8062](https://issues.apache.org/jira/browse/ARROW-8062))
- It has several new features compared to the current `read_parquet`. I think the most relevant ones for dask are:
  - Support for a "/2019/01/15/" directory layout in addition to "/year=2019/month=01/day=15/"-style
  - Basic schema normalization (eg it allows a certain column to not be present, or to be of null type, which is something you can run into right now with dask)
  - Full row filtering: it does now row group filtering (which dask already had, but pyarrow not yet before), but in addition also does an actual pass over the data batch by batch (which gives less memory consumption / less data to convert to pandas dataframes)
- The very simple timings I did in the linked notebook indicate that it is faster than the current `read_parquet`. But I didn't really look into this, so don't know how much it is worth. I assume it is due to reduced python overhead. Because in the end, it's the same code that is doing the actual reading of the parquet files (and depending on the use case, this will probably dominate in real settings).
- It should allow eliminating some python code from dask (since pyarrow now takes care of more things such as row group filtering). But, I didn't yet think about how this could fit into the Engine structure / python code that is needed for the fastparquet engine anyway.
- Currently it needs a pyarrow filesystem, and doesn't work with fsspec-compatible filesystems (but this is probably worth a separate discussion)

So while we think the basics are there to make it usable in a dask context, there are probably use cases / corner cases / requirements that we didn't think of. 
So what would be useful for Arrow is to get feedback on this, so we can further figure out the requirements that dask has, so that hopefully we can implement those in the next pyarrow release and stabilize the API, and dask can actually start using this. 









Discussion:
cc @rjzamora @martindurant 
> take over the maintenance of the python implementation of ParquetDataset

I don't think we would want to maintain the old implementation.

It'd be nice to provide this as an `engine` to `read_parquet` to get a sense for what percent of the tests pass. You mention `_metadata` / `_common_metadata` files, which are being worked on. Is there anything we use in Dask that there *won't* be a generic Datasets API for?
> It'd be nice to provide this as an engine to read_parquet to get a sense for what percent of the tests pass. 

Yes, for now I wrote it as a separate `read_arrow_dataset` (as I wanted to start fresh for an initial version and not from the existing Engine class), but plugging it into the existing ``read_parquet`` is the logical next step. Being able to run the dask tests with it will certainly already bring up some issues / make it clearer which things are not yet supported. 

> Is there anything we use in Dask that there won't be a generic Datasets API for?

What I am currently aware of myself:

- pyarrow does not yet directly expose the Parquet metadata / statistics in the new API. Dask won't need that anymore for rowgroup filtering (as that is handled by pyarrow now), but AFAIK dask also uses this to infer the index column / determine the `divisions` of the index column . 
  (exposing the metadata on the ParquetFileFragments shouldn't be too hard though, and this might already be done in the PR adding `_metadata` support)


Personally (@rjzamora might disagree) I think the most valuable next step would be to try to integrate the new Datasets API into the `ArrowEngine` class, probably as a new `ArrowDatasetEngine` class for testing. That would highlight the pieces of `pyarrow.parquet` that we were using directly, for which there isn't a direct analog in the datasets API.

Once we have a sense for what is missing, we'll be able to better decide who should implement and maintain the wrappers we need (Dask or Arrow).
Thanks for working on #6172  and for compiling this issue @jorisvandenbossche !

I agree with @TomAugspurger - We will proabbly want to avoid taking responsibility for `pyarrow.ParquetDataset` maintenance.  Rather, it would be great to integrate a fully-functional `ArrowDatasetEngine` (to eventually deprecate/replace `ArrowEngine`).

It is also worth mentioning that the parquet reader in dask_cudf is inheriting from `ArrowEngine`.  So, I personally have vested interest in help making its replacement as "complete" and efficient as possible.  For example, it is critical to have `"_metadata"`support for large-scale workflows.  It would also be nice to avoid the need to manually split files into row-group pieces within dask.  Much of the python code in the current `ArrowEngine` implementation is to handle the fact that `ParquetDataset` does not support the `split_row_groups` argument. 
I am not sure it is that easy / possible to let it fit in into the existing (Arrow)Engine, as the Engine classes are quite ingrained with the current way things are implemented (it expects the `parts` in a specific format, the `ParquetSubgraph` is slightly different in my POC, as I need to pass different things to the partition reader function, I currently don't return statistics as row group filtering is handled by arrow, ...) 
But I will need to take a closer look to get a better idea (can do that tomorrow).

(now, for being able to run tests, I can also plug it in directly into `read_parquet`, to have that call my POC `read_arrow_dataset`, without using the Engine infrastructure. That might be easier to get, short term, some feedback from running the tests)

> It is also worth mentioning that the parquet reader in dask_cudf is inheriting from ArrowEngine. So, I personally have vested interest in help making its replacement as "complete" and efficient as possible. 

@rjzamora Yes, and that's very important. So a basic requirement from dask will be that any replacement based on Arrow Datasets should allow dask_cudf to continue re-using this infrastructure. 
But: for dask_cudf, does it need to be exactly the ArrowEngine interface? For sure, that would be the easiest, as then dask_cudf doesn't need to update (much). But in the end, what you need in dask_cudf basically is the path (+ filesystem) and optionally a row group index for each partition, in order to use cudf's parquet file / row group reader inside `read_partition` ?

> For example, it is critical to have `"_metadata"` support for large-scale workflows.

Yes, and that's being worked upon (see link in the top post). So it's certainly planned to add support for this on the Arrow side, just not yet in the current POC.

>  It would also be nice to avoid the need to manually split files into row-group pieces within dask.

As mentioned above, this is now implemented already in pyarrow. The POC PR has a `split_row_groups` keyword, that allows pointing each dask partition to a row group of a parquet file instead of to a full parquet file. So I *think* that should be covered, unless I am misunderstanding how dask uses this. 
(note this was not yet the case when we spoke in person at the workshop, but based on your feedback, we have implemented that since then in `pyarrow`)

> I am not sure it is that easy / possible to let it fit in into the existing (Arrow)Engine, as the Engine classes are quite ingrained with the current way things are implemented (it expects the parts in a specific format, the ParquetSubgraph is slightly different in my POC, as I need to pass different things to the partition reader function, I currently don't return statistics as row group filtering is handled by arrow, ...)

Agreed - Although the original read_parquet logic was intended to be very flexible and modular, it can certainly be tricky to fit a new engine into the mold.  Since there are many cases where the curent reader is having performance issues (splitting and aggregating row-groups and processing metadata/statistics) I am very open to breaking/improving this mold.  As my time opens up a bit in the coming weeks, I will be happy to help with this process.

> (now, for being able to run tests, I can also plug it in directly into read_parquet, to have that call my POC read_arrow_dataset, without using the Engine infrastructure. That might be easier to get, short term, some feedback from running the tests)

Yeah - You are probably right that the easiest next step is to see how much functionality can be implemented without worrying about the constraints of the `Engine` framework.

> But: for dask_cudf, does it need to be exactly the ArrowEngine interface? For sure, that would be the easiest, as then dask_cudf doesn't need to update (much). But in the end, what you need in dask_cudf basically is the path (+ filesystem) and optionally a row group index for each partition, in order to use cudf's parquet file / row group reader inside read_partition ?

I am more than happy to update dask_cudf to use soemthing other than `ArrowEngine`.  We are actually very motivated to improve the current performance of our dask_cudf parquet reader now that cudf.read_parquet offers functionality/optimizations that `dask.dataframe`'s read_parquet cannot leverage.  For example, the cudf reader can be given a row-group index as well as a row-group count to read a collection of row-groups very efficiently.  The dask.dataframe code still forces a loop over the row grops in this case.   With this in mind, what we really want in dask_cudf is to get `(file_path, row_group_offset, row_group_count)` for each output partition.

> As mentioned above, this is now implemented already in pyarrow. The POC PR has a split_row_groups keyword, that allows pointing each dask partition to a row group of a parquet file instead of to a full parquet file. So I think that should be covered, unless I am misunderstanding how dask uses this.

Nice! In that case, the `"_metadata"` handling (and possibly fsspec filesystem support) is likely the only critical blocker here for dask_cudf.
About fsspec specifically: I notice that the implementation here specifically uses the arrow local filesystem. I''m not certain this code is ever called, but I regard it as a regression, since all the fsspec implementations are pyarrow-compatible (and actually use python's builtin/os functions for most things). If you wanted to add a memory-map option to fsspec LocalFileSystem's open, that would be fine; but, again, I'm not certain that in the dask setting this is a useful thing to do.
> I notice that the implementation here specifically uses the arrow local filesystem.

@martindurant That's just for the proof-of-concept (and the memory map was actually to test feather files, not parquet, so not relevant here). 

But yes, as I noted in the top post, the POC currently requires a pyarrow filesystem, and doesn't work with an fsspec-compatible filesystem. This is for sure something that needs to be addressed, but as mentioned there, it's a big enough topic worth a separate discussion (and already enough other things to discuss here ;)). 
(I have been planning to start a discussion about it (maybe in the fsspec repo), since there have been some changes related to filesystems on the pyarrow side, but still need to get to it)

> the POC currently requires a pyarrow filesystem, and doesn't work with an fsspec-compatible filesystem

All fsspec filesystems should be compatible with pyarrow - please let me know if there's a problem/
```
In [17]: import fsspec

In [18]: fs = fsspec.filesystem('file')

In [19]: import pyarrow

In [20]: isinstance(fs, pyarrow.FileSystem)
Out[20]: True
```
> All fsspec filesystems should be compatible with pyarrow - please let me know if there's a problem/

The "old" ones, yes, but we have been developing new filesystems in pyarrow (which this Dataset API uses), and for those we don't yet have such an automatic compatibility layer (sorry for the confusion here, my wording "pyarrow filesystem" was not clear about that). 
But again, let's please leave this for another issue (there are already more than enough parquet-specific behaviours to discuss here), I will open another issue with more details / options to discuss.
> we have been developing new filesystems in pyarrow

I'm pretty sad to hear it
> I'm pretty sad to hear it

Well, we are not, because there are good reasons for it (amongst other things it enables this improved Datasets API). But it's not because we implemented new ones, that we don't care about preserving compatibility in some way (as I will try to explain in a separate issue).
Intermediate report from trying to run the dask parquet tests. I made a variant of my POC but plugging it in `read_parquet` with a ArrowEngine subclass. See last commit in #6172. 
Since the signatures of the methods on my subclass are different, I needed some hacks in `read_parquet` to pass the correct ones (which is the reason I said before it might not nicely fit into the existing Engine structure). Now, temporarily for testing purposes, this seems fine though. 

I am halfway through the tests, and encountered those issues:

- The `divisions` don't yet get computed (when statistics would be available). This of course fails many tests (so for now I am adding `check_divisions=False` everywhere to exercise the rest of the tests). 
  We don't need the statistics anymore for row group filtering (which is the reason for now I didn't bother implementing this), but I assume setting the `divisions` is also quite important? 
  So on the pyarrow side, we need to figure out the best way to expose the parquet FileMetadata. If there is a `_metadata` file available (being added in [ARROW-8062](https://issues.apache.org/jira/browse/ARROW-8062)), we will certainly make this available to the user as well. But it might be good to have access to it on the "ParquetFileFragment" as well, even if this means reading it in from each data file in the partitioned dataset (which is what dask now does when forcing this with `gather_statistics=True` when a `_metadata` is not present, I think?). 
  I opened https://issues.apache.org/jira/browse/ARROW-8733 for this.

- The new Datasets API returns the partition columns as "plain" columns (eg ints or strings), and not as categorical columns. So this is also a change compared to the current behaviour in dask / ParquetDataset. I don't know if we want to change the default behaviour for this, but at least providing the option to get dictionary encoded columns would be good to have on the pyarrow side. There is already an issue for this: https://issues.apache.org/jira/browse/ARROW-8647

- Reading only a partition columns apparently doesn't work yet -> reported as https://issues.apache.org/jira/browse/ARROW-8729

- Filtering out of empty partitions (or empty parquet files) doesn't happen yet: this might be something that can be solved in the implementation of the new engine on the dask side, but still need to figure out how this works for the other engines.
> For example, the cudf reader can be given a row-group index as well as a row-group count to read a collection of row-groups very efficiently. The dask.dataframe code still forces a loop over the row grops in this case. With this in mind, what we really want in dask_cudf is to get (file_path, row_group_offset, row_group_count) for each output partition.

@rjzamora currently the ParquetFileFragment implementation supports a set of indices (`vector<int>` on the C++ side). That should be fine as well? 
> We don't need the statistics anymore for row group filtering (which is the reason for now I didn't bother implementing this), but I assume setting the divisions is also quite important?

Right - Much of dask.dataframe relies on the index divisions, and the read_parquet logic expects to contruct these divisions from the statistics before actually reading in the data.  With that said, there are certainly dask.dataframe users who explicitly set `gather_statistics=False` (loosing/ignoring the divisions) just to avoid this overhead.  In the future, I would love to improve the performance of the parquet reader in cases where the user doesn't need divisions.  What this means for you:  **We do need to be able to gather statistics in dask and construct the divisions and choose the index.  However, I think it is fine if we implement `gather_statistics=False` (no divisions) to be more efficient (and still allow filtering during the read).** 

> If there is a _metadata file available (being added in ARROW-8062), we will certainly make this available to the user as well. But it might be good to have access to it on the "ParquetFileFragment" as well, even if this means reading it in from each data file in the partitioned dataset (which is what dask now does when forcing this with gather_statistics=True when a _metadata is not present, I think?).

Yes, you have this right - We want to be able to use the pyarrow API to access the metadata in both/either of these ways.

> @rjzamora currently the ParquetFileFragment implementation supports a set of indices (vector<int> on the C++ side). That should be fine as well?

If I understand correctly, then that is perfect :)
> However, I think it is fine if we implement gather_statistics=False (no divisions) to be more efficient (and still allow filtering during the read).

Yes, and so this is a feature with this new API that row group filtering will work even when the full metadata are not available or not gathered from the start (since we will filter the row groups when actually reading the files)
Note that an important test case here will be the ability to load from GCS and azure storage.
> Note that an important test case here will be the ability to load from GCS and azure storage.

I opened an issue specifically about the filesystem issue here: https://github.com/intake/filesystem_spec/issues/295
Small update here: I have been further working on the branch (https://github.com/dask/dask/pull/6172, I still need to push the latest version), which uncovered a few more issues in the new datasets implementation which are being fixed in pyarrow. 
The filesystem compatibility questions should also be solved by https://github.com/apache/arrow/pull/7395

The main issue I am struggling with, though, is the divisions / index inference. 
Quoting from @rjzamora:

> we will still want direct access to actual min/max statistics in python so that we to construct divisions. Is this currently possible? If not, is it a planned feature?

So the problem is that in the new implementation, dask doesn't need to access the parquet file metadata anymore to do any filtering, as this is handled by pyarrow.dataset. 
However, to support the current logic of finding the divisions/index, dask would still need to access and process all statistics:

- this means that there is basically no code simplification possible on the long term, as all this code dealing with statistics needs to stay
- it might actually become more complex, because the parquet file metadata won't be directly (by dask) be used to create the partitions, but would still need to process those for the statistics. But in practice, those also need to be filtered to ensure the divisions/index is determined for the actual, filtered partitions. (which is probably not a good idea to have those two filtering logics, one in pyarrow and one in dask, because if there is a mismatch between both, that might give problems)

There is an issue on the Arrow side about exposing the parquet FileMetadata / statistics on the fragments: https://issues.apache.org/jira/browse/ARROW-8733 
Some input on this would be very welcome. 
For example: what does dask need from this? I suppose only the statistics? (and not anything else from the FileMetadata)

But it seems that it will be important for dask that Arrow provides a way to get access to the statistics of a set of filtered fragments, even when the dataset was initialized from a `_metadata` object (and in this case also without accessing the actual data files, but getting by getting those statistics from the `_metadata` file).



Thanks for the summary @jorisvandenbossche !

I agree that using a combination of dask and pyarrow for filtering and statistics parsing can become a confusing mess.  I am thinking that we will want to do a bit of reorganizing in `read_parquet` to allow the statistics processing to be skiped altogether on the dask side if (1) the fragments have already been filtered, (2) the index has been detected/selected, and (3) the divisions have already been calculated by the pyarrow engine (during something like the usual `read_metadata` step).  For example, it seems reasonable for the pyarrow engine to filter the row-group fragments and then calculate the divisions from the `min`/`max` statistics of the final fragments (for the known/detected index column) within `read_metadata`.

The challenge here is that the engine will need to be able to select the index column using pandas metadata (if it isn't already provided) and will need to find the `min`/`max` statistics for each final (filltered) fragment.

I may be missing other challenges as well :)

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -1,13 +1,15 @@
 from functools import partial
 from collections import defaultdict
+from datetime import datetime
 import json
 import warnings
 from distutils.version import LooseVersion
 
+import numpy as np
 import pandas as pd
 import pyarrow as pa
 import pyarrow.parquet as pq
-from ....utils import getargspec
+from ....utils import getargspec, natural_sort_key
 from ..utils import _get_pyarrow_dtypes, _meta_from_dtypes
 from ...utils import clear_known_categories
 from ....core import flatten
@@ -20,15 +22,27 @@
     _analyze_paths,
 )
 
+
+# Check PyArrow version for feature support
 preserve_ind_supported = pa.__version__ >= LooseVersion("0.15.0")
+read_row_groups_supported = preserve_ind_supported
+if pa.__version__ >= LooseVersion("1.0.0"):
+    from pyarrow import dataset as pa_ds
+else:
+    pa_ds = None
 schema_field_supported = pa.__version__ >= LooseVersion("0.15.0")
 
 #
-#  Private Helper Functions
+#  Helper Utilities
 #
 
 
 def _append_row_groups(metadata, md):
+    """Append row-group metadata and include a helpful
+    error message if an inconsistent schema is detected.
+
+    Used by `ArrowDatasetEngine` and `ArrowLegacyEngine`.
+    """
     try:
         metadata.append_row_groups(md)
     except RuntimeError as err:
@@ -51,6 +65,9 @@ def _write_partitioned(
     Logic copied from pyarrow.parquet.
     (arrow/python/pyarrow/parquet.py::write_to_dataset)
 
+    Used by `ArrowDatasetEngine` (and by `ArrowLegacyEngine`,
+    through inherited `write_partition` method).
+
     TODO: Remove this in favor of pyarrow's `write_to_dataset`
           once ARROW-8244 is addressed.
     """
@@ -98,6 +115,12 @@ def _write_partitioned(
 
 
 def _index_in_schema(index, schema):
+    """Simple utility to check if all `index` columns are included
+    in the known `schema`.
+
+    Used by `ArrowDatasetEngine` (and by `ArrowLegacyEngine`,
+    through inherited `write_partition` method).
+    """
     if index and schema is not None:
         # Make sure all index columns are in user-defined schema
         return len(set(index).intersection(schema.names)) == len(index)
@@ -107,528 +130,299 @@ def _index_in_schema(index, schema):
         return False  # No index to check
 
 
-def _get_dataset_object(paths, fs, filters, dataset_kwargs):
-    """Generate a ParquetDataset object"""
-    kwargs = dataset_kwargs.copy()
-    if "validate_schema" not in kwargs:
-        kwargs["validate_schema"] = False
-    if len(paths) > 1:
-        # This is a list of files
-        base, fns = _analyze_paths(paths, fs)
-        proxy_metadata = None
-        if "_metadata" in fns:
-            # We have a _metadata file. PyArrow cannot handle
-            #  "_metadata" when `paths` is a list. So, we shuld
-            # open "_metadata" separately.
-            paths.remove(fs.sep.join([base, "_metadata"]))
-            fns.remove("_metadata")
-            with fs.open(fs.sep.join([base, "_metadata"]), mode="rb") as fil:
-                proxy_metadata = pq.ParquetFile(fil).metadata
-        # Create our dataset from the list of data files.
-        # Note #1: that this will not parse all the files (yet)
-        # Note #2: Cannot pass filters for legacy pyarrow API (see issue#6512).
-        #          We can handle partitions + filtering for list input after
-        #          adopting new pyarrow.dataset API.
-        dataset = pq.ParquetDataset(paths, filesystem=fs, **kwargs)
-        if proxy_metadata:
-            dataset.metadata = proxy_metadata
-    elif fs.isdir(paths[0]):
-        # This is a directory.  We can let pyarrow do its thing.
-        # Note: In the future, it may be best to avoid listing the
-        #       directory if we can get away with checking for the
-        #       existence of _metadata.  Listing may be much more
-        #       expensive in storage systems like S3.
-        allpaths = fs.glob(paths[0] + fs.sep + "*")
-        base, fns = _analyze_paths(allpaths, fs)
-        dataset = pq.ParquetDataset(paths[0], filesystem=fs, filters=filters, **kwargs)
-    else:
-        # This is a single file.  No danger in gathering statistics
-        # and/or splitting row-groups without a "_metadata" file
-        base = paths[0]
-        fns = [None]
-        dataset = pq.ParquetDataset(paths[0], filesystem=fs, **kwargs)
-
-    return dataset, base, fns
-
-
-def _gather_metadata(
-    paths, fs, split_row_groups, gather_statistics, filters, dataset_kwargs
-):
-    """Gather parquet metadata into a single data structure.
+class PartitionObj:
+    """Simple object to provide a `name` and `keys` attribute
+    for a single partition column. `ArrowDatasetEngine` will use
+    a list of these objects to "duck type" a `ParquetPartitions`
+    object (used in `ArrowLegacyEngine`). The larger purpose of this
+    class is to allow the same `read_partition` definition to handle
+    both Engine instances.
 
-    Use _metadata or aggregate footer metadata into a single
-    object.  Also, collect other information necessary for
-    parquet-to-ddf mapping (e.g. schema, partition_info).
+    Used by `ArrowDatasetEngine` only.
     """
 
-    # Step 1: Create a ParquetDataset object
-    dataset, base, fns = _get_dataset_object(paths, fs, filters, dataset_kwargs)
-    if fns == [None]:
-        # This is a single file. No danger in gathering statistics
-        # and/or splitting row-groups without a "_metadata" file
-        if gather_statistics is None:
-            gather_statistics = True
-        if split_row_groups is None:
-            split_row_groups = True
-
-    # Step 2: Construct necessary (parquet) partitioning information
-    partition_info = {"partitions": None, "partition_keys": {}, "partition_names": []}
-    fn_partitioned = False
-    if dataset.partitions is not None:
-        fn_partitioned = True
-        partition_info["partition_names"] = [
-            n for n in dataset.partitions.partition_names if n is not None
-        ]
-        partition_info["partitions"] = dataset.partitions
-        for piece in dataset.pieces:
-            partition_info["partition_keys"][piece.path] = piece.partition_keys
-
-    # Step 3: Construct a single `metadata` object. We can
-    #         directly use dataset.metadata if it is available.
-    #         Otherwise, if `gather_statistics` or `split_row_groups`,
-    #         we need to gether the footer metadata manually
-    metadata = None
-    if dataset.metadata:
-        # We have a _metadata file.
-        # PyArrow already did the work for us
-        schema = dataset.metadata.schema.to_arrow_schema()
-        if gather_statistics is None:
-            gather_statistics = True
-        if split_row_groups is None:
-            split_row_groups = True
-        return (
-            schema,
-            dataset.metadata,
-            base,
-            partition_info,
-            split_row_groups,
-            gather_statistics,
-        )
-    else:
-        # No _metadata file.
-        # May need to collect footer metadata manually
-        if dataset.schema is not None:
-            schema = dataset.schema.to_arrow_schema()
-        else:
-            schema = None
-        if gather_statistics is None:
-            gather_statistics = False
-        if split_row_groups is None:
-            split_row_groups = False
-        metadata = None
-        if not (split_row_groups or gather_statistics):
-            # Don't need to construct real metadata if
-            # we are not gathering statistics or splitting
-            # by row-group
-            metadata = [p.path for p in dataset.pieces]
-            if schema is None:
-                schema = dataset.pieces[0].get_metadata().schema.to_arrow_schema()
-            return (
-                schema,
-                metadata,
-                base,
-                partition_info,
-                split_row_groups,
-                gather_statistics,
-            )
-        # We have not detected a _metadata file, and the user has specified
-        # that they want to split by row-group and/or gather statistics.
-        # This is the only case where we MUST scan all files to collect
-        # metadata.
-        for piece, fn in zip(dataset.pieces, fns):
-            md = piece.get_metadata()
-            if schema is None:
-                schema = md.schema.to_arrow_schema()
-            if fn_partitioned:
-                md.set_file_path(piece.path.replace(base + fs.sep, ""))
-            elif fn:
-                md.set_file_path(fn)
-            if metadata:
-                _append_row_groups(metadata, md)
-            else:
-                metadata = md
-        return (
-            schema,
-            metadata,
-            base,
-            partition_info,
-            split_row_groups,
-            gather_statistics,
-        )
-
-
-def _generate_dd_meta(schema, index, categories, partition_info):
-    partition_obj = partition_info["partitions"]
-    partitions = partition_info["partition_names"]
-    columns = None
-
-    has_pandas_metadata = schema.metadata is not None and b"pandas" in schema.metadata
-
-    if has_pandas_metadata:
-        pandas_metadata = json.loads(schema.metadata[b"pandas"].decode("utf8"))
-        (
-            index_names,
-            column_names,
-            storage_name_mapping,
-            column_index_names,
-        ) = _parse_pandas_metadata(pandas_metadata)
-        if categories is None:
-            categories = []
-            for col in pandas_metadata["columns"]:
-                if (col["pandas_type"] == "categorical") and (
-                    col["name"] not in categories
-                ):
-                    categories.append(col["name"])
-    else:
-        # No pandas metadata implies no index, unless selected by the user
-        index_names = []
-        column_names = schema.names
-        storage_name_mapping = {k: k for k in column_names}
-        column_index_names = [None]
-
-    if index is None and index_names:
-        index = index_names
-
-    if set(column_names).intersection(partitions):
-        raise ValueError(
-            "partition(s) should not exist in columns.\n"
-            "categories: {} | partitions: {}".format(column_names, partitions)
-        )
-
-    column_names, index_names = _normalize_index_columns(
-        columns, column_names + partitions, index, index_names
-    )
+    def __init__(self, name, keys):
+        self.name = name
+        self.keys = sorted(keys)
 
-    all_columns = index_names + column_names
 
-    # Check that categories are included in columns
-    if categories and not set(categories).intersection(all_columns):
-        raise ValueError(
-            "categories not in available columns.\n"
-            "categories: {} | columns: {}".format(categories, list(all_columns))
-        )
+def _get_all_partition_keys(ds):
+    """Collect all partition categories (without applying filters).
+    This is needed for proper categorical encoding in `read_partition`.
+    We also need to know the mapping between paths and partition
+    values (`pkeys`).
 
-    dtypes = _get_pyarrow_dtypes(schema, categories)
-    dtypes = {storage_name_mapping.get(k, k): v for k, v in dtypes.items()}
+    Used by `ArrowDatasetEngine` only.
+    """
+    categories = defaultdict(list)
+    pkeys = defaultdict(list)
+    for file_frag in ds.get_fragments():
+        keys = pa_ds._get_partition_keys(file_frag.partition_expression)
+        pkeys[file_frag.path] = keys
+        for k, v in keys.items():
+            if v not in categories[k]:
+                categories[k].append(v)
+    return categories, pkeys
 
-    index_cols = index or ()
-    meta = _meta_from_dtypes(all_columns, dtypes, index_cols, column_index_names)
-    if categories:
-        # Make sure all categories are set to "unknown".
-        # Cannot include index names in the `cols` argument.
-        meta = clear_known_categories(
-            meta, cols=[c for c in categories if c not in meta.index.names]
-        )
 
-    if partition_obj:
-        for partition in partition_obj:
-            if isinstance(index, list) and partition.name == index[0]:
-                # Index from directory structure
-                meta.index = pd.CategoricalIndex(
-                    categories=partition.keys, name=index[0]
-                )
-            elif partition.name == meta.index.name:
-                # Index created from a categorical column
-                meta.index = pd.CategoricalIndex(
-                    categories=partition.keys, name=meta.index.name
-                )
-            elif partition.name in meta.columns:
-                meta[partition.name] = pd.Series(
-                    pd.Categorical(categories=partition.keys, values=[]),
-                    index=meta.index,
-                )
+def _frag_subset(old_frag, row_groups):
+    """Create new fragment with row-group subset.
 
-    return meta, index_cols, categories, index
+    Used by `ArrowDatasetEngine` only.
+    """
+    return old_frag.format.make_fragment(
+        old_frag.path,
+        old_frag.filesystem,
+        old_frag.partition_expression,
+        row_groups=row_groups,
+    )
 
 
-def _aggregate_stats(
-    file_path, file_row_group_stats, file_row_group_column_stats, stat_col_indices
+def _collect_pyarrow_dataset_frags(
+    ds, filters, valid_paths, fs, split_row_groups, gather_statistics, read_from_paths
 ):
-    """Utility to aggregate the statistics for N row-groups
-    into a single dictionary.
-    """
-    if len(file_row_group_stats) < 1:
-        # Empty statistics
-        return {}
-    elif len(file_row_group_column_stats) == 0:
-        assert len(file_row_group_stats) == 1
-        return file_row_group_stats[0]
-    else:
-        # Note: It would be better to avoid df_rgs and df_cols
-        #       construction altogether. It makes it fast to aggregate
-        #       the statistics for many row groups, but isn't
-        #       worthwhile for a small number of row groups.
-        if len(file_row_group_stats) > 1:
-            df_rgs = pd.DataFrame(file_row_group_stats)
-            s = {
-                "file_path_0": file_path,
-                "num-rows": df_rgs["num-rows"].sum(),
-                "total_byte_size": df_rgs["total_byte_size"].sum(),
-                "columns": [],
-            }
-        else:
-            s = {
-                "file_path_0": file_path,
-                "num-rows": file_row_group_stats[0]["num-rows"],
-                "total_byte_size": file_row_group_stats[0]["total_byte_size"],
-                "columns": [],
-            }
+    """Collect all dataset fragments while applying filters.
 
-        df_cols = None
-        if len(file_row_group_column_stats) > 1:
-            df_cols = pd.DataFrame(file_row_group_column_stats)
-        for ind, name in enumerate(stat_col_indices):
-            i = ind * 3
-            if df_cols is None:
-                s["columns"].append(
-                    {
-                        "name": name,
-                        "min": file_row_group_column_stats[0][i],
-                        "max": file_row_group_column_stats[0][i + 1],
-                        "null_count": file_row_group_column_stats[0][i + 2],
-                    }
-                )
-            else:
-                s["columns"].append(
-                    {
-                        "name": name,
-                        "min": df_cols.iloc[:, i].min(),
-                        "max": df_cols.iloc[:, i + 1].max(),
-                        "null_count": df_cols.iloc[:, i + 2].sum(),
-                    }
-                )
-        return s
+    Used by `ArrowDatasetEngine` only.
+    """
 
+    if not (split_row_groups or filters or gather_statistics):
+        # Don't need to process real metadata if
+        # we are not gathering statistics, splitting
+        # by row-group, filtering, or dealing with partitions.
+        metadata = []
+        for i, frag in enumerate(ds.get_fragments()):
+            if i == 0:
+                if pa_ds._get_partition_keys(frag.partition_expression):
+                    break
+            metadata.append(frag.path)
+        if len(metadata):
+            partition_info = {
+                "partitions": [],
+                "partition_keys": {},
+                "partition_names": [],
+            }
+            return metadata, partition_info
+
+    # Get/transate filters
+    ds_filters = None
+    if filters is not None:
+        ds_filters = pq._filters_to_expression(filters)
+
+    # Get all partition keys (without filters) to populate partition_obj
+    partition_obj = []  # See `partition_info` description below
+    categories, pkeys = _get_all_partition_keys(ds)
+    partition_names = list(categories)
+    for name in partition_names:
+        partition_obj.append(PartitionObj(name, categories[name]))
+
+    # Split by row-groups and apply filters
+    partition_keys = {}  # See `partition_info` description below
+    metadata = []  # List of (fragment, row_group_info) tuples
+
+    # Start with sorted (by path) list of file-based fragments
+    file_frags = sorted(
+        [frag for frag in ds.get_fragments(ds_filters)],
+        key=lambda x: natural_sort_key(x.path),
+    )
 
-def _process_metadata(
-    metadata, single_rg_parts, gather_statistics, stat_col_indices, no_filters
-):
-    # Get the number of row groups per file
-    file_row_groups = defaultdict(list)
-    file_row_group_stats = defaultdict(list)
-    file_row_group_column_stats = defaultdict(list)
-    cmax_last = {}
-    for rg in range(metadata.num_row_groups):
-        row_group = metadata.row_group(rg)
-        fpath = row_group.column(0).file_path
-        if fpath is None:
-            raise ValueError(
-                "Global metadata structure is missing a file_path string. "
-                "If the dataset includes a _metadata file, that file may "
-                "have one or more missing file_path fields."
-            )
-        if file_row_groups[fpath]:
-            file_row_groups[fpath].append(file_row_groups[fpath][-1] + 1)
-        else:
-            file_row_groups[fpath].append(0)
-        if gather_statistics:
-            if single_rg_parts:
-                s = {
-                    "file_path_0": fpath,
-                    "num-rows": row_group.num_rows,
-                    "total_byte_size": row_group.total_byte_size,
-                    "columns": [],
-                }
-            else:
-                s = {
-                    "num-rows": row_group.num_rows,
-                    "total_byte_size": row_group.total_byte_size,
-                }
-            cstats = []
-            for name, i in stat_col_indices.items():
-                column = row_group.column(i)
-                if column.statistics:
-                    cmin = column.statistics.min
-                    cmax = column.statistics.max
-                    cnull = column.statistics.null_count
-                    last = cmax_last.get(name, None)
-                    if no_filters:
-                        # Only think about bailing if we don't need
-                        # stats for filtering
-                        if cmin is None or (last and cmin < last):
-                            # We are collecting statistics for divisions
-                            # only (no filters) - Column isn't sorted, or
-                            # we have an all-null partition, so lets bail.
-                            #
-                            # Note: This assumes ascending order.
-                            #
-                            gather_statistics = False
-                            file_row_group_stats = {}
-                            file_row_group_column_stats = {}
-                            break
+    # Loop over file fragments
+    for file_frag in file_frags:
+        # If valid_paths is not None, the user passed in a list
+        # of files containing a _metadata file.  Since we used
+        # the _metadata file to generate our dataset object , we need
+        # to ignore any file fragments that are not in the list.
+        if valid_paths and file_frag.path.split(fs.sep)[-1] not in valid_paths:
+            continue
+        # Store (filtered) partition keys
+        if pkeys:
+            partition_keys[file_frag.path] = [
+                (name, pkeys[file_frag.path].get(name, None))
+                for name in partition_names
+            ]
 
-                    if single_rg_parts:
-                        to_ts = column.statistics.logical_type.type == "TIMESTAMP"
-                        s["columns"].append(
-                            {
-                                "name": name,
-                                "min": cmin if not to_ts else pd.Timestamp(cmin),
-                                "max": cmax if not to_ts else pd.Timestamp(cmax),
-                                "null_count": cnull,
-                            }
+        # Append fragements to our "metadata" list
+        if ds_filters or (split_row_groups and not read_from_paths):
+            # If we have filters, we need to split the row groups to apply them.
+            # If any row-groups are filtered out, we convert the remaining row-groups
+            # to a NEW (filtered) fragment, and append the filtered fragment to our
+            # metadata.
+
+            # Collect list of filtered row-group fragments
+            filtered_row_group_frags = []
+            row_group_info = []
+            row_group_ids = []
+            for rg_frag in file_frag.split_by_row_group(ds_filters, schema=ds.schema):
+                filtered_row_group_frags.append(rg_frag)
+                row_group_info.append(rg_frag.row_groups[0])
+                row_group_ids.append(rg_frag.row_groups[0].id)
+
+            # Row group count before (`num_row_groups_i`) and
+            # after (`num_row_groups`) filtering
+            num_row_groups_f = len(filtered_row_group_frags)
+            num_row_groups_i = len(file_frag.row_groups)
+
+            if split_row_groups:
+                # Splitting by row-group.
+                k = int(split_row_groups)
+                if k == 1:
+                    # Each output partition corresponds to a single
+                    # row-group - The work is already done.
+                    metadata.extend([(f, None) for f in filtered_row_group_frags])
+                elif k >= num_row_groups_f:
+                    # Split is larger than the number of row-groups in the file.
+                    if num_row_groups_f < num_row_groups_i:
+                        # 1+ row-groups are filtered - Need new fragment.
+                        metadata.append(
+                            (_frag_subset(file_frag, row_group_ids), row_group_info)
                         )
                     else:
-                        cstats += [cmin, cmax, cnull]
-                    cmax_last[name] = cmax
+                        # Nothing was filtered - Use original fragment.
+                        metadata.append((file_frag, row_group_info))
                 else:
+                    # Splits are smaller than the number of row-groups in the file.
+                    # We will need to create multiple new fragments.
+                    for rg in range(0, num_row_groups_f, k):
+                        new_row_groups = [
+                            i for i in range(rg, rg + k) if i < num_row_groups_f
+                        ]
+                        if len(new_row_groups) == 1:
+                            # Avoid creating new fragment if we don't need to,
+                            # because it will require us to parse statistics twice.
+                            metadata.append(
+                                (filtered_row_group_frags[new_row_groups[0]], None)
+                            )
+                        else:
+                            metadata.append(
+                                (
+                                    _frag_subset(file_frag, new_row_groups),
+                                    [row_group_info[i] for i in new_row_groups],
+                                )
+                            )
+            elif num_row_groups_f < num_row_groups_i:
+                # 1+ row-groups are filtered - Need new fragment.
+                metadata.append(
+                    (_frag_subset(file_frag, row_group_ids), row_group_info)
+                )
+            else:
+                # Use original fragment
+                metadata.append((file_frag, row_group_info))
+        else:
+            # No filtering or splitting by row-group - Use original fragment.
+            metadata.append((file_frag, None))
+
+    # The final `metadata` object is a list of (fragment, row_group_info)
+    # tuples.  This is different from a `FileMetadata` object (used by the
+    # legacy code path), but it does contain much of the same information.
+    #
+    # The `partition_info` dict summarizes information needed to handle
+    # nested-directory (hive) partitioning.
+    #
+    #    - "partitions" : (list of PartitionObj) This is a list of simple
+    #          objects providing `name` and `keys` attributes for each
+    #          partition column. The list is designed to "duck type" a
+    #          `ParquetPartitions` object, so that the same code path can
+    #          be used for both legacy and pyarrow.dataset-based logic.
+    #    - "partition_keys" : (dict) The keys and values correspond to
+    #          file paths and partition values, respectively. The partition
+    #          values (or partition "keys") will be represented as a list
+    #          of tuples. E.g. `[("year", 2020), ("state", "CA")]`
+    #    - "partition_names" : (list)  This is a list containing the names
+    #          of partitioned columns.
+    partition_info = {
+        "partitions": partition_obj,
+        "partition_keys": partition_keys,
+        "partition_names": partition_names,
+    }
+    return metadata, partition_info
+
+
+def _get_pandas_metadata(schema):
+    """Get pandas-specific metadata from schema.
+
+    Used by `ArrowDatasetEngine` and `ArrowLegacyEngine`.
+    """
 
-                    if no_filters and column.num_values > 0:
-                        # We are collecting statistics for divisions
-                        # only (no filters) - Lets bail.
-                        gather_statistics = False
-                        file_row_group_stats = {}
-                        file_row_group_column_stats = {}
-                        break
+    has_pandas_metadata = schema.metadata is not None and b"pandas" in schema.metadata
+    if has_pandas_metadata:
+        return json.loads(schema.metadata[b"pandas"].decode("utf8"))
+    else:
+        return {}
 
-                    if single_rg_parts:
-                        s["columns"].append({"name": name})
-                    else:
-                        cstats += [None, None, None]
-            if gather_statistics:
-                file_row_group_stats[fpath].append(s)
-                if not single_rg_parts:
-                    file_row_group_column_stats[fpath].append(tuple(cstats))
 
+def _flatten_filters(filters):
+    """Flatten DNF-formatted filters (list of tuples)"""
     return (
-        file_row_groups,
-        file_row_group_stats,
-        file_row_group_column_stats,
-        gather_statistics,
+        set(flatten(tuple(flatten(filters, container=list)), container=tuple))
+        if filters
+        else []
     )
 
 
-def _construct_parts(
+def _read_table_from_path(
+    path,
     fs,
-    metadata,
+    row_groups,
+    columns,
     schema,
     filters,
-    index_cols,
-    data_path,
-    partition_info,
-    categories,
-    split_row_groups,
-    gather_statistics,
+    partitions,
+    partition_keys,
+    piece_to_arrow_func,
+    **kwargs,
 ):
-    """Construct ``parts`` for ddf construction
+    """Read arrow table from file path.
 
-    Use metadata (along with other data) to define a tuple
-    for each ddf partition.  Also gather statistics if
-    ``gather_statistics=True``, and other criteria is met.
+    Used in all cases by `ArrowLegacyEngine._read_table`.
+    Used by `ArrowDatasetEngine._read_table` when dataset is
+    not partitioned and no filters are specified (otherwise
+    fragments are converted directly into tables).
     """
+    if partition_keys or (not read_row_groups_supported and row_groups != [None]):
+        tables = []
+        for rg in row_groups:
+            piece = pq.ParquetDatasetPiece(
+                path,
+                row_group=rg,
+                partition_keys=partition_keys,
+                open_file_func=partial(fs.open, mode="rb"),
+            )
+            arrow_table = piece_to_arrow_func(piece, columns, partitions, **kwargs)
+            tables.append(arrow_table)
 
-    parts = []
-    stats = []
-
-    partition_keys = partition_info["partition_keys"]
-    partition_obj = partition_info["partitions"]
+        if len(row_groups) > 1:
+            # NOTE: Not covered by pytest
+            return pa.concat_tables(tables)
+        else:
+            return tables[0]
+    else:
+        with fs.open(path, mode="rb") as fil:
+            if row_groups == [None]:
+                return pq.ParquetFile(fil).read(
+                    columns=columns,
+                    use_threads=False,
+                    use_pandas_metadata=True,
+                )
+            else:
+                return pq.ParquetFile(fil).read_row_groups(
+                    row_groups,
+                    columns=columns,
+                    use_threads=False,
+                    use_pandas_metadata=True,
+                )
 
-    # Check if `metadata` is just a list of paths
-    # (not splitting by row-group or collecting statistics)
-    if isinstance(metadata, list) and isinstance(metadata[0], str):
-        for full_path in metadata:
-            part = {
-                "piece": (full_path, None, partition_keys.get(full_path, None)),
-                "kwargs": {"partitions": partition_obj, "categories": categories},
-            }
-            parts.append(part)
-        return parts, stats
 
-    # Determine which columns need statistics
-    flat_filters = (
-        set(flatten(tuple(flatten(filters, container=list)), container=tuple))
-        if filters
-        else []
-    )
-    stat_col_indices = {}
-    for i, name in enumerate(schema.names):
-        if name in index_cols or name in flat_filters:
-            stat_col_indices[name] = i
-    stat_cols = list(stat_col_indices.keys())
-    gather_statistics = gather_statistics and len(stat_cols) > 0
-
-    # Convert metadata into simple dictionary structures
-    (
-        file_row_groups,
-        file_row_group_stats,
-        file_row_group_column_stats,
-        gather_statistics,
-    ) = _process_metadata(
-        metadata,
-        int(split_row_groups) == 1,
-        gather_statistics,
-        stat_col_indices,
-        flat_filters == [],
-    )
+#
+#  ArrowDatasetEngine
+#
 
-    if split_row_groups:
-        # Create parts from each file,
-        # limiting the number of row_groups in each piece
-        split_row_groups = int(split_row_groups)
-        for filename, row_groups in file_row_groups.items():
-            row_group_count = len(row_groups)
-            for i in range(0, row_group_count, split_row_groups):
-                i_end = i + split_row_groups
-                rg_list = row_groups[i:i_end]
-                full_path = (
-                    fs.sep.join([data_path, filename])
-                    if filename != ""
-                    else data_path  # This is a single file
-                )
-                pkeys = partition_keys.get(full_path, None)
-                if partition_obj and pkeys is None:
-                    continue  # This partition was filtered
-                part = {
-                    "piece": (full_path, rg_list, pkeys),
-                    "kwargs": {
-                        "partitions": partition_obj,
-                        "categories": categories,
-                        "filters": filters,
-                        "schema": schema,
-                    },
-                }
-                parts.append(part)
-                if gather_statistics:
-                    stat = _aggregate_stats(
-                        filename,
-                        file_row_group_stats[filename][i:i_end],
-                        file_row_group_column_stats[filename][i:i_end],
-                        stat_col_indices,
-                    )
-                    stats.append(stat)
-    else:
-        for filename, row_groups in file_row_groups.items():
-            full_path = (
-                fs.sep.join([data_path, filename])
-                if filename != ""
-                else data_path  # This is a single file
-            )
-            pkeys = partition_keys.get(full_path, None)
-            if partition_obj and pkeys is None:
-                continue  # This partition was filtered
-            rgs = None
-            part = {
-                "piece": (full_path, rgs, pkeys),
-                "kwargs": {
-                    "partitions": partition_obj,
-                    "categories": categories,
-                    "filters": filters,
-                    "schema": schema,
-                },
-            }
-            parts.append(part)
-            if gather_statistics:
-                stat = _aggregate_stats(
-                    filename,
-                    file_row_group_stats[filename],
-                    file_row_group_column_stats[filename],
-                    stat_col_indices,
-                )
-                stats.append(stat)
 
-    return parts, stats
+class ArrowDatasetEngine(Engine):
 
+    #
+    # Public Class Methods
+    #
 
-class ArrowEngine(Engine):
     @classmethod
     def read_metadata(
         cls,
@@ -639,11 +433,15 @@ def read_metadata(
         gather_statistics=None,
         filters=None,
         split_row_groups=None,
+        read_from_paths=None,
         **kwargs,
     ):
-
-        # Check if we are using pyarrow.dataset API
-        dataset_kwargs = kwargs.get("dataset", {})
+        # Reading from fragments by default if we are filtering.
+        # Otherwise, we will read from (path, row-group) info
+        # to avoid passing large object in the task graph.
+        # (This does not affect `ArrowLegacyEngine` behavior)
+        if read_from_paths is None:
+            read_from_paths = filters is None
 
         # Gather necessary metadata information. This includes
         # the schema and (parquet) partitioning information.
@@ -656,43 +454,25 @@ def read_metadata(
             partition_info,
             split_row_groups,
             gather_statistics,
-        ) = _gather_metadata(
-            paths, fs, split_row_groups, gather_statistics, filters, dataset_kwargs
+        ) = cls._gather_metadata(
+            paths,
+            fs,
+            split_row_groups,
+            gather_statistics,
+            filters,
+            index,
+            read_from_paths,
+            kwargs.get("dataset", {}),
         )
 
         # Process metadata to define `meta` and `index_cols`
-        meta, index_cols, categories, index = _generate_dd_meta(
+        meta, index_cols, categories, index = cls._generate_dd_meta(
             schema, index, categories, partition_info
         )
 
-        # Cannot gather_statistics if our `metadata` is a list
-        # of paths, or if we are building a multiindex (for now).
-        # We also don't "need" to gather statistics if we don't
-        # want to apply any filters or calculate divisions
-        if (isinstance(metadata, list) and isinstance(metadata[0], str)) or len(
-            index_cols
-        ) > 1:
-            gather_statistics = False
-        elif filters is None and len(index_cols) == 0:
-            gather_statistics = False
-
-        # Make sure gather_statistics allows filtering
-        # (if filters are desired)
-        if filters:
-            # Filters may require us to gather statistics
-            if gather_statistics is False and partition_info["partition_names"]:
-                warnings.warn(
-                    "Filtering with gather_statistics=False. "
-                    "Only partition columns will be filtered correctly."
-                )
-            elif gather_statistics is False:
-                raise ValueError("Cannot apply filters with gather_statistics=False")
-            elif not gather_statistics:
-                gather_statistics = True
-
         # Finally, construct our list of `parts`
         # (and a corresponding list of statistics)
-        parts, stats = _construct_parts(
+        parts, stats = cls._construct_parts(
             fs,
             metadata,
             schema,
@@ -703,6 +483,7 @@ def read_metadata(
             categories,
             split_row_groups,
             gather_statistics,
+            read_from_paths,
         )
 
         return (meta, stats, parts, index)
@@ -720,6 +501,10 @@ def read_partition(
         schema=None,
         **kwargs,
     ):
+        """Read in a single output partition.
+
+        This method is also used by `ArrowLegacyEngine`.
+        """
         if isinstance(index, list):
             for level in index:
                 # unclear if we can use set ops here. I think the order matters.
@@ -730,44 +515,77 @@ def read_partition(
 
         # Ensure `columns` and `partitions` do not overlap
         columns_and_parts = columns.copy()
-        if columns_and_parts and partitions:
-            for part_name in partitions.partition_names:
-                if part_name in columns:
-                    columns.remove(part_name)
-                else:
-                    columns_and_parts.append(part_name)
-            columns = columns or None
+        if not isinstance(partitions, (list, tuple)):
+            if columns_and_parts and partitions:
+                for part_name in partitions.partition_names:
+                    if part_name in columns:
+                        columns.remove(part_name)
+                    else:
+                        columns_and_parts.append(part_name)
+                columns = columns or None
 
         if isinstance(piece, str):
             # `piece` is a file-path string
-            path = piece
+            path_or_frag = piece
             row_group = None
             partition_keys = None
         else:
             # `piece` contains (path, row_group, partition_keys)
-            (path, row_group, partition_keys) = piece
+            (path_or_frag, row_group, partition_keys) = piece
 
         if not isinstance(row_group, list):
             row_group = [row_group]
 
-        dfs = []
-        for rg in row_group:
-            piece = pq.ParquetDatasetPiece(
-                path,
-                row_group=rg,
-                partition_keys=partition_keys,
-                open_file_func=partial(fs.open, mode="rb"),
-            )
-            arrow_table = cls._parquet_piece_as_arrow(
-                piece, columns, partitions, **kwargs
-            )
-            df = cls._arrow_table_to_pandas(arrow_table, categories, **kwargs)
-
-            if len(row_group) > 1:
-                dfs.append(df)
+        # Read in arrow table and convert to pandas
+        arrow_table = cls._read_table(
+            path_or_frag,
+            fs,
+            row_group,
+            columns,
+            schema,
+            filters,
+            partitions,
+            partition_keys,
+            **kwargs,
+        )
 
-        if len(row_group) > 1:
-            df = pd.concat(dfs)
+        # For pyarrow.dataset api, if we did not read directly from
+        # fragments, we need to add the partitioned columns here.
+        if partitions and isinstance(partitions, list):
+            if not isinstance(path_or_frag, pa_ds.ParquetFileFragment):
+                keys_dict = {k: v for (k, v) in partition_keys}
+            else:
+                keys_dict = {}
+            for partition in partitions:
+                if partition.name not in arrow_table.schema.names:
+                    # We read from file paths, so the partition
+                    # columns are NOT in our table yet.
+                    cat = keys_dict.get(partition.name, None)
+                    cat_ind = np.full(
+                        len(arrow_table), partition.keys.index(cat), dtype="i4"
+                    )
+                    arr = pa.DictionaryArray.from_arrays(
+                        cat_ind, pa.array(partition.keys)
+                    )
+                    arrow_table = arrow_table.append_column(partition.name, arr)
+
+        df = cls._arrow_table_to_pandas(arrow_table, categories, **kwargs)
+
+        # For pyarrow.dataset api, need to convert partition columns
+        # to categorigal manually for integer types.
+        if partitions and isinstance(partitions, list):
+            for partition in partitions:
+                if df[partition.name].dtype != pd.Categorical:
+                    # We read directly from fragments, so the partition
+                    # columns are already in our dataframe.  We just
+                    # need to convert non-categorical types.
+                    df[partition.name] = pd.Series(
+                        pd.Categorical(
+                            categories=partition.keys,
+                            values=df[partition.name].values,
+                        ),
+                        index=df.index,
+                    )
 
         # Note that `to_pandas(ignore_metadata=False)` means
         # pyarrow will use the pandas metadata to set the index.
@@ -800,28 +618,6 @@ def read_partition(
             df = df.set_index(index)
         return df
 
-    @classmethod
-    def _arrow_table_to_pandas(
-        cls, arrow_table: pa.Table, categories, **kwargs
-    ) -> pd.DataFrame:
-        _kwargs = kwargs.get("arrow_to_pandas", {})
-        _kwargs.update({"use_threads": False, "ignore_metadata": False})
-
-        return arrow_table.to_pandas(categories=categories, **_kwargs)
-
-    @classmethod
-    def _parquet_piece_as_arrow(
-        cls, piece: pq.ParquetDatasetPiece, columns, partitions, **kwargs
-    ) -> pa.Table:
-        arrow_table = piece.read(
-            columns=columns,
-            partitions=partitions,
-            use_pandas_metadata=True,
-            use_threads=False,
-            **kwargs.get("read", {}),
-        )
-        return arrow_table
-
     @staticmethod
     def initialize_write(
         df,
@@ -1058,6 +854,1088 @@ def write_metadata(parts, fmd, fs, path, append=False, **kwargs):
             with fs.open(metadata_path, "wb") as fil:
                 _meta.write_metadata_file(fil)
 
+    #
+    # Private Class Methods
+    #
+
+    @classmethod
+    def _gather_metadata(
+        cls,
+        paths,
+        fs,
+        split_row_groups,
+        gather_statistics,
+        filters,
+        index,
+        read_from_paths,
+        dataset_kwargs,
+    ):
+        """pyarrow.dataset version of _gather_metadata
+        Use pyarrow.dataset API to collect list of row-group fragments.
+        Also, collect other information necessary for parquet-to-ddf
+        mapping (e.g. schema, partition_info).
+
+        This method is overridden in `ArrowLegacyEngine`.
+        """
+        # Use pyarrow.dataset API
+        ds = None
+        valid_paths = None  # Only used if `paths` is a list containing _metadata
+        default_partitioning = pa_ds.HivePartitioning.discover(
+            max_partition_dictionary_size=-1
+        )
+        if len(paths) == 1 and fs.isdir(paths[0]):
+
+            # Use _analyze_paths to avoid relative-path
+            # problems (see GH#5608)
+            base, fns = _analyze_paths(paths, fs)
+            paths = fs.sep.join([base, fns[0]])
+
+            meta_path = fs.sep.join([paths, "_metadata"])
+            if fs.exists(meta_path):
+                # Use _metadata file
+                ds = pa_ds.parquet_dataset(
+                    meta_path,
+                    filesystem=fs,
+                    partitioning=dataset_kwargs.get(
+                        "partitioning", default_partitioning
+                    ),
+                )
+                if gather_statistics is None:
+                    gather_statistics = True
+        elif len(paths) > 1:
+            base, fns = _analyze_paths(paths, fs)
+            meta_path = fs.sep.join([base, "_metadata"])
+            if "_metadata" in fns:
+                # Pyarrow cannot handle "_metadata" when `paths` is a list
+                # Use _metadata file
+                ds = pa_ds.parquet_dataset(
+                    meta_path,
+                    filesystem=fs,
+                    partitioning=dataset_kwargs.get(
+                        "partitioning", default_partitioning
+                    ),
+                )
+                if gather_statistics is None:
+                    gather_statistics = True
+
+                # Populate valid_paths, since the original path list
+                # must be used to filter the _metadata-based dataset
+                fns.remove("_metadata")
+                valid_paths = fns
+
+        if ds is None:
+            ds = pa_ds.dataset(
+                paths,
+                filesystem=fs,
+                format="parquet",
+                partitioning=dataset_kwargs.get(
+                    "partitioning", default_partitioning
+                ),  # Assume "hive" by default
+            )
+        schema = ds.schema
+        base = ""
+
+        # At this point, we know if `split_row_groups` should be
+        # set to `True` by default.  If the user has not specified
+        # this option, we will only collect statistics if there is
+        # a global "_metadata" file available, otherwise we will
+        # opt for `gather_statistics=False`. For `ArrowDatasetEngine`,
+        # statistics are only required to calculate divisions
+        # and/or aggregate row-groups using `chunksize` (not for
+        # filtering).
+        #
+        # By default, we will create an output partition for each
+        # row group in the dataset (`split_row_groups=True`).
+        # However, we will NOT split by row-group if
+        # `gather_statistics=False`, because this can be
+        # interpreted as an indication that metadata overhead should
+        # be avoided at all costs.
+        if gather_statistics is None:
+            gather_statistics = False
+        if split_row_groups is None:
+            if gather_statistics:
+                split_row_groups = True
+            else:
+                split_row_groups = False
+
+        # Generate list of (fragment, row_group_info) tuples
+        # and call it `metadata`
+        metadata, partition_info = _collect_pyarrow_dataset_frags(
+            ds,
+            filters,
+            valid_paths,
+            fs,
+            split_row_groups,
+            gather_statistics,
+            read_from_paths,
+        )
+
+        return (
+            schema,
+            metadata,
+            base,
+            partition_info,
+            split_row_groups,
+            gather_statistics,
+        )
+
+    @classmethod
+    def _generate_dd_meta(cls, schema, index, categories, partition_info):
+        """Use parquet metadata to construct DataFrame metadata.
+
+        This method is used by both `ArrowDatasetEngine`
+        and `ArrowLegacyEngine`.
+        """
+        partition_obj = partition_info["partitions"]
+        partitions = partition_info["partition_names"]
+        columns = None
+
+        pandas_metadata = _get_pandas_metadata(schema)
+        if pandas_metadata:
+            (
+                index_names,
+                column_names,
+                storage_name_mapping,
+                column_index_names,
+            ) = _parse_pandas_metadata(pandas_metadata)
+            if categories is None:
+                categories = []
+                for col in pandas_metadata["columns"]:
+                    if (col["pandas_type"] == "categorical") and (
+                        col["name"] not in categories
+                    ):
+                        categories.append(col["name"])
+        else:
+            # No pandas metadata implies no index, unless selected by the user
+            index_names = []
+            column_names = schema.names
+            storage_name_mapping = {k: k for k in column_names}
+            column_index_names = [None]
+
+        if index is None and index_names:
+            index = index_names
+
+        if set(column_names).intersection(partitions):
+            raise ValueError(
+                "partition(s) should not exist in columns.\n"
+                "categories: {} | partitions: {}".format(column_names, partitions)
+            )
+
+        column_names, index_names = _normalize_index_columns(
+            columns, column_names + partitions, index, index_names
+        )
+
+        all_columns = index_names + column_names
+
+        # Check that categories are included in columns
+        if categories and not set(categories).intersection(all_columns):
+            raise ValueError(
+                "categories not in available columns.\n"
+                "categories: {} | columns: {}".format(categories, list(all_columns))
+            )
+
+        dtypes = _get_pyarrow_dtypes(schema, categories)
+        dtypes = {storage_name_mapping.get(k, k): v for k, v in dtypes.items()}
+
+        index_cols = index or ()
+        meta = _meta_from_dtypes(all_columns, dtypes, index_cols, column_index_names)
+        if categories:
+            # Make sure all categories are set to "unknown".
+            # Cannot include index names in the `cols` argument.
+            meta = clear_known_categories(
+                meta, cols=[c for c in categories if c not in meta.index.names]
+            )
+
+        if partition_obj:
+
+            for partition in partition_obj:
+                if isinstance(index, list) and partition.name == index[0]:
+                    # Index from directory structure
+                    meta.index = pd.CategoricalIndex(
+                        categories=partition.keys, name=index[0]
+                    )
+                elif partition.name == meta.index.name:
+                    # Index created from a categorical column
+                    meta.index = pd.CategoricalIndex(
+                        categories=partition.keys, name=meta.index.name
+                    )
+                elif partition.name in meta.columns:
+                    meta[partition.name] = pd.Series(
+                        pd.Categorical(categories=partition.keys, values=[]),
+                        index=meta.index,
+                    )
+
+        return meta, index_cols, categories, index
+
+    @classmethod
+    def _construct_parts(
+        cls,
+        fs,
+        metadata,
+        schema,
+        filters,
+        index_cols,
+        data_path,
+        partition_info,
+        categories,
+        split_row_groups,
+        gather_statistics,
+        read_from_paths,
+    ):
+        """Construct ``parts`` for ddf construction
+
+        Use metadata (along with other data) to define a tuple
+        for each ddf partition.  Also gather statistics if
+        ``gather_statistics=True``, and other criteria is met.
+
+        This method is used by both `ArrowDatasetEngine`
+        and `ArrowLegacyEngine`.
+        """
+
+        partition_keys = partition_info["partition_keys"]
+        partition_obj = partition_info["partitions"]
+
+        # Check if `metadata` is just a list of paths
+        # (not splitting by row-group or collecting statistics)
+        if (
+            isinstance(metadata, list)
+            and len(metadata)
+            and isinstance(metadata[0], str)
+        ):
+            parts = []
+            stats = []
+            for full_path in metadata:
+                part = {
+                    "piece": (full_path, None, partition_keys.get(full_path, None)),
+                    "kwargs": {"partitions": partition_obj, "categories": categories},
+                }
+                parts.append(part)
+            return parts, stats
+
+        # Use final metadata info to update our options for
+        # `parts`/`stats` construnction
+        (
+            gather_statistics,
+            split_row_groups,
+            stat_col_indices,
+        ) = cls._update_metadata_options(
+            gather_statistics,
+            split_row_groups,
+            metadata,
+            schema,
+            index_cols,
+            filters,
+            partition_info,
+        )
+
+        # Convert metadata into `parts` and `stats`
+        return cls._process_metadata(
+            metadata,
+            schema,
+            split_row_groups,
+            gather_statistics,
+            stat_col_indices,
+            filters,
+            categories,
+            partition_info,
+            data_path,
+            fs,
+            read_from_paths,
+        )
+
+    @classmethod
+    def _update_metadata_options(
+        cls,
+        gather_statistics,
+        split_row_groups,
+        metadata,
+        schema,
+        index_cols,
+        filters,
+        partition_info,
+    ):
+        """Update read_parquet options given up-to-data metadata.
+
+        The primary focus here is `gather_statistics`. We want to
+        avoid setting this option to `True` if it is unnecessary.
+
+        This method is used by both `ArrowDatasetEngine`
+        and `ArrowLegacyEngine`.
+        """
+
+        # Cannot gather_statistics if our `metadata` is a list
+        # of paths, or if we are building a multiindex (for now).
+        # We also don't "need" to gather statistics if we don't
+        # want to apply any filters or calculate divisions. Note
+        # that the `ArrowDatasetEngine` doesn't even require
+        # `gather_statistics=True` for filtering.
+        if (
+            isinstance(metadata, list)
+            and len(metadata)
+            and isinstance(metadata[0], str)
+        ) or len(index_cols) > 1:
+            gather_statistics = False
+        elif filters is None and len(index_cols) == 0:
+            gather_statistics = False
+
+        # Determine which columns need statistics.
+        flat_filters = _flatten_filters(filters)
+        stat_col_indices = {}
+        for i, name in enumerate(schema.names):
+            if name in index_cols or name in flat_filters:
+                if name in partition_info["partition_names"]:
+                    # Partition columns wont have statistics
+                    continue
+                stat_col_indices[name] = i
+
+        # If the user has not specified `gather_statistics`,
+        # we will only do so if there are specific columns in
+        # need of statistics.
+        if gather_statistics is None:
+            gather_statistics = len(stat_col_indices.keys()) > 0
+        if split_row_groups is None:
+            split_row_groups = False
+
+        return (
+            gather_statistics,
+            split_row_groups,
+            stat_col_indices,
+        )
+
+    @classmethod
+    def _process_metadata(
+        cls,
+        metadata,
+        schema,
+        split_row_groups,
+        gather_statistics,
+        stat_col_indices,
+        filters,
+        categories,
+        partition_info,
+        data_path,
+        fs,
+        read_from_paths,
+    ):
+        """Process row-groups and statistics.
+
+        This method is overridden in `ArrowLegacyEngine`.
+        """
+
+        partition_keys = partition_info["partition_keys"]
+        partition_obj = partition_info["partitions"]
+
+        # Get the number of row groups per file
+        frag_map = {}
+        single_rg_parts = int(split_row_groups) == 1
+        file_row_groups = defaultdict(list)
+        file_row_group_stats = defaultdict(list)
+        file_row_group_column_stats = defaultdict(list)
+        cmax_last = {}
+        for (frag, row_group_info) in metadata:
+            fpath = frag.path
+            # Note that we include an optional `row_group_info` list
+            # in each element of `metadata` to avoid the need to
+            # re-read row-group statistics here. Once pyarrow allows
+            # row-group statistics to be preserved after a `make_fragment`
+            # call, we can always rely on `frag.row_groups`.
+            row_group_info = row_group_info or frag.row_groups
+            if gather_statistics or split_row_groups:
+                # If we are gathering statistics or splitting by
+                # row-group, we may need to ensure our fragment
+                # metadata is complete.
+                if row_group_info is None:
+                    frag.ensure_complete_metadata()
+                    row_group_info = frag.row_groups
+                elif gather_statistics and (row_group_info[0].statistics is None):
+                    frag.ensure_complete_metadata()
+                    row_group_info = frag.row_groups
+                frag_map[(fpath, row_group_info[0].id)] = frag
+            else:
+                file_row_groups[fpath] = [None]
+                frag_map[(fpath, None)] = frag
+                continue
+            for row_group in row_group_info:
+                file_row_groups[fpath].append(row_group.id)
+                if gather_statistics:
+                    statistics = row_group.statistics
+                    if single_rg_parts:
+                        s = {
+                            "file_path_0": fpath,
+                            "num-rows": row_group.num_rows,
+                            "total_byte_size": row_group.total_byte_size,
+                            "columns": [],
+                        }
+                    else:
+                        s = {
+                            "num-rows": row_group.num_rows,
+                            "total_byte_size": row_group.total_byte_size,
+                        }
+                    cstats = []
+                    for name, i in stat_col_indices.items():
+                        if name in statistics:
+                            cmin = statistics[name]["min"]
+                            cmax = statistics[name]["max"]
+                            cnull = 0  # Not yet available/needed
+                            last = cmax_last.get(name, None)
+                            if not filters:
+                                # Only think about bailing if we don't need
+                                # stats for filtering
+                                if cmin is None or (last and cmin < last):
+                                    # We are collecting statistics for divisions
+                                    # only (no filters) - Column isn't sorted, or
+                                    # we have an all-null partition, so lets bail.
+                                    #
+                                    # Note: This assumes ascending order.
+                                    #
+                                    gather_statistics = False
+                                    file_row_group_stats = {}
+                                    file_row_group_column_stats = {}
+                                    break
+
+                            if single_rg_parts:
+                                s["columns"].append(
+                                    {
+                                        "name": name,
+                                        "min": pd.Timestamp(cmin)
+                                        if isinstance(cmin, datetime)
+                                        else cmin,
+                                        "max": pd.Timestamp(cmax)
+                                        if isinstance(cmax, datetime)
+                                        else cmax,
+                                        "null_count": cnull,
+                                    }
+                                )
+                            else:
+                                cstats += [cmin, cmax, cnull]
+                            cmax_last[name] = cmax
+                        else:
+                            if single_rg_parts:
+                                s["columns"].append({"name": name})
+                            else:
+                                cstats += [None, None, None]
+                    if gather_statistics:
+                        file_row_group_stats[fpath].append(s)
+                        if not single_rg_parts:
+                            file_row_group_column_stats[fpath].append(tuple(cstats))
+
+        # Construct `parts` and `stats`
+        parts = []
+        stats = []
+        if split_row_groups:
+            # Create parts from each file,
+            # limiting the number of row_groups in each piece
+            split_row_groups = int(split_row_groups)
+            for filename, row_groups in file_row_groups.items():
+                row_group_count = len(row_groups)
+                for i in range(0, row_group_count, split_row_groups):
+                    i_end = i + split_row_groups
+                    rg_list = row_groups[i:i_end]
+                    # Get full path (empty strings should be ignored)
+                    full_path = fs.sep.join(
+                        [p for p in [data_path, filename] if p != ""]
+                    )
+                    pkeys = partition_keys.get(full_path, None)
+                    if partition_obj and pkeys is None:
+                        continue  # This partition was filtered
+                    part = {
+                        "piece": (
+                            full_path
+                            if read_from_paths
+                            else frag_map[(full_path, rg_list[0])],
+                            rg_list,
+                            pkeys,
+                        ),
+                        "kwargs": {
+                            "partitions": partition_obj,
+                            "categories": categories,
+                            "filters": filters,
+                            "schema": schema,
+                        },
+                    }
+                    parts.append(part)
+                    if gather_statistics:
+                        stat = cls._aggregate_stats(
+                            filename,
+                            file_row_group_stats[filename][i:i_end],
+                            file_row_group_column_stats[filename][i:i_end],
+                            stat_col_indices,
+                        )
+                        stats.append(stat)
+        else:
+            for filename, row_groups in file_row_groups.items():
+                # Get full path (empty strings should be ignored)
+                full_path = fs.sep.join([p for p in [data_path, filename] if p != ""])
+                pkeys = partition_keys.get(full_path, None)
+                if partition_obj and pkeys is None:
+                    continue  # This partition was filtered
+                part = {
+                    "piece": (
+                        full_path
+                        if read_from_paths
+                        else frag_map[(full_path, row_groups[0])],
+                        row_groups,
+                        pkeys,
+                    ),
+                    "kwargs": {
+                        "partitions": partition_obj,
+                        "categories": categories,
+                        "filters": filters,
+                        "schema": schema,
+                    },
+                }
+                parts.append(part)
+                if gather_statistics:
+                    stat = cls._aggregate_stats(
+                        filename,
+                        file_row_group_stats[filename],
+                        file_row_group_column_stats[filename],
+                        stat_col_indices,
+                    )
+                    stats.append(stat)
+
+        return parts, stats
+
+    @classmethod
+    def _aggregate_stats(
+        cls,
+        file_path,
+        file_row_group_stats,
+        file_row_group_column_stats,
+        stat_col_indices,
+    ):
+        """Utility to aggregate the statistics for N row-groups
+        into a single dictionary.
+
+        Used by `_construct_parts`
+        """
+        if len(file_row_group_stats) < 1:
+            # Empty statistics
+            return {}
+        elif len(file_row_group_column_stats) == 0:
+            assert len(file_row_group_stats) == 1
+            return file_row_group_stats[0]
+        else:
+            # Note: It would be better to avoid df_rgs and df_cols
+            #       construction altogether. It makes it fast to aggregate
+            #       the statistics for many row groups, but isn't
+            #       worthwhile for a small number of row groups.
+            if len(file_row_group_stats) > 1:
+                df_rgs = pd.DataFrame(file_row_group_stats)
+                s = {
+                    "file_path_0": file_path,
+                    "num-rows": df_rgs["num-rows"].sum(),
+                    "total_byte_size": df_rgs["total_byte_size"].sum(),
+                    "columns": [],
+                }
+            else:
+                s = {
+                    "file_path_0": file_path,
+                    "num-rows": file_row_group_stats[0]["num-rows"],
+                    "total_byte_size": file_row_group_stats[0]["total_byte_size"],
+                    "columns": [],
+                }
+
+            df_cols = None
+            if len(file_row_group_column_stats) > 1:
+                df_cols = pd.DataFrame(file_row_group_column_stats)
+            for ind, name in enumerate(stat_col_indices):
+                i = ind * 3
+                if df_cols is None:
+                    s["columns"].append(
+                        {
+                            "name": name,
+                            "min": file_row_group_column_stats[0][i],
+                            "max": file_row_group_column_stats[0][i + 1],
+                            "null_count": file_row_group_column_stats[0][i + 2],
+                        }
+                    )
+                else:
+                    s["columns"].append(
+                        {
+                            "name": name,
+                            "min": df_cols.iloc[:, i].min(),
+                            "max": df_cols.iloc[:, i + 1].max(),
+                            "null_count": df_cols.iloc[:, i + 2].sum(),
+                        }
+                    )
+            return s
+
+    @classmethod
+    def _read_table(
+        cls,
+        path_or_frag,
+        fs,
+        row_groups,
+        columns,
+        schema,
+        filters,
+        partitions,
+        partition_keys,
+        **kwargs,
+    ):
+        """Read in a pyarrow table.
+
+        This method is overridden in `ArrowLegacyEngine`.
+        """
+        if isinstance(path_or_frag, pa_ds.ParquetFileFragment):
+            cols = []
+            for name in columns:
+                if name is None:
+                    if "__index_level_0__" in schema.names:
+                        columns.append("__index_level_0__")
+                else:
+                    cols.append(name)
+
+            return path_or_frag.to_table(
+                use_threads=False,
+                schema=schema,
+                columns=cols,
+                filter=pq._filters_to_expression(filters) if filters else None,
+            )
+        else:
+            return _read_table_from_path(
+                path_or_frag,
+                fs,
+                row_groups,
+                columns,
+                schema,
+                filters,
+                None,  # partitions,
+                [],  # partition_keys,
+                cls._parquet_piece_as_arrow,
+                **kwargs,
+            )
+
+    @classmethod
+    def _arrow_table_to_pandas(
+        cls, arrow_table: pa.Table, categories, **kwargs
+    ) -> pd.DataFrame:
+        _kwargs = kwargs.get("arrow_to_pandas", {})
+        _kwargs.update({"use_threads": False, "ignore_metadata": False})
+
+        return arrow_table.to_pandas(categories=categories, **_kwargs)
+
+    @classmethod
+    def _parquet_piece_as_arrow(
+        cls, piece: pq.ParquetDatasetPiece, columns, partitions, **kwargs
+    ) -> pa.Table:
+        arrow_table = piece.read(
+            columns=columns,
+            partitions=partitions,
+            use_pandas_metadata=True,
+            use_threads=False,
+            **kwargs.get("read", {}),
+        )
+        return arrow_table
+
+
+#
+#  PyArrow Legacy API [PyArrow<1.0.0]
+#
+
+
+def _get_dataset_object(paths, fs, filters, dataset_kwargs):
+    """Generate a ParquetDataset object"""
+    kwargs = dataset_kwargs.copy()
+    if "validate_schema" not in kwargs:
+        kwargs["validate_schema"] = False
+    if len(paths) > 1:
+        # This is a list of files
+        base, fns = _analyze_paths(paths, fs)
+        proxy_metadata = None
+        if "_metadata" in fns:
+            # We have a _metadata file. PyArrow cannot handle
+            #  "_metadata" when `paths` is a list. So, we shuld
+            # open "_metadata" separately.
+            paths.remove(fs.sep.join([base, "_metadata"]))
+            fns.remove("_metadata")
+            with fs.open(fs.sep.join([base, "_metadata"]), mode="rb") as fil:
+                proxy_metadata = pq.ParquetFile(fil).metadata
+        # Create our dataset from the list of data files.
+        # Note #1: that this will not parse all the files (yet)
+        # Note #2: Cannot pass filters for legacy pyarrow API (see issue#6512).
+        #          We can handle partitions + filtering for list input after
+        #          adopting new pyarrow.dataset API.
+        dataset = pq.ParquetDataset(paths, filesystem=fs, **kwargs)
+        if proxy_metadata:
+            dataset.metadata = proxy_metadata
+    elif fs.isdir(paths[0]):
+        # This is a directory.  We can let pyarrow do its thing.
+        # Note: In the future, it may be best to avoid listing the
+        #       directory if we can get away with checking for the
+        #       existence of _metadata.  Listing may be much more
+        #       expensive in storage systems like S3.
+        allpaths = fs.glob(paths[0] + fs.sep + "*")
+        base, fns = _analyze_paths(allpaths, fs)
+        dataset = pq.ParquetDataset(paths[0], filesystem=fs, filters=filters, **kwargs)
+    else:
+        # This is a single file.  No danger in gathering statistics
+        # and/or splitting row-groups without a "_metadata" file
+        base = paths[0]
+        fns = [None]
+        dataset = pq.ParquetDataset(paths[0], filesystem=fs, **kwargs)
+
+    return dataset, base, fns
+
+
+class ArrowLegacyEngine(ArrowDatasetEngine):
+
+    #
+    # Private Class Methods
+    #
+
+    @classmethod
+    def _gather_metadata(
+        cls,
+        paths,
+        fs,
+        split_row_groups,
+        gather_statistics,
+        filters,
+        index,
+        read_from_paths,
+        dataset_kwargs,
+    ):
+        """Gather parquet metadata into a single data structure.
+
+        Use _metadata or aggregate footer metadata into a single
+        object.  Also, collect other information necessary for
+        parquet-to-ddf mapping (e.g. schema, partition_info).
+
+        This method overrides `ArrowDatasetEngine._gather_metadata`.
+        """
+
+        # Step 1: Create a ParquetDataset object
+        dataset, base, fns = _get_dataset_object(paths, fs, filters, dataset_kwargs)
+        if fns == [None]:
+            # This is a single file. No danger in gathering statistics
+            # and/or splitting row-groups without a "_metadata" file
+            if gather_statistics is None:
+                gather_statistics = True
+            if split_row_groups is None:
+                split_row_groups = True
+
+        # Step 2: Construct necessary (parquet) partitioning information
+        partition_info = {
+            "partitions": None,
+            "partition_keys": {},
+            "partition_names": [],
+        }
+        # The `partition_info` dict summarizes information needed to handle
+        # nested-directory (hive) partitioning.
+        #
+        #    - "partitions" : (ParquetPartitions) PyArrow-specific  object
+        #          needed to read in each partition correctly
+        #    - "partition_keys" : (dict) The keys and values correspond to
+        #          file paths and partition values, respectively. The partition
+        #          values (or partition "keys") will be represented as a list
+        #          of tuples. E.g. `[("year", 2020), ("state", "CA")]`
+        #    - "partition_names" : (list)  This is a list containing the names
+        #          of partitioned columns.
+        fn_partitioned = False
+        if dataset.partitions is not None:
+            fn_partitioned = True
+            partition_info["partition_names"] = [
+                n for n in dataset.partitions.partition_names if n is not None
+            ]
+            partition_info["partitions"] = dataset.partitions
+            for piece in dataset.pieces:
+                partition_info["partition_keys"][piece.path] = piece.partition_keys
+
+        # Make sure gather_statistics allows filtering
+        # (if filters are desired)
+        if filters:
+            # Filters may require us to gather statistics
+            if gather_statistics is False and partition_info["partition_names"]:
+                warnings.warn(
+                    "Filtering with gather_statistics=False. "
+                    "Only partition columns will be filtered correctly."
+                )
+            elif gather_statistics is False:
+                raise ValueError("Cannot apply filters with gather_statistics=False")
+            elif not gather_statistics:
+                gather_statistics = True
+
+        # Step 3: Construct a single `metadata` object. We can
+        #         directly use dataset.metadata if it is available.
+        #         Otherwise, if `gather_statistics` or `split_row_groups`,
+        #         we need to gether the footer metadata manually
+        metadata = None
+        if dataset.metadata:
+            # We have a _metadata file.
+            # PyArrow already did the work for us
+            schema = dataset.metadata.schema.to_arrow_schema()
+            if gather_statistics is None:
+                gather_statistics = True
+            if split_row_groups is None:
+                split_row_groups = True
+            return (
+                schema,
+                dataset.metadata,
+                base,
+                partition_info,
+                split_row_groups,
+                gather_statistics,
+            )
+        else:
+            # No _metadata file.
+            # May need to collect footer metadata manually
+            if dataset.schema is not None:
+                schema = dataset.schema.to_arrow_schema()
+            else:
+                schema = None
+            if gather_statistics is None:
+                gather_statistics = False
+            if split_row_groups is None:
+                split_row_groups = False
+            metadata = None
+            if not (split_row_groups or gather_statistics):
+                # Don't need to construct real metadata if
+                # we are not gathering statistics or splitting
+                # by row-group
+                metadata = [p.path for p in dataset.pieces]
+                if schema is None:
+                    schema = dataset.pieces[0].get_metadata().schema.to_arrow_schema()
+                return (
+                    schema,
+                    metadata,
+                    base,
+                    partition_info,
+                    split_row_groups,
+                    gather_statistics,
+                )
+            # We have not detected a _metadata file, and the user has specified
+            # that they want to split by row-group and/or gather statistics.
+            # This is the only case where we MUST scan all files to collect
+            # metadata.
+            for piece, fn in zip(dataset.pieces, fns):
+                md = piece.get_metadata()
+                if schema is None:
+                    schema = md.schema.to_arrow_schema()
+                if fn_partitioned:
+                    md.set_file_path(piece.path.replace(base + fs.sep, ""))
+                elif fn:
+                    md.set_file_path(fn)
+                if metadata:
+                    _append_row_groups(metadata, md)
+                else:
+                    metadata = md
+            return (
+                schema,
+                metadata,
+                base,
+                partition_info,
+                split_row_groups,
+                gather_statistics,
+            )
+
+    @classmethod
+    def _process_metadata(
+        cls,
+        metadata,
+        schema,
+        split_row_groups,
+        gather_statistics,
+        stat_col_indices,
+        filters,
+        categories,
+        partition_info,
+        data_path,
+        fs,
+        read_from_paths,
+    ):
+        """Process row-groups and statistics.
+
+        This method is overrides the `ArrowDatasetEngine` implementation.
+        """
+
+        partition_keys = partition_info["partition_keys"]
+        partition_obj = partition_info["partitions"]
+
+        # Get the number of row groups per file
+        single_rg_parts = int(split_row_groups) == 1
+        file_row_groups = defaultdict(list)
+        file_row_group_stats = defaultdict(list)
+        file_row_group_column_stats = defaultdict(list)
+        cmax_last = {}
+        for rg in range(metadata.num_row_groups):
+            row_group = metadata.row_group(rg)
+            fpath = row_group.column(0).file_path
+            if fpath is None:
+                raise ValueError(
+                    "Global metadata structure is missing a file_path string. "
+                    "If the dataset includes a _metadata file, that file may "
+                    "have one or more missing file_path fields."
+                )
+            if file_row_groups[fpath]:
+                file_row_groups[fpath].append(file_row_groups[fpath][-1] + 1)
+            else:
+                file_row_groups[fpath].append(0)
+            if gather_statistics:
+                if single_rg_parts:
+                    s = {
+                        "file_path_0": fpath,
+                        "num-rows": row_group.num_rows,
+                        "total_byte_size": row_group.total_byte_size,
+                        "columns": [],
+                    }
+                else:
+                    s = {
+                        "num-rows": row_group.num_rows,
+                        "total_byte_size": row_group.total_byte_size,
+                    }
+                cstats = []
+                for name, i in stat_col_indices.items():
+                    column = row_group.column(i)
+                    if column.statistics:
+                        cmin = column.statistics.min
+                        cmax = column.statistics.max
+                        cnull = column.statistics.null_count
+                        last = cmax_last.get(name, None)
+                        if not filters:
+                            # Only think about bailing if we don't need
+                            # stats for filtering
+                            if cmin is None or (last and cmin < last):
+                                # We are collecting statistics for divisions
+                                # only (no filters) - Column isn't sorted, or
+                                # we have an all-null partition, so lets bail.
+                                #
+                                # Note: This assumes ascending order.
+                                #
+                                gather_statistics = False
+                                file_row_group_stats = {}
+                                file_row_group_column_stats = {}
+                                break
+
+                        if single_rg_parts:
+                            to_ts = column.statistics.logical_type.type == "TIMESTAMP"
+                            s["columns"].append(
+                                {
+                                    "name": name,
+                                    "min": cmin if not to_ts else pd.Timestamp(cmin),
+                                    "max": cmax if not to_ts else pd.Timestamp(cmax),
+                                    "null_count": cnull,
+                                }
+                            )
+                        else:
+                            cstats += [cmin, cmax, cnull]
+                        cmax_last[name] = cmax
+                    else:
+
+                        if not filters and column.num_values > 0:
+                            # We are collecting statistics for divisions
+                            # only (no filters) - Lets bail.
+                            gather_statistics = False
+                            file_row_group_stats = {}
+                            file_row_group_column_stats = {}
+                            break
+
+                        if single_rg_parts:
+                            s["columns"].append({"name": name})
+                        else:
+                            cstats += [None, None, None]
+                if gather_statistics:
+                    file_row_group_stats[fpath].append(s)
+                    if not single_rg_parts:
+                        file_row_group_column_stats[fpath].append(tuple(cstats))
+
+        # Construct `parts` and `stats`
+        parts = []
+        stats = []
+        if split_row_groups:
+            # Create parts from each file,
+            # limiting the number of row_groups in each piece
+            split_row_groups = int(split_row_groups)
+            for filename, row_groups in file_row_groups.items():
+                row_group_count = len(row_groups)
+                for i in range(0, row_group_count, split_row_groups):
+                    i_end = i + split_row_groups
+                    rg_list = row_groups[i:i_end]
+                    # Get full path (empty strings should be ignored)
+                    full_path = fs.sep.join(
+                        [p for p in [data_path, filename] if p != ""]
+                    )
+                    pkeys = partition_keys.get(full_path, None)
+                    if partition_obj and pkeys is None:
+                        continue  # This partition was filtered
+                    part = {
+                        "piece": (full_path, rg_list, pkeys),
+                        "kwargs": {
+                            "partitions": partition_obj,
+                            "categories": categories,
+                            "filters": filters,
+                            "schema": schema,
+                        },
+                    }
+                    parts.append(part)
+                    if gather_statistics:
+                        stat = cls._aggregate_stats(
+                            filename,
+                            file_row_group_stats[filename][i:i_end],
+                            file_row_group_column_stats[filename][i:i_end],
+                            stat_col_indices,
+                        )
+                        stats.append(stat)
+        else:
+            for filename, row_groups in file_row_groups.items():
+                # Get full path (empty strings should be ignored)
+                full_path = fs.sep.join([p for p in [data_path, filename] if p != ""])
+                pkeys = partition_keys.get(full_path, None)
+                if partition_obj and pkeys is None:
+                    continue  # This partition was filtered
+                part = {
+                    "piece": (full_path, None, pkeys),
+                    "kwargs": {
+                        "partitions": partition_obj,
+                        "categories": categories,
+                        "filters": filters,
+                        "schema": schema,
+                    },
+                }
+                parts.append(part)
+                if gather_statistics:
+                    stat = cls._aggregate_stats(
+                        filename,
+                        file_row_group_stats[filename],
+                        file_row_group_column_stats[filename],
+                        stat_col_indices,
+                    )
+                    stats.append(stat)
+
+        return parts, stats
+
+    @classmethod
+    def _read_table(
+        cls,
+        path,
+        fs,
+        row_groups,
+        columns,
+        schema,
+        filters,
+        partitions,
+        partition_keys,
+        **kwargs,
+    ):
+        """Read in a pyarrow table.
+
+        This method is overrides the `ArrowLegacyEngine` implementation.
+        """
+
+        return _read_table_from_path(
+            path,
+            fs,
+            row_groups,
+            columns,
+            schema,
+            filters,
+            partitions,
+            partition_keys,
+            cls._parquet_piece_as_arrow,
+            **kwargs,
+        )
+
     @classmethod
     def collect_file_metadata(cls, path, fs, file_path):
         with fs.open(path, "rb") as f:
@@ -1083,3 +1961,8 @@ def aggregate_metadata(cls, meta_list, fs, out_path):
             return None
         else:
             return meta
+
+
+# Compatibility access to legacy ArrowEngine
+# (now called `ArrowLegacyEngine`)
+ArrowEngine = ArrowLegacyEngine
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -173,6 +173,7 @@ def read_parquet(
     engine="auto",
     gather_statistics=None,
     split_row_groups=None,
+    read_from_paths=None,
     chunksize=None,
     **kwargs,
 ):
@@ -196,8 +197,10 @@ def read_parquet(
         metadata, if present). Provide a single field name instead of a list to
         read in the data as a Series.
     filters : Union[List[Tuple[str, str, Any]], List[List[Tuple[str, str, Any]]]]
-        List of filters to apply, like ``[[('x', '=', 0), ...], ...]``. This
-        implements partition-level (hive) filtering only, i.e., to prevent the
+        List of filters to apply, like ``[[('x', '=', 0), ...], ...]``. Using this
+        argument will NOT result in row-wise filtering of the final partitions
+        unless `engine="pyarrow-dataset"` is also specified.  For other engines,
+        filtering is only performed at the partition level, i.e., to prevent the
         loading of some row-groups and/or files.
 
         Predicates can be expressed in disjunctive normal form (DNF). This means
@@ -222,9 +225,19 @@ def read_parquet(
         data written by dask/fastparquet, not otherwise.
     storage_options : dict
         Key/value pairs to be passed on to the file-system backend, if any.
-    engine : {'auto', 'fastparquet', 'pyarrow'}, default 'auto'
-        Parquet reader library to use. If only one library is installed, it
-        will use that one; if both, it will use 'fastparquet'
+    engine : str, default 'auto'
+        Parquet reader library to use. Options include: 'auto', 'fastparquet',
+        'pyarrow', 'pyarrow-dataset', and 'pyarrow-legacy'. Defaults to 'auto',
+        which selects the FastParquetEngine if fastparquet is installed (and
+        ArrowLegacyEngine otherwise).  If 'pyarrow-dataset' is specified, the
+        ArrowDatasetEngine (which leverages the pyarrow.dataset API) will be used
+        for newer PyArrow versions (>=1.0.0). If 'pyarrow' or 'pyarrow-legacy' are
+        specified, the ArrowLegacyEngine will be used (which leverages the
+        pyarrow.parquet.ParquetDataset API).
+        NOTE: 'pyarrow-dataset' enables row-wise filtering, but requires
+        pyarrow>=1.0. The behavior of 'pyarrow' will most likely change to
+        ArrowDatasetEngine in a future release, and the 'pyarrow-legacy'
+        option will be deprecated once the ParquetDataset API is deprecated.
     gather_statistics : bool or None (default).
         Gather the statistics for each dataset partition. By default,
         this will only be done if the _metadata file is available. Otherwise,
@@ -238,6 +251,15 @@ def read_parquet(
         complete file.  If a positive integer value is given, each dataframe
         partition will correspond to that number of parquet row-groups (or fewer).
         Only the "pyarrow" engine supports this argument.
+    read_from_paths : bool or None (default)
+        Only used by `ArrowDatasetEngine`. Determines whether the engine should
+        avoid inserting large pyarrow (`ParquetFileFragment`) objects in the
+        task graph.  If this option is `True`, `read_partition` will need to
+        depend on the file path and row-group ID list (rather than a fragment).
+        This option will reduce the size of the task graph, but will add minor
+        overhead to `read_partition`, and will disable row-wise filtering at
+        read time.  By default (None), `ArrowDatasetEngine` will set this
+        option to `False`.
     chunksize : int, str
         The target task partition size.  If set, consecutive row-groups
         from the same file will be aggregated into the same output
@@ -271,6 +293,9 @@ def read_parquet(
             storage_options,
             engine,
             gather_statistics,
+            split_row_groups,
+            read_from_paths,
+            chunksize,
         )
         return df[columns]
 
@@ -286,6 +311,8 @@ def read_parquet(
         storage_options,
         engine,
         gather_statistics,
+        read_from_paths,
+        chunksize,
     )
 
     if isinstance(engine, str):
@@ -309,9 +336,10 @@ def read_parquet(
         paths,
         categories=categories,
         index=index,
-        gather_statistics=gather_statistics,
+        gather_statistics=True if chunksize else gather_statistics,
         filters=filters,
         split_row_groups=split_row_groups,
+        read_from_paths=read_from_paths,
         **kwargs,
     )
 
@@ -769,9 +797,20 @@ def get_engine(engine):
 
     Parameters
     ----------
-    engine : {'auto', 'fastparquet', 'pyarrow'}, default 'auto'
-        Parquet reader library to use. Defaults to fastparquet if both are
-        installed
+    engine : str, default 'auto'
+        Backend parquet library to use. Options include: 'auto', 'fastparquet',
+        'pyarrow', 'pyarrow-dataset', and 'pyarrow-legacy'. Defaults to 'auto',
+        which selects the FastParquetEngine if fastparquet is installed (and
+        ArrowLegacyEngine otherwise).  If 'pyarrow-dataset' is specified, the
+        ArrowDatasetEngine (which leverages the pyarrow.dataset API) will be used
+        for newer PyArrow versions (>=1.0.0). If 'pyarrow' or 'pyarrow-legacy' are
+        specified, the ArrowLegacyEngine will be used (which leverages the
+        pyarrow.parquet.ParquetDataset API).
+        NOTE: 'pyarrow-dataset' enables row-wise filtering, but requires
+        pyarrow>=1.0. The behavior of 'pyarrow' will most likely change to
+        ArrowDatasetEngine in a future release, and the 'pyarrow-legacy'
+        option will be deprecated once the ParquetDataset API is deprecated.
+    gather_statistics : bool or None (default).
 
     Returns
     -------
@@ -796,14 +835,20 @@ def get_engine(engine):
         _ENGINES["fastparquet"] = eng = FastParquetEngine
         return eng
 
-    elif engine == "pyarrow" or engine == "arrow":
+    elif engine in ("pyarrow", "arrow", "pyarrow-legacy", "pyarrow-dataset"):
         pa = import_required("pyarrow", "`pyarrow` not installed")
-        from .arrow import ArrowEngine
 
         if LooseVersion(pa.__version__) < "0.13.1":
             raise RuntimeError("PyArrow version >= 0.13.1 required")
 
-        _ENGINES["pyarrow"] = eng = ArrowEngine
+        if engine == "pyarrow-dataset" and LooseVersion(pa.__version__) >= "1.0.0":
+            from .arrow import ArrowDatasetEngine
+
+            _ENGINES[engine] = eng = ArrowDatasetEngine
+        else:
+            from .arrow import ArrowLegacyEngine
+
+            _ENGINES[engine] = eng = ArrowLegacyEngine
         return eng
 
     else:


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -1,4 +1,5 @@
 import math
+import glob
 import os
 import glob
 import sys
@@ -53,6 +54,14 @@
         SKIP_PYARROW_REASON = "pyarrow not found"
 PYARROW_MARK = pytest.mark.skipif(SKIP_PYARROW, reason=SKIP_PYARROW_REASON)
 
+if pa and pa.__version__ < LooseVersion("1.0.0"):
+    SKIP_PYARROW_DS = True
+    SKIP_PYARROW_DS_REASON = "pyarrow >= 1.0.0 required for pyarrow dataset API"
+else:
+    SKIP_PYARROW_DS = not pa
+    SKIP_PYARROW_DS_REASON = "pyarrow not found"
+PYARROW_DS_MARK = pytest.mark.skipif(SKIP_PYARROW_DS, reason=SKIP_PYARROW_DS_REASON)
+
 
 def check_fastparquet():
     if SKIP_FASTPARQUET:
@@ -85,7 +94,8 @@ def check_engine():
 @pytest.fixture(
     params=[
         pytest.param("fastparquet", marks=FASTPARQUET_MARK),
-        pytest.param("pyarrow", marks=PYARROW_MARK),
+        pytest.param("pyarrow-legacy", marks=PYARROW_MARK),
+        pytest.param("pyarrow-dataset", marks=PYARROW_DS_MARK),
     ]
 )
 def engine(request):
@@ -97,13 +107,14 @@ def write_read_engines(**kwargs):
 
     To add custom marks, pass keyword of the form: `mark_writer_reader=reason`,
     or `mark_engine=reason` to apply to all parameters with that engine."""
-    backends = {"pyarrow", "fastparquet"}
+    backends = {"pyarrow-dataset", "pyarrow-legacy", "fastparquet"}
     marks = {(w, r): [] for w in backends for r in backends}
 
     # Skip if uninstalled
     for name, skip, reason in [
         ("fastparquet", SKIP_FASTPARQUET, SKIP_FASTPARQUET_REASON),
-        ("pyarrow", SKIP_PYARROW, SKIP_PYARROW_REASON),
+        ("pyarrow-legacy", SKIP_PYARROW_DS, SKIP_PYARROW_DS_REASON),
+        ("pyarrow-dataset", SKIP_PYARROW, SKIP_PYARROW_REASON),
     ]:
         if skip:
             val = pytest.mark.skip(reason=reason)
@@ -137,11 +148,19 @@ def write_read_engines(**kwargs):
 
 pyarrow_fastparquet_msg = "fastparquet fails reading pyarrow written directories"
 write_read_engines_xfail = write_read_engines(
-    xfail_pyarrow_fastparquet=pyarrow_fastparquet_msg
+    **{
+        "xfail_pyarrow-dataset_fastparquet": pyarrow_fastparquet_msg,
+        "xfail_pyarrow-legacy_fastparquet": pyarrow_fastparquet_msg,
+    }
 )
 
 fp_pandas_msg = "pandas with fastparquet engine does not preserve index"
-fp_pandas_xfail = write_read_engines(xfail_fastparquet_pyarrow=fp_pandas_msg)
+fp_pandas_xfail = write_read_engines(
+    **{
+        "xfail_fastparquet_pyarrow-dataset": fp_pandas_msg,
+        "xfail_fastparquet_pyarrow-legacy": fp_pandas_msg,
+    }
+)
 
 
 @write_read_engines()
@@ -529,7 +548,9 @@ def test_roundtrip_from_pandas(tmpdir, write_engine, read_engine):
     fn = str(tmpdir.join("test.parquet"))
     dfp = df.copy()
     dfp.index.name = "index"
-    dfp.to_parquet(fn, engine=write_engine)
+    dfp.to_parquet(
+        fn, engine="pyarrow" if write_engine.startswith("pyarrow") else "fastparquet"
+    )
     ddf = dd.read_parquet(fn, index="index", engine=read_engine)
     assert_eq(dfp, ddf)
 
@@ -548,7 +569,7 @@ def test_categorical(tmpdir, write_engine, read_engine):
     assert ddf2.compute().x.cat.categories.tolist() == ["a", "b", "c"]
 
     # autocat
-    if read_engine != "pyarrow":
+    if read_engine == "fastparquet":
         ddf2 = dd.read_parquet(tmp, engine=read_engine)
         assert ddf2.compute().x.cat.categories.tolist() == ["a", "b", "c"]
 
@@ -812,7 +833,6 @@ def test_ordering(tmpdir, write_engine, read_engine):
 
 
 def test_read_parquet_custom_columns(tmpdir, engine):
-
     tmp = str(tmpdir)
     data = pd.DataFrame(
         {"i32": np.arange(1000, dtype=np.int32), "f": np.arange(1000, dtype=np.float64)}
@@ -918,7 +938,7 @@ def test_categories_unnamed_index(tmpdir, engine):
     # Check that we can handle an unnamed categorical index
     # https://github.com/dask/dask/issues/6885
 
-    if engine == "pyarrow" and pa.__version__ < LooseVersion("0.15.0"):
+    if engine.startswith("pyarrow") and pa.__version__ < LooseVersion("0.15.0"):
         pytest.skip("PyArrow>=0.15 Required.")
 
     tmpdir = str(tmpdir)
@@ -992,14 +1012,19 @@ def test_to_parquet_pyarrow_w_inconsistent_schema_by_partition_fails_by_default(
     )
 
     # Test that schema is not validated by default
-    # (shouldn't raise error)
-    dd.read_parquet(str(tmpdir), engine="pyarrow", gather_statistics=False).compute()
+    # (shouldn't raise error with legacy dataset)
+    dd.read_parquet(
+        str(tmpdir),
+        engine="pyarrow-legacy",
+        gather_statistics=False,
+    ).compute()
 
     # Test that read fails when validate_schema=True
+    # Note: This fails differently for pyarrow.dataset api
     with pytest.raises(ValueError) as e_info:
         dd.read_parquet(
             str(tmpdir),
-            engine="pyarrow",
+            engine="pyarrow-legacy",
             gather_statistics=False,
             dataset={"validate_schema": True},
         ).compute()
@@ -1289,8 +1314,14 @@ def test_filters(tmpdir, write_engine, read_engine):
 def test_filters_v0(tmpdir, write_engine, read_engine):
     if write_engine == "fastparquet" or read_engine == "fastparquet":
         pytest.importorskip("fastparquet", minversion="0.3.1")
-    fn = str(tmpdir)
 
+    # Recent versions of pyarrow support full row-wise filtering
+    # (fastparquet and older pyarrow versions do not)
+    pyarrow_row_filtering = (
+        read_engine == "pyarrow-dataset" and pa.__version__ >= LooseVersion("1.0.0")
+    )
+
+    fn = str(tmpdir)
     df = pd.DataFrame({"at": ["ab", "aa", "ba", "da", "bb"]})
     ddf = dd.from_pandas(df, npartitions=1)
 
@@ -1301,11 +1332,14 @@ def test_filters_v0(tmpdir, write_engine, read_engine):
     ddf2 = dd.read_parquet(
         fn, index=False, engine=read_engine, filters=[("at", "==", "aa")]
     ).compute()
-    assert_eq(ddf2, ddf)
+    if pyarrow_row_filtering:
+        assert_eq(ddf2, ddf[ddf["at"] == "aa"], check_index=False)
+    else:
+        assert_eq(ddf2, ddf)
 
     # with >1 partition and no filters
     ddf.repartition(npartitions=2, force=True).to_parquet(fn, engine=write_engine)
-    dd.read_parquet(fn, engine=read_engine).compute()
+    ddf2 = dd.read_parquet(fn, engine=read_engine).compute()
     assert_eq(ddf2, ddf)
 
     # with >1 partition and filters using base fastparquet
@@ -1316,10 +1350,38 @@ def test_filters_v0(tmpdir, write_engine, read_engine):
 
     # with >1 partition and filters
     ddf.repartition(npartitions=2, force=True).to_parquet(fn, engine=write_engine)
-    dd.read_parquet(fn, engine=read_engine, filters=[("at", "==", "aa")]).compute()
+    ddf2 = dd.read_parquet(
+        fn, engine=read_engine, filters=[("at", "==", "aa")]
+    ).compute()
     assert len(ddf2) > 0
 
 
+def test_filtering_pyarrow_dataset(tmpdir, engine):
+    pytest.importorskip("pyarrow", minversion="1.0.0")
+
+    fn = str(tmpdir)
+    df = pd.DataFrame({"aa": range(100), "bb": ["cat", "dog"] * 50})
+    ddf = dd.from_pandas(df, npartitions=10)
+    ddf.to_parquet(fn, write_index=False, engine=engine)
+
+    # Filtered read
+    aa_lim = 40
+    bb_val = "dog"
+    filters = [[("aa", "<", aa_lim), ("bb", "==", bb_val)]]
+    ddf2 = dd.read_parquet(fn, index=False, engine="pyarrow-dataset", filters=filters)
+
+    # Check that partitions are filetered for "aa" filter
+    nonempty = 0
+    for part in ddf[ddf["aa"] < aa_lim].partitions:
+        nonempty += int(len(part.compute()) > 0)
+    assert ddf2.npartitions == nonempty
+
+    # Check that rows are filtered for "aa" and "bb" filters
+    df = df[df["aa"] < aa_lim]
+    df = df[df["bb"] == bb_val]
+    assert_eq(df, ddf2.compute(), check_index=False)
+
+
 def test_fiters_file_list(tmpdir, engine):
     df = pd.DataFrame({"x": range(10), "y": list("aabbccddee")})
     ddf = dd.from_pandas(df, npartitions=5)
@@ -1343,6 +1405,44 @@ def test_fiters_file_list(tmpdir, engine):
     assert len(ddf2) == 0
 
 
+def test_pyarrow_filter_divisions(tmpdir):
+    pytest.importorskip("pyarrow")
+
+    # Write simple dataset with an index that will only
+    # have a sorted index if certain row-groups are filtered out.
+    # In this case, we filter "a" <= 3 to get a sorted
+    # index. Otherwise, "a" is NOT monotonically increasing.
+    df = pd.DataFrame({"a": [0, 1, 10, 12, 2, 3, 8, 9], "b": range(8)}).set_index("a")
+    df.iloc[:4].to_parquet(
+        str(tmpdir.join("file.0.parquet")), engine="pyarrow", row_group_size=2
+    )
+    df.iloc[4:].to_parquet(
+        str(tmpdir.join("file.1.parquet")), engine="pyarrow", row_group_size=2
+    )
+
+    if pa.__version__ >= LooseVersion("1.0.0"):
+        # Only works for ArrowDatasetEngine.
+        # Legacy code will not apply filters on individual row-groups
+        # when `split_row_groups=False`.
+        ddf = dd.read_parquet(
+            str(tmpdir),
+            engine="pyarrow-dataset",
+            split_row_groups=False,
+            gather_statistics=True,
+            filters=[("a", "<=", 3)],
+        )
+        assert ddf.divisions == (0, 2, 3)
+
+    ddf = dd.read_parquet(
+        str(tmpdir),
+        engine="pyarrow-dataset",
+        split_row_groups=True,
+        gather_statistics=True,
+        filters=[("a", "<=", 3)],
+    )
+    assert ddf.divisions == (0, 2, 3)
+
+
 def test_divisions_read_with_filters(tmpdir):
     pytest.importorskip("fastparquet", minversion="0.3.1")
     tmpdir = str(tmpdir)
@@ -1844,13 +1944,14 @@ def test_writing_parquet_with_kwargs(tmpdir, engine):
     ddf = dd.from_pandas(df, npartitions=3)
 
     engine_kwargs = {
-        "pyarrow": {
+        "pyarrow-dataset": {
             "compression": "snappy",
             "coerce_timestamps": None,
             "use_dictionary": True,
         },
         "fastparquet": {"compression": "snappy", "times": "int64", "fixed_text": None},
     }
+    engine_kwargs["pyarrow-legacy"] = engine_kwargs["pyarrow-dataset"]
 
     ddf.to_parquet(path1, engine=engine, **engine_kwargs[engine])
     out = dd.read_parquet(path1, engine=engine)
@@ -1898,7 +1999,7 @@ def my_get(*args, **kwargs):
 
 def test_select_partitioned_column(tmpdir, engine):
     pytest.importorskip("snappy")
-    if engine == "pyarrow":
+    if engine.startswith("pyarrow"):
         import pyarrow as pa
 
         if pa.__version__ < LooseVersion("0.9.0"):
@@ -1925,7 +2026,7 @@ def test_select_partitioned_column(tmpdir, engine):
 
 
 def test_with_tz(tmpdir, engine):
-    if engine == "pyarrow" and pa.__version__ < LooseVersion("0.11.0"):
+    if engine.startswith("pyarrow") and pa.__version__ < LooseVersion("0.11.0"):
         pytest.skip("pyarrow<0.11.0 did not support this")
     if engine == "fastparquet" and fastparquet.__version__ < LooseVersion("0.3.0"):
         pytest.skip("fastparquet<0.3.0 did not support this")
@@ -1964,7 +2065,7 @@ def test_arrow_partitioning(tmpdir):
 
 def test_sorted_warnings(tmpdir, engine):
 
-    if engine == "pyarrow":
+    if engine.startswith("pyarrow"):
         pytest.skip(
             "ArrowEngine will only collect statistics for "
             "known index columns and/or filtered columns."
@@ -2100,7 +2201,7 @@ def test_categories_large(tmpdir, engine):
 
 
 @write_read_engines()
-def test_read_glob_no_stats(tmpdir, write_engine, read_engine):
+def test_read_glob_no_meta(tmpdir, write_engine, read_engine):
     tmp_path = str(tmpdir)
     ddf.to_parquet(tmp_path, engine=write_engine)
 
@@ -2111,10 +2212,9 @@ def test_read_glob_no_stats(tmpdir, write_engine, read_engine):
 
 
 @write_read_engines()
-def test_read_glob_yes_stats(tmpdir, write_engine, read_engine):
+def test_read_glob_yes_meta(tmpdir, write_engine, read_engine):
     tmp_path = str(tmpdir)
     ddf.to_parquet(tmp_path, engine=write_engine)
-
     paths = glob.glob(os.path.join(tmp_path, "*.parquet"))
     paths.append(os.path.join(tmp_path, "_metadata"))
     ddf2 = dd.read_parquet(paths, engine=read_engine, gather_statistics=False)
@@ -2144,7 +2244,7 @@ def test_timeseries_nulls_in_schema(tmpdir, engine, schema):
 
     if (
         schema == "infer"
-        and engine == "pyarrow"
+        and engine.startswith("pyarrow")
         and pa.__version__ < LooseVersion("0.15.0")
     ):
         pytest.skip("PyArrow>=0.15 Required.")
@@ -2162,21 +2262,27 @@ def test_timeseries_nulls_in_schema(tmpdir, engine, schema):
     ddf2.name = ddf2.name.where(ddf2.timestamp == "2000-01-01", None)
 
     # Note: `append_row_groups` will fail with pyarrow>0.17.1 for _metadata write
-    ddf2.to_parquet(tmp_path, engine=engine, write_metadata_file=False, schema=schema)
-    ddf_read = dd.read_parquet(
-        tmp_path, engine=engine, dataset={"validate_schema": False}
+    dataset = {"validate_schema": False}
+    engine_use = engine
+    if schema != "infer" and engine.startswith("pyarrow"):
+        engine_use = "pyarrow-legacy"
+    ddf2.to_parquet(
+        tmp_path, engine=engine_use, write_metadata_file=False, schema=schema
     )
+    ddf_read = dd.read_parquet(tmp_path, engine=engine_use, dataset=dataset)
 
     assert_eq(ddf_read, ddf2, check_divisions=False, check_index=False)
 
     # Can force schema validation on each partition in pyarrow
-    if engine == "pyarrow" and schema is None:
+    if engine.startswith("pyarrow") and schema is None:
+        dataset = {"validate_schema": True}
+        engine_use = engine
+        if schema != "infer":
+            engine_use = "pyarrow-legacy"
         # The schema mismatch should raise an error if the
         # dataset was written with `schema=None` (no inference)
         with pytest.raises(ValueError):
-            ddf_read = dd.read_parquet(
-                tmp_path, dataset={"validate_schema": True}, engine=engine
-            )
+            ddf_read = dd.read_parquet(tmp_path, dataset=dataset, engine=engine_use)
 
 
 @pytest.mark.parametrize("numerical", [True, False])
@@ -2268,7 +2374,9 @@ def test_graph_size_pyarrow(tmpdir, engine):
     ddf1.to_parquet(fn, engine=engine)
     ddf2 = dd.read_parquet(fn, engine=engine)
 
-    assert len(pickle.dumps(ddf2.__dask_graph__())) < 10000
+    # pyarrow.dataset requires fragments to be passed in dict.
+    # This requires slightly more space than "legacy"
+    assert len(pickle.dumps(ddf2.__dask_graph__())) < 32000
 
 
 @pytest.mark.parametrize("preserve_index", [True, False])
@@ -2488,6 +2596,8 @@ def test_split_row_groups_filter_pyarrow(tmpdir):
         filters=filters,
     )
 
+    ddf2.compute(scheduler="sync")
+
     assert search_val in ddf3["i32"]
     assert_eq(
         ddf2[ddf2["i32"] == search_val].compute(),
@@ -2597,7 +2707,9 @@ def test_roundtrip_pandas_chunksize(tmpdir, write_engine, read_engine):
     path = str(tmpdir.join("test.parquet"))
     pdf = df.copy()
     pdf.index.name = "index"
-    pdf.to_parquet(path, engine=write_engine)
+    pdf.to_parquet(
+        path, engine="pyarrow" if write_engine.startswith("pyarrow") else "fastparquet"
+    )
 
     ddf_read = dd.read_parquet(
         path,
@@ -2629,7 +2741,8 @@ def test_read_parquet_getitem_skip_when_getting_getitem(tmpdir, engine):
     # https://github.com/dask/dask/issues/5893
     pdf = pd.DataFrame({"A": [1, 2, 3, 4, 5, 6], "B": ["a", "b", "c", "d", "e", "f"]})
     path = os.path.join(str(tmpdir), "data.parquet")
-    pdf.to_parquet(path, engine=engine)
+    pd_engine = "pyarrow" if engine.startswith("pyarrow") else "fastparquet"
+    pdf.to_parquet(path, engine=pd_engine)
 
     ddf = dd.read_parquet(path, engine=engine)
     a, b = dask.optimize(ddf["A"], ddf)
@@ -2707,7 +2820,7 @@ def test_pandas_timestamp_overflow_pyarrow(tmpdir):
 
     # This will raise by default due to overflow
     with pytest.raises(pa.lib.ArrowInvalid) as e:
-        dd.read_parquet(str(tmpdir), engine="pyarrow").compute()
+        dd.read_parquet(str(tmpdir), engine="pyarrow-legacy").compute()
     assert "out of bounds" in str(e.value)
 
     from dask.dataframe.io.parquet.arrow import ArrowEngine
@@ -2765,7 +2878,7 @@ def _arrow_table_to_pandas(
 @write_read_engines_xfail
 def test_partitioned_preserve_index(tmpdir, write_engine, read_engine):
 
-    if write_engine == "pyarrow" and pa.__version__ < LooseVersion("0.15.0"):
+    if write_engine.startswith("pyarrow") and pa.__version__ < LooseVersion("0.15.0"):
         pytest.skip("PyArrow>=0.15 Required.")
 
     tmp = str(tmpdir)
@@ -2797,7 +2910,11 @@ def test_from_pandas_preserve_none_index(tmpdir, engine):
     fn = str(tmpdir.join("test.parquet"))
     df = pd.DataFrame({"a": [1, 2], "b": [4, 5], "c": [6, 7]}).set_index("c")
     df.index.name = None
-    df.to_parquet(fn, engine="pyarrow", index=True)
+    df.to_parquet(
+        fn,
+        engine="pyarrow" if engine.startswith("pyarrow") else "fastparquet",
+        index=True,
+    )
 
     expect = pd.read_parquet(fn)
     got = dd.read_parquet(fn, engine=engine)
@@ -2809,7 +2926,9 @@ def test_from_pandas_preserve_none_rangeindex(tmpdir, write_engine, read_engine)
     # See GitHub Issue#6348
     fn = str(tmpdir.join("test.parquet"))
     df0 = pd.DataFrame({"t": [1, 2, 3]}, index=pd.RangeIndex(start=1, stop=4))
-    df0.to_parquet(fn, engine=write_engine)
+    df0.to_parquet(
+        fn, engine="pyarrow" if write_engine.startswith("pyarrow") else "fastparquet"
+    )
 
     df1 = dd.read_parquet(fn, engine=read_engine)
     assert_eq(df0, df1.compute())
@@ -2845,6 +2964,78 @@ def test_divisions_with_null_partition(tmpdir, engine):
     assert ddf_read.divisions == (None, None, None)
 
 
+def test_pyarrow_dataset_simple(tmpdir, engine):
+    check_pyarrow()
+    fn = str(tmpdir)
+    df = pd.DataFrame({"a": [4, 5, 6], "b": ["a", "b", "b"]})
+    df.set_index("a", inplace=True, drop=True)
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf.to_parquet(fn, engine=engine)
+    read_df = dd.read_parquet(fn, engine="pyarrow-legacy")
+    read_df.compute(scheduler="synchronous")
+    assert_eq(ddf, read_df)
+
+
+@pytest.mark.parametrize("test_filter", [True, False])
+def test_pyarrow_dataset_partitioned(tmpdir, engine, test_filter):
+    check_pyarrow()
+
+    if pa.__version__ <= LooseVersion("0.17.1"):
+        # Using pyarrow.dataset API does not produce
+        # Categorical type for partitioned columns.
+        pytest.skip("PyArrow>0.17.1 Required.")
+
+    fn = str(tmpdir)
+    df = pd.DataFrame({"a": [4, 5, 6], "b": ["a", "b", "b"]})
+    df["b"] = df["b"].astype("category")
+    ddf = dd.from_pandas(df, npartitions=2)
+    ddf.to_parquet(fn, engine=engine, partition_on="b")
+    read_df = dd.read_parquet(
+        fn,
+        engine="pyarrow",
+        filters=[("b", "==", "a")] if test_filter else None,
+    )
+
+    if test_filter:
+        assert_eq(ddf[ddf["b"] == "a"].compute(), read_df.compute())
+    else:
+        assert_eq(ddf, read_df)
+
+
+@pytest.mark.parametrize("read_from_paths", [True, False])
+@pytest.mark.parametrize("test_filter_partitioned", [True, False])
+def test_pyarrow_dataset_read_from_paths(
+    tmpdir, read_from_paths, test_filter_partitioned
+):
+    check_pyarrow()
+
+    if pa.__version__ <= LooseVersion("0.17.1"):
+        # Using pyarrow.dataset API does not produce
+        # Categorical type for partitioned columns.
+        pytest.skip("PyArrow>0.17.1 Required.")
+
+    fn = str(tmpdir)
+    df = pd.DataFrame({"a": [4, 5, 6], "b": ["a", "b", "b"]})
+    df["b"] = df["b"].astype("category")
+    ddf = dd.from_pandas(df, npartitions=2)
+
+    if test_filter_partitioned:
+        ddf.to_parquet(fn, engine="pyarrow", partition_on="b")
+    else:
+        ddf.to_parquet(fn, engine="pyarrow")
+    read_df = dd.read_parquet(
+        fn,
+        engine="pyarrow",
+        filters=[("b", "==", "a")] if test_filter_partitioned else None,
+        read_from_paths=read_from_paths,
+    )
+
+    if test_filter_partitioned:
+        assert_eq(ddf[ddf["b"] == "a"].compute(), read_df.compute())
+    else:
+        assert_eq(ddf, read_df)
+
+
 def test_parquet_pyarrow_write_empty_metadata(tmpdir):
     # https://github.com/dask/dask/issues/6600
     check_pyarrow()