Repository: dask/dask

Issue:
`read_parquet` performance issue with PyArrow and S3
I've been running some tests on `dd.read_parquet` to see how fast we can load parquet data from S3. The test data I've been using is the Ursa labs copy of the NYC taxi dataset, (`s3://ursa-labs-taxi-data`), which is partitioned by year and month into ~400-500 MB files. I'm not trying to do anything scalable, just see how fast of throughput we can get. I've been running on raw EC2 instances in the same region as the s3 bucket (`us-east-2`).

I've run into an issue where `dd.read_parquet` on a single file is *much* slower than simply downloading the file, or reading it into a single partition using raw pyarrow calls. The maximum per-HTTP-connection download speed for such a setup seems to be ~100 MB/s, so loading the parquet dataset in 5-10 seconds should be considered "good".

As far as I can tell, the biggest difference is that [`pq.read_table`](https://github.com/apache/arrow/blob/bd66c7e8a4a68a1fc9d30e8c60829518db68fded/python/pyarrow/parquet/__init__.py#L2697-L2738) dispatches to `pyarrow.dataset`, but `ParquetFile` does not. Dask currently uses the latter.

**Minimal Complete Verifiable Example**:

This tests three cases:
1. Using `dd.read_parquet` to read a single partition
2. Using `pq.read_table` to read a single partition
3. using `pq.ParquetFile` to read a single partition

The first one is fast, the latter two are slow (at least in my testing). My example shows running them in a `LocalCluster`, but I have also reproduced the issue just executing the functions directly, so it seems likely to be an issue with our usage of pyarrow.

```python
import contextlib
import time

import dask
import dask.dataframe as dd
import distributed
import pyarrow.parquet as pq
import s3fs

@contextlib.contextmanager
def timer(label="Block"):
    """small timing utility"""
    start = time.time()
    yield
    end = time.time()
    print(f"{label} took {end-start} seconds")

fs = s3fs.S3FileSystem(anon=True)
paths = fs.glob("s3://ursa-labs-taxi-data/2009/01/data.parquet")


def pq_ParquetFile(path):
    """Read using pq.ParquetFile"""
    with fs.open("s3://" + path) as f:
        table = pq.ParquetFile(f).read(
            columns=None,
            use_threads=False,
            use_pandas_metadata=True,
        )
        del table

def pq_read_table(path):
    """read using pq.read_table"""
    with fs.open("s3://" + path) as f:
        table = pq.read_table(f, use_threads=False, partitioning="hive")
        del table

if __name__ == "__main__":
    ddf = dd.read_parquet("s3://" + paths[0], engine="pyarrow", storage_options={"anon": True})

    client = distributed.Client(n_workers=1)

    with timer("read_table"):
        f = client.submit(pq_read_table, paths[0], pure=False)
        distributed.wait(f)

    with timer("Dask"):
        ddf = ddf.persist()
        distributed.wait(ddf)

    with timer("ParquetFile"):
        f = client.submit(pq_ParquetFile, paths[0], pure=False)
        distributed.wait(f)
```
produces
```
read_table took 6.480799198150635 seconds
Dask took 94.15848159790039 seconds
ParquetFile took 121.40401339530945 seconds
```

**Environment**:

- Dask version: 2022.4.1
- PyArrow version: 7.0
- Python version: 3.8
- Operating System: Ubuntu
- Install method (conda, pip, source): conda

@rjzamora and perhaps @jorisvandenbossche might have some insight as to what's going on here.


Discussion:
Thanks for writing this up @ian-r-rose !

In order to compare Dask with pyarrow, you need to add `.to_pandas()` after creating the table. This should slow down the "read_table" case a bit.  The `dd.read_parquet` case is still pretty slow (and I'll look into exactly *why*). However, I did notice that using #8944 (and replacing `dd.read_parquet` with `dd.from_dataset(..., engine="pyarrow-parquet")` makes the Dask case much faster ~approximately the same as pyarrow~ [EDITED]
Thanks @rjzamora! 

My writeup didn't include all of the combinations I tried, and I did check both `to_pandas` as well as `pd.read_parquet()`, which had similar results, and the overall story didn't change. I was mostly trying to remove layers in the above examples to gesture towards what seems to be the important difference between the code paths, even though I didn't find the root cause.

>  makes the Dask case much faster 

That's exciting to hear :)

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -36,6 +36,7 @@
 from pyarrow import dataset as pa_ds
 
 subset_stats_supported = _pa_version > parse_version("2.0.0")
+pre_buffer_supported = _pa_version >= parse_version("5.0.0")
 del _pa_version
 
 #
@@ -214,6 +215,11 @@ def _read_table_from_path(
         ),
     )
 
+    pre_buffer = (
+        {"pre_buffer": read_kwargs.pop("pre_buffer", True)}
+        if pre_buffer_supported
+        else {}
+    )
     with _open_input_files(
         [path],
         fs=fs,
@@ -221,14 +227,14 @@ def _read_table_from_path(
         **open_file_options,
     )[0] as fil:
         if row_groups == [None]:
-            return pq.ParquetFile(fil).read(
+            return pq.ParquetFile(fil, **pre_buffer).read(
                 columns=columns,
                 use_threads=False,
                 use_pandas_metadata=True,
                 **read_kwargs,
             )
         else:
-            return pq.ParquetFile(fil).read_row_groups(
+            return pq.ParquetFile(fil, **pre_buffer).read_row_groups(
                 row_groups,
                 columns=columns,
                 use_threads=False,


Test patch: