Repository: dask/dask

Issue:
`calculate_divisions` note in `read_parquet` is incorrect / incomplete
**What happened**:

In the `read_parquet` docstring at https://github.com/dask/dask/blob/7b054f1d8f4a2343268f5182d47996a5469b10dc/dask/dataframe/io/parquet/core.py#L253, we note:

> Note that ``calculate_divisions=True`` may be extremely slow
> on some systems, and should be avoided when reading from remote storage.

I think this is not necessarily accurate, or perhaps I'm misreading something.

IIUC, the presence of a `_metadata` file will ensure that divisions are set, at the cost of (just) a single extra HTTP call to blob storage and parsing that parquet metadata structure.

When that file is *not* present, then perhaps it can be slow? And of course, there's the issue that the docs don't know whether or not the user's application *needs* to know the divisions. If you need the divisions, then you need the divisions.

**Minimal Complete Verifiable Example**:

```python
%%time
import dask.dataframe
import requests

storage_options = {
    "account_name": "ai4edataeuwest",
    "credential": requests.get("https://planetarycomputer.microsoft.com/api/sas/v1/token/ai4edataeuwest/us-census").json()["token"],
}

pop = dask.dataframe.read_parquet(
    "abfs://us-census/2020/census_blocks_population.parquet",
    storage_options=storage_options,
    calculate_divisions=True
)
```

This takes 300-400 ms (when accessed from the datacenter as the data: Azure's West Europe region).

@rjzamora git blame points to you. Do you recall what you had in mind when you wrote that section?



Discussion:
Thanks for pointing that out @TomAugspurger - I totally agree that the doc-string is incomplete, and that we should improve it. The note should probably clarify that calculating divisions can be slow for remote storage when when there is no `_metadata` file present, but that it may be necessary for some workflows (like those that need to join on a sorted index).

Overall, I don’t feel strongly at all about the message in the docstring. However, I do feel somewhat strongly that the default setting for `calculate_divisions` should be `False`, and that it should not depend on the presence of a `_metadata` file. I had a few discussions with @jcrist about this, and my reasoning was that (1) we no longer write out the `_metadata` file by default for scaling/stability reasons and therefore should make sure the behavior of the `_metadata`-free case is optimal, and (2) we gain a lot more by avoiding unpredictable `divisions` output than we do by letting some users avoid an explicit flag.
Sounds good. I think I got confused because the first line mentions that the divisions can come from the min/max statistics or the global `_metadata`. I'll update the note to say that the extreme slowness can occur when you don't have the global `_metadata` file.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -257,7 +257,9 @@ def read_parquet(
         This option will be ignored if ``index`` is not specified and there is
         no physical index column specified in the custom "pandas" Parquet
         metadata. Note that ``calculate_divisions=True`` may be extremely slow
-        on some systems, and should be avoided when reading from remote storage.
+        when no global ``_metadata`` file is present, especially when reading
+        from remote storage. Set this to ``True`` only when known divisions
+        are needed for your workload (see :ref:`dataframe-design-partitions).
     ignore_metadata_file : bool, default False
         Whether to ignore the global ``_metadata`` file (when one is present).
         If ``True``, or if the global ``_metadata`` file is missing, the parquet


Test patch: