Repository: dask/dask

Issue:
`quantile` calculation using dask default method gives incorrect results
This is similar to #731 but reveals a broader issue and thus I would like to create a new issue for it. I'm using `dask==2.24.0` and `pandas==1.0.5`
```
import pandas as pd
import dask.dataframe as dd
x = pd.DataFrame({'a': [1,2,3]})
xd = dd.from_pandas(x, npartitions=3)
print(x['a'].quantile(0.5))
print(xd['a'].quantile(0.5).compute())
```

This difference persists as I try all natural number column `'a'` with length greater than 2 (i.e. `'a'` being natural numbers 1,...,N where N>2).

```
for i in range(1, 100):
    x = pd.DataFrame({'a': list(range(i))})
    xd = dd.from_pandas(x, npartitions=i+1)
    pd_quantile = x['a'].quantile(0.5)
    dd_quantile = xd['a'].quantile(0.5).compute()
    if pd_quantile == dd_quantile:
        print('same')
    else:
        print('different')
```

Using `method='tdigest'` does solve the issue, but the default algorithm shouldn't be giving a wrong answer either..


Discussion:
How is this different than #731?
@TomAugspurger It's different in the sense that this error actually happens for simple cases as opposed to just the edge cases described in #731
Thanks. Do you think that fixing #731 is likely to fix this as well?

And I haven't looked closely at the root cause, but does having as many partitions as points make this problem likely to occur? I think that'd be uncommon in practice.
@TomAugspurger I think the root cause for this and #731 is the same but I would like to emphasize on how prevalent it is. I just did another test with random numbers. The conclusion is:
- the quantile calculation in dask dataframe is only correct when `npartitions=1`
- the average absolute error of dask's quantile calculation declines as the ratio of (number of data points / number of partitions) increases, i.e. either when data size increases with fixed partition size, or when number of partition decreases for fixed amount of data

I think we can conclude that there must be something wrong with the reduce step for the quantile calculation.

Here is the code I used to test it:
```python
import pandas as pd
import numpy as np
import dask.dataframe as dd

# for fixed npartitions, see the effect of increasing dataset size

data_sizes = [2 * 10**i for i in range(5)]
npartitions = 2
size_res = {}
for s in data_sizes:
    # 100 experiments for each data size
    for j in range(100):
        size_res[s] = []
        x = pd.DataFrame({'a': np.random.uniform(size=s)})
        xd = dd.from_pandas(x, npartitions=npartitions)
        xm = x['a'].quantile()
        xdm = xd['a'].quantile().compute()
        size_res[s].append(xm-xdm)

# Calculate the mean absolute error of dask's quantile calculation at each dataset size
size_mae = {}
for s in size_res:
    size_mae[s] = np.mean(np.abs(size_res[s]))

#size_mae={
#    2: 0.0,
#    20: 0.023239827911832822,
#    200: 0.011693968030884405,
#    2000: 0.030041772340274497,
#    20000: 0.0055823681147273985
#}

# For fixed data size, see the effect of increasing partitions

data_size = 10000
npartitions = [4**i for i in range(7)]
npar_res = {}
for npar in npartitions:
    # 100 experiments for each partition size
    for j in range(100):
        npar_res[npar] = []
        x = pd.DataFrame({'a': np.random.uniform(size=data_size)})
        xd = dd.from_pandas(x, npartitions=npar)
        xm = x['a'].quantile()
        xdm = xd['a'].quantile().compute()
        npar_res[npar].append(xm-xdm)


# Calculate the mean absolute error of dask's quantile calculation at each npartitions
npar_mae = {}
for s in npar_res:
    npar_mae[s] = np.mean(np.abs(npar_res[s]))

# npar_mae={
#    1: 0.0,
#    4: 0.00337763961928661,
#    16: 0.02724938859132664,
#    64: 0.11335921282273076,
#    256: 0.21650419213080002,
#    1024: 0.3861034689856373,
#    4096: 0.4905171995922507
#}

```
Even further interesting results - The ordering of the original array matters (i.e. when the data in each partition is different). I think this serves another evidence that the problem is in the reduction step

```python
res = {}
base = np.array([0,0,0,0,0,1,1])
cur = list(base.copy())
while (not (np.array(cur) == base).all() or len(res) == 0):
    x = pd.DataFrame({'a': cur})
    xd = dd.from_pandas(x, npartitions=3)
    xm = x['a'].quantile()
    xdm = xd['a'].quantile().compute()
    res[','.join([str(j) for j in cur])] = xm - xdm
    cur.insert(0, cur.pop())

```

![image](https://user-images.githubusercontent.com/3460073/91253129-234cc600-e72d-11ea-9169-fda54a1b3735.png)

Neat.  I'm curious, how do things behave if we always include 0 and 1 in the quantiles to calculate?
```python
In [17]: xd['a'].quantile([0.5]).compute()
Out[17]:
0.5    2.5
Name: a, dtype: float64

In [18]: xd['a'].quantile([0, 0.5, 1]).compute()
Out[18]:
0.0    1.0
0.5    2.0
1.0    3.0
Name: a, dtype: float64
```
Adding 0 and 1 internally (if necessary) could be an easy fix for edge cases such as these, and should generally make quantile calculations better.

Resolving patch:
diff --git a/dask/array/percentile.py b/dask/array/percentile.py
--- a/dask/array/percentile.py
+++ b/dask/array/percentile.py
@@ -126,10 +126,12 @@ def percentile(a, q, interpolation="linear", method="default"):
 
     # Otherwise use the custom percentile algorithm
     else:
-
+        # Add 0 and 100 during calculation for more robust behavior (hopefully)
+        calc_q = np.pad(q, 1, mode="constant")
+        calc_q[-1] = 100
         name = "percentile_chunk-" + token
         dsk = dict(
-            ((name, i), (_percentile, key, q, interpolation))
+            ((name, i), (_percentile, key, calc_q, interpolation))
             for i, key in enumerate(a.__dask_keys__())
         )
 
@@ -138,7 +140,7 @@ def percentile(a, q, interpolation="linear", method="default"):
             (name2, 0): (
                 merge_percentiles,
                 q,
-                [q] * len(a.chunks[0]),
+                [calc_q] * len(a.chunks[0]),
                 sorted(dsk),
                 interpolation,
             )
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -5457,16 +5457,19 @@ def quantile(df, q, method="default"):
 
         from dask.array.percentile import _percentile, merge_percentiles
 
+        # Add 0 and 100 during calculation for more robust behavior (hopefully)
+        calc_qs = np.pad(qs, 1, mode="constant")
+        calc_qs[-1] = 100
         name = "quantiles-1-" + token
         val_dsk = {
-            (name, i): (_percentile, (getattr, key, "values"), qs)
+            (name, i): (_percentile, (getattr, key, "values"), calc_qs)
             for i, key in enumerate(df.__dask_keys__())
         }
 
         name2 = "quantiles-2-" + token
         merge_dsk = {
             (name2, 0): finalize_tsk(
-                (merge_percentiles, qs, [qs] * df.npartitions, sorted(val_dsk))
+                (merge_percentiles, qs, [calc_qs] * df.npartitions, sorted(val_dsk))
             )
         }
     dsk = merge(val_dsk, merge_dsk)


Test patch:
diff --git a/dask/array/tests/test_array_function.py b/dask/array/tests/test_array_function.py
--- a/dask/array/tests/test_array_function.py
+++ b/dask/array/tests/test_array_function.py
@@ -179,7 +179,13 @@ def test_non_existent_func():
 
 @pytest.mark.skipif(missing_arrfunc_cond, reason=missing_arrfunc_reason)
 @pytest.mark.parametrize(
-    "func", [np.equal, np.matmul, np.dot, lambda x, y: np.stack([x, y])]
+    "func",
+    [
+        np.equal,
+        np.matmul,
+        np.dot,
+        lambda x, y: np.stack([x, y]),
+    ],
 )
 @pytest.mark.parametrize(
     "arr_upcast, arr_downcast",
diff --git a/dask/array/tests/test_percentiles.py b/dask/array/tests/test_percentiles.py
--- a/dask/array/tests/test_percentiles.py
+++ b/dask/array/tests/test_percentiles.py
@@ -80,6 +80,15 @@ def test_percentiles_with_empty_arrays(method):
     )
 
 
+@percentile_methods
+def test_percentiles_with_empty_q(method):
+    x = da.ones(10, chunks=((5, 0, 5),))
+    assert_eq(
+        da.percentile(x, [], method=method),
+        np.array([], dtype=x.dtype),
+    )
+
+
 @percentile_methods
 @pytest.mark.parametrize("q", [5, 5.0, np.int64(5), np.float64(5)])
 def test_percentiles_with_scaler_percentile(method, q):
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -1190,7 +1190,7 @@ def test_nbytes():
 
 @pytest.mark.parametrize(
     "method,expected",
-    [("tdigest", (0.35, 3.80, 2.5, 6.5, 2.0)), ("dask", (0.0, 5.4, 1.2, 7.8, 5.0))],
+    [("tdigest", (0.35, 3.80, 2.5, 6.5, 2.0)), ("dask", (0.0, 4.0, 1.2, 6.2, 2.0))],
 )
 def test_quantile(method, expected):
     if method == "tdigest":
@@ -1278,9 +1278,9 @@ def test_empty_quantile(method):
         (
             "dask",
             (
-                pd.Series([16.5, 36.5, 26.5], index=["A", "X", "B"]),
+                pd.Series([7.0, 27.0, 17.0], index=["A", "X", "B"]),
                 pd.DataFrame(
-                    [[1.50, 21.50, 11.50], [17.75, 37.75, 27.75]],
+                    [[1.50, 21.50, 11.50], [14.0, 34.0, 24.0]],
                     index=[0.25, 0.75],
                     columns=["A", "X", "B"],
                 ),
@@ -1355,6 +1355,14 @@ def test_quantile_for_possibly_unsorted_q():
     assert_eq(r, 25.0)
 
 
+def test_quantile_tiny_partitions():
+    """ See https://github.com/dask/dask/issues/6551 """
+    df = pd.DataFrame({"a": [1, 2, 3]})
+    ddf = dd.from_pandas(df, npartitions=3)
+    r = ddf["a"].quantile(0.5).compute()
+    assert r == 2
+
+
 def test_index():
     assert_eq(d.index, full.index)