Repository: pandas-dev/pandas

Issue:
Groupby Array-Type Quantiles Broken in 0.25.0
#### Code Sample

```python
import pandas as pd

df = pd.DataFrame({
    'category': ['A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B'],
    'value': [1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6]
})
quantiles = df.groupby('category').quantile([0.25, 0.5, 0.75])
print(quantiles)
```
#### Problem description

In previous versions of Pandas `< 0.25.0`  and in the documentation it is possible to pass an array-type of quantiles into the `DataFrameGroupBy.quantile()` method to return multiple quantile values in a single call. However, upon installation of `0.25.0` the following error results instead:

```
Traceback (most recent call last):
  File "example.py", line 8, in <module>
    quantiles = df.groupby('category').quantile([0.25, 0.5, 0.75])
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 1908, in quantile
    interpolation=interpolation,
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 2248, in _get_cythonized_result
    func(**kwargs)  # Call func to modify indexer values in place
  File "pandas/_libs/groupby.pyx", line 69
```

#### Expected Output

Using Pandas `0.24.2` the output is:

```
               value
category
A        0.25   2.25
         0.50   3.50
         0.75   4.75
B        0.25   2.25
         0.50   3.50
         0.75   4.75
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.125-linuxkit
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : None
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : None
pandas_datareader: None
bs4              : 4.8.0
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : 2.6.9
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : 0.3.0
scipy            : 1.3.0
sqlalchemy       : None
tables           : 3.5.2
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>



Discussion:
I got this error  message when using a numpy array (form np.linspace()):

    TypeError: only size-1 arrays can be converted to Python scalars

Downgrade to pandas 0.24 solves this.

my test code (snipplet):

        percs = (np.linspace(0, 1, num=intervals + 1).round(decimals=3))
        d = df[['x', 'y']]
        g = d.groupby('x')
        quants = g.quantile(percs)

breaks in last line with 0.25, works in 0.24

there is a PR #27473 which solves this and just needs some touching up to fix
That PR was about #20405 not validating inputs. This issue is about #20405 deleting functionality so different bugs.
Is the fix to change https://github.com/pandas-dev/pandas/blob/c0ff67a22df9c18da1172766e313732ed2ab6c30/pandas/core/groupby/groupby.py#L1900-L1911 to be called once per value in `q`, when a list of quintiles is provide? Then concat the results together with `concat(results, axis=1, keys=q)`?
The output of `DataFrameGroupBy.quantile` is a DataFrame whose 

* index is the group keys
* columns are the (numeric) columns

```python
In [68]: df = pd.DataFrame({"A": [0, 1, 2, 3, 4]})

In [69]: df.groupby([0, 0, 1, 1, 1]).quantile(0.25)
Out[69]:
      A
0  0.25
```

What's the expected output of `.quantile(List[float])`?

It's not the most useful, but I think the best option is a MultiIndex in the columns.

```python
In [70]: a = df.iloc[:2].quantile([0.25]).unstack()

In [71]: b = df.iloc[2:].quantile([0.25]).unstack()

In [72]: pd.concat([a, b], keys=[0, 1]).unstack([1, 2])
Out[72]:
      A
   0.25
0  0.25
1  2.50

```

The other option is to have the `q`s in the index, but that breaks my mental model that the index should be the unique group keys.


Oh, whoops, I missed the 0.24 output. We'll match that.

Resolving patch:
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -1870,6 +1870,7 @@ def quantile(self, q=0.5, interpolation="linear"):
         a    2.0
         b    3.0
         """
+        from pandas import concat
 
         def pre_processor(vals: np.ndarray) -> Tuple[np.ndarray, Optional[Type]]:
             if is_object_dtype(vals):
@@ -1897,18 +1898,57 @@ def post_processor(vals: np.ndarray, inference: Optional[Type]) -> np.ndarray:
 
             return vals
 
-        return self._get_cythonized_result(
-            "group_quantile",
-            self.grouper,
-            aggregate=True,
-            needs_values=True,
-            needs_mask=True,
-            cython_dtype=np.float64,
-            pre_processing=pre_processor,
-            post_processing=post_processor,
-            q=q,
-            interpolation=interpolation,
-        )
+        if is_scalar(q):
+            return self._get_cythonized_result(
+                "group_quantile",
+                self.grouper,
+                aggregate=True,
+                needs_values=True,
+                needs_mask=True,
+                cython_dtype=np.float64,
+                pre_processing=pre_processor,
+                post_processing=post_processor,
+                q=q,
+                interpolation=interpolation,
+            )
+        else:
+            results = [
+                self._get_cythonized_result(
+                    "group_quantile",
+                    self.grouper,
+                    aggregate=True,
+                    needs_values=True,
+                    needs_mask=True,
+                    cython_dtype=np.float64,
+                    pre_processing=pre_processor,
+                    post_processing=post_processor,
+                    q=qi,
+                    interpolation=interpolation,
+                )
+                for qi in q
+            ]
+            result = concat(results, axis=0, keys=q)
+            # fix levels to place quantiles on the inside
+            # TODO(GH-10710): Ideally, we could write this as
+            #  >>> result.stack(0).loc[pd.IndexSlice[:, ..., q], :]
+            #  but this hits https://github.com/pandas-dev/pandas/issues/10710
+            #  which doesn't reorder the list-like `q` on the inner level.
+            order = np.roll(list(range(result.index.nlevels)), -1)
+            result = result.reorder_levels(order)
+            result = result.reindex(q, level=-1)
+
+            # fix order.
+            hi = len(q) * self.ngroups
+            arr = np.arange(0, hi, self.ngroups)
+            arrays = []
+
+            for i in range(self.ngroups):
+                arr = arr + i
+                arrays.append(arr)
+
+            indices = np.concatenate(arrays)
+            assert len(indices) == len(result)
+            return result.take(indices)
 
     @Substitution(name="groupby")
     def ngroup(self, ascending=True):


Test patch:
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1238,6 +1238,57 @@ def test_quantile(interpolation, a_vals, b_vals, q):
     tm.assert_frame_equal(result, expected)
 
 
+def test_quantile_array():
+    # https://github.com/pandas-dev/pandas/issues/27526
+    df = pd.DataFrame({"A": [0, 1, 2, 3, 4]})
+    result = df.groupby([0, 0, 1, 1, 1]).quantile([0.25])
+
+    index = pd.MultiIndex.from_product([[0, 1], [0.25]])
+    expected = pd.DataFrame({"A": [0.25, 2.50]}, index=index)
+    tm.assert_frame_equal(result, expected)
+
+    df = pd.DataFrame({"A": [0, 1, 2, 3], "B": [4, 5, 6, 7]})
+    index = pd.MultiIndex.from_product([[0, 1], [0.25, 0.75]])
+
+    result = df.groupby([0, 0, 1, 1]).quantile([0.25, 0.75])
+    expected = pd.DataFrame(
+        {"A": [0.25, 0.75, 2.25, 2.75], "B": [4.25, 4.75, 6.25, 6.75]}, index=index
+    )
+    tm.assert_frame_equal(result, expected)
+
+
+def test_quantile_array_no_sort():
+    df = pd.DataFrame({"A": [0, 1, 2], "B": [3, 4, 5]})
+    result = df.groupby([1, 0, 1], sort=False).quantile([0.25, 0.5, 0.75])
+    expected = pd.DataFrame(
+        {"A": [0.5, 1.0, 1.5, 1.0, 1.0, 1.0], "B": [3.5, 4.0, 4.5, 4.0, 4.0, 4.0]},
+        index=pd.MultiIndex.from_product([[1, 0], [0.25, 0.5, 0.75]]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+    result = df.groupby([1, 0, 1], sort=False).quantile([0.75, 0.25])
+    expected = pd.DataFrame(
+        {"A": [1.5, 0.5, 1.0, 1.0], "B": [4.5, 3.5, 4.0, 4.0]},
+        index=pd.MultiIndex.from_product([[1, 0], [0.75, 0.25]]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+
+def test_quantile_array_multiple_levels():
+    df = pd.DataFrame(
+        {"A": [0, 1, 2], "B": [3, 4, 5], "c": ["a", "a", "a"], "d": ["a", "a", "b"]}
+    )
+    result = df.groupby(["c", "d"]).quantile([0.25, 0.75])
+    index = pd.MultiIndex.from_tuples(
+        [("a", "a", 0.25), ("a", "a", 0.75), ("a", "b", 0.25), ("a", "b", 0.75)],
+        names=["c", "d", None],
+    )
+    expected = pd.DataFrame(
+        {"A": [0.25, 0.75, 2.0, 2.0], "B": [3.25, 3.75, 5.0, 5.0]}, index=index
+    )
+    tm.assert_frame_equal(result, expected)
+
+
 def test_quantile_raises():
     df = pd.DataFrame(
         [["foo", "a"], ["foo", "b"], ["foo", "c"]], columns=["key", "val"]