Repository: pandas-dev/pandas
Pull request: BUG: Fixed groupby quantile for listlike q
Closes #27526

Just a WIP right now. This is under tested for 

- [ ] `groupby(sort=False)`
- [ ] non-sorted `q`
- [ ] `groupby(multiple levels)`

Problem statement:
Groupby Array-Type Quantiles Broken in 0.25.0
#### Code Sample

```python
import pandas as pd

df = pd.DataFrame({
    'category': ['A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'B', 'B'],
    'value': [1, 2, 3, 4, 5, 6, 1, 2, 3, 4, 5, 6]
})
quantiles = df.groupby('category').quantile([0.25, 0.5, 0.75])
print(quantiles)
```
#### Problem description

In previous versions of Pandas `< 0.25.0`  and in the documentation it is possible to pass an array-type of quantiles into the `DataFrameGroupBy.quantile()` method to return multiple quantile values in a single call. However, upon installation of `0.25.0` the following error results instead:

```
Traceback (most recent call last):
  File "example.py", line 8, in <module>
    quantiles = df.groupby('category').quantile([0.25, 0.5, 0.75])
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 1908, in quantile
    interpolation=interpolation,
  File "/usr/local/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 2248, in _get_cythonized_result
    func(**kwargs)  # Call func to modify indexer values in place
  File "pandas/_libs/groupby.pyx", line 69
```

#### Expected Output

Using Pandas `0.24.2` the output is:

```
               value
category
A        0.25   2.25
         0.50   3.50
         0.75   4.75
B        0.25   2.25
         0.50   3.50
         0.75   4.75
```

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.9.125-linuxkit
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : None
pytest           : 5.0.1
hypothesis       : None
sphinx           : 2.1.2
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : None
pandas_datareader: None
bs4              : 4.8.0
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.1
numexpr          : 2.6.9
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : 0.3.0
scipy            : 1.3.0
sqlalchemy       : None
tables           : 3.5.2
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
I got this error  message when using a numpy array (form np.linspace()):

    TypeError: only size-1 arrays can be converted to Python scalars

Downgrade to pandas 0.24 solves this.

my test code (snipplet):

        percs = (np.linspace(0, 1, num=intervals + 1).round(decimals=3))
        d = df[['x', 'y']]
        g = d.groupby('x')
        quants = g.quantile(percs)

breaks in last line with 0.25, works in 0.24

there is a PR #27473 which solves this and just needs some touching up to fix
That PR was about #20405 not validating inputs. This issue is about #20405 deleting functionality so different bugs.
Is the fix to change https://github.com/pandas-dev/pandas/blob/c0ff67a22df9c18da1172766e313732ed2ab6c30/pandas/core/groupby/groupby.py#L1900-L1911 to be called once per value in `q`, when a list of quintiles is provide? Then concat the results together with `concat(results, axis=1, keys=q)`?
The output of `DataFrameGroupBy.quantile` is a DataFrame whose 

* index is the group keys
* columns are the (numeric) columns

```python
In [68]: df = pd.DataFrame({"A": [0, 1, 2, 3, 4]})

In [69]: df.groupby([0, 0, 1, 1, 1]).quantile(0.25)
Out[69]:
      A
0  0.25
```

What's the expected output of `.quantile(List[float])`?

It's not the most useful, but I think the best option is a MultiIndex in the columns.

```python
In [70]: a = df.iloc[:2].quantile([0.25]).unstack()

In [71]: b = df.iloc[2:].quantile([0.25]).unstack()

In [72]: pd.concat([a, b], keys=[0, 1]).unstack([1, 2])
Out[72]:
      A
   0.25
0  0.25
1  2.50

```

The other option is to have the `q`s in the index, but that breaks my mental model that the index should be the unique group keys.


Oh, whoops, I missed the 0.24 output. We'll match that.
Thanks for the fix. I just ran into this!

Commit message: Remove dtype selection

Patch to review:
diff --git a/doc/source/whatsnew/v0.25.1.rst b/doc/source/whatsnew/v0.25.1.rst
index b97f45efbeae9..34b9d2f138692 100644
--- a/doc/source/whatsnew/v0.25.1.rst
+++ b/doc/source/whatsnew/v0.25.1.rst
@@ -118,6 +118,7 @@ Plotting
 Groupby/resample/rolling
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
+- Fixed regression in :meth:`pands.core.groupby.DataFrameGroupBy.quantile` raising when multiple quantiles are given (:issue:`27526`)
 - Bug in :meth:`pandas.core.groupby.DataFrameGroupBy.transform` where applying a timezone conversion lambda function would drop timezone information (:issue:`27496`)
 - Bug in windowing over read-only arrays (:issue:`27766`)
 - Fixed segfault in `pandas.core.groupby.DataFrameGroupBy.quantile` when an invalid quantile was passed (:issue:`27470`)
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index c5e81e21e9fd5..c00bd83982c8f 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -1870,6 +1870,7 @@ def quantile(self, q=0.5, interpolation="linear"):
         a    2.0
         b    3.0
         """
+        from pandas import concat
 
         def pre_processor(vals: np.ndarray) -> Tuple[np.ndarray, Optional[Type]]:
             if is_object_dtype(vals):
@@ -1897,18 +1898,53 @@ def post_processor(vals: np.ndarray, inference: Optional[Type]) -> np.ndarray:
 
             return vals
 
-        return self._get_cythonized_result(
-            "group_quantile",
-            self.grouper,
-            aggregate=True,
-            needs_values=True,
-            needs_mask=True,
-            cython_dtype=np.float64,
-            pre_processing=pre_processor,
-            post_processing=post_processor,
-            q=q,
-            interpolation=interpolation,
-        )
+        if is_scalar(q):
+            return self._get_cythonized_result(
+                "group_quantile",
+                self.grouper,
+                aggregate=True,
+                needs_values=True,
+                needs_mask=True,
+                cython_dtype=np.float64,
+                pre_processing=pre_processor,
+                post_processing=post_processor,
+                q=q,
+                interpolation=interpolation,
+            )
+        else:
+            results = [
+                self._get_cythonized_result(
+                    "group_quantile",
+                    self.grouper,
+                    aggregate=True,
+                    needs_values=True,
+                    needs_mask=True,
+                    cython_dtype=np.float64,
+                    pre_processing=pre_processor,
+                    post_processing=post_processor,
+                    q=qi,
+                    interpolation=interpolation,
+                )
+                for qi in q
+            ]
+            # fix levels to place quantiles on the inside
+            result = concat(results, axis=0, keys=q)
+            order = np.roll(list(range(result.index.nlevels)), -1)
+            result = result.reorder_levels(order)
+            result = result.reindex(q, level=-1)
+
+            # fix order.
+            hi = len(q) * self.ngroups
+            arr = np.arange(0, hi, self.ngroups)
+            arrays = []
+
+            for i in range(self.ngroups):
+                arr = arr + i
+                arrays.append(arr)
+
+            indices = np.concatenate(arrays)
+            assert len(indices) == len(result)
+            return result.take(indices)
 
     @Substitution(name="groupby")
     def ngroup(self, ascending=True):
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
index 3794120281e1f..509d7c33b643b 100644
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1238,6 +1238,57 @@ def test_quantile(interpolation, a_vals, b_vals, q):
     tm.assert_frame_equal(result, expected)
 
 
+def test_quantile_array():
+    # https://github.com/pandas-dev/pandas/issues/27526
+    df = pd.DataFrame({"A": [0, 1, 2, 3, 4]})
+    result = df.groupby([0, 0, 1, 1, 1]).quantile([0.25])
+
+    index = pd.MultiIndex.from_product([[0, 1], [0.25]])
+    expected = pd.DataFrame({"A": [0.25, 2.50]}, index=index)
+    tm.assert_frame_equal(result, expected)
+
+    df = pd.DataFrame({"A": [0, 1, 2, 3], "B": [4, 5, 6, 7]})
+    index = pd.MultiIndex.from_product([[0, 1], [0.25, 0.75]])
+
+    result = df.groupby([0, 0, 1, 1]).quantile([0.25, 0.75])
+    expected = pd.DataFrame(
+        {"A": [0.25, 0.75, 2.25, 2.75], "B": [4.25, 4.75, 6.25, 6.75]}, index=index
+    )
+    tm.assert_frame_equal(result, expected)
+
+
+def test_quantile_array_no_sort():
+    df = pd.DataFrame({"A": [0, 1, 2], "B": [3, 4, 5]})
+    result = df.groupby([1, 0, 1], sort=False).quantile([0.25, 0.5, 0.75])
+    expected = pd.DataFrame(
+        {"A": [0.5, 1.0, 1.5, 1.0, 1.0, 1.0], "B": [3.5, 4.0, 4.5, 4.0, 4.0, 4.0]},
+        index=pd.MultiIndex.from_product([[1, 0], [0.25, 0.5, 0.75]]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+    result = df.groupby([1, 0, 1], sort=False).quantile([0.75, 0.25])
+    expected = pd.DataFrame(
+        {"A": [1.5, 0.5, 1.0, 1.0], "B": [4.5, 3.5, 4.0, 4.0]},
+        index=pd.MultiIndex.from_product([[1, 0], [0.75, 0.25]]),
+    )
+    tm.assert_frame_equal(result, expected)
+
+
+def test_quantile_array_multiple_levels():
+    df = pd.DataFrame(
+        {"A": [0, 1, 2], "B": [3, 4, 5], "c": ["a", "a", "a"], "d": ["a", "a", "b"]}
+    )
+    result = df.groupby(["c", "d"]).quantile([0.25, 0.75])
+    index = pd.MultiIndex.from_tuples(
+        [("a", "a", 0.25), ("a", "a", 0.75), ("a", "b", 0.25), ("a", "b", 0.75)],
+        names=["c", "d", None],
+    )
+    expected = pd.DataFrame(
+        {"A": [0.25, 0.75, 2.0, 2.0], "B": [3.25, 3.75, 5.0, 5.0]}, index=index
+    )
+    tm.assert_frame_equal(result, expected)
+
+
 def test_quantile_raises():
     df = pd.DataFrame(
         [["foo", "a"], ["foo", "b"], ["foo", "c"]], columns=["key", "val"]


Code review comments:
diff_hunk: @@ -1897,18 +1898,53 @@ def post_processor(vals: np.ndarray, inference: Optional[Type]) -> np.ndarray:
 
             return vals
 
-        return self._get_cythonized_result(
-            "group_quantile",
-            self.grouper,
-            aggregate=True,
-            needs_values=True,
-            needs_mask=True,
-            cython_dtype=np.float64,
-            pre_processing=pre_processor,
-            post_processing=post_processor,
-            q=q,
-            interpolation=interpolation,
-        )
+        if is_scalar(q):
+            return self._get_cythonized_result(
+                "group_quantile",
+                self.grouper,
+                aggregate=True,
+                needs_values=True,
+                needs_mask=True,
+                cython_dtype=np.float64,
+                pre_processing=pre_processor,
+                post_processing=post_processor,
+                q=q,
+                interpolation=interpolation,
+            )
+        else:
+            results = [
+                self._get_cythonized_result(
+                    "group_quantile",
+                    self.grouper,
+                    aggregate=True,
+                    needs_values=True,
+                    needs_mask=True,
+                    cython_dtype=np.float64,
+                    pre_processing=pre_processor,
+                    post_processing=post_processor,
+                    q=qi,
+                    interpolation=interpolation,
+                )
+                for qi in q
+            ]
+            # fix levels to place quantiles on the inside
path: pandas/core/groupby/groupby.py
text: @author:
Would welcome improvements to this. Essentially, we have two things to do after concating the results

1. Reorder the levels. By default `concat` places the `keys` on the outermost level, we need it on the innermost
2. Re-sort things. Initially we're sorted by quantile (the `q`-th quantile for every group is together, then the next quantile). We need to be grouped by group key, then quantiles. We can't just use `sort_index`, since we need to preserve the user-provided group key order with `sort=False` and the order of the quintile list.

@user1:
yah this isn't fun.  just checking: we can have mixed dtypes here?  if we had a single dtype then stack/unstack would be cheap and there might be a prettier workaround

@user1:
`concat(results, axis=1, keys=q).stack(0)` gives the same answer for at least one of the test cases

@author:
Unfortunately that sorts the axis. I theory, we should be able to follow it up with a `loc` to restore the correct order on that level,

```diff
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index c00bd8398..617466b8d 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -1928,23 +1928,10 @@ class GroupBy(_GroupBy):
                 for qi in q
             ]
             # fix levels to place quantiles on the inside
-            result = concat(results, axis=0, keys=q)
-            order = np.roll(list(range(result.index.nlevels)), -1)
-            result = result.reorder_levels(order)
-            result = result.reindex(q, level=-1)
-
-            # fix order.
-            hi = len(q) * self.ngroups
-            arr = np.arange(0, hi, self.ngroups)
-            arrays = []
-
-            for i in range(self.ngroups):
-                arr = arr + i
-                arrays.append(arr)
-
-            indices = np.concatenate(arrays)
-            assert len(indices) == len(result)
-            return result.take(indices)
+            result = concat(results, axis=1, keys=q).stack(0)
+            slices = [slice(None)] * result.index.ndim
+            result = result.loc[tuple(slices), :]
+            return result
 
     @user2(name="groupby")
     def ngroup(self, ascending=True):
```

But that runs into the (I think known) issue with loc & a MultiIndex 

```python
In [21]: s = pd.Series(list(range(12)), index=pd.MultiIndex.from_product([['a', 'b', 'c'], [1, 2, 3, 4]]))

In [22]: s.loc[pd.IndexSlice[:, [3, 1, 2, 4]]]
Out[22]:
a  1     0
   2     1
   3     2
   4     3
b  1     4
   2     5
   3     6
   4     7
c  1     8
   2     9
   3    10
   4    11
dtype: int64

```

@user1:
Darn.  Can you add a my-idea-doesnt-work case to the tests