Repository: pandas-dev/pandas
Pull request: Bug groupby quantile listlike q and int columns
When columns are integers, `df.groupby(label).quantile(<arraylike>)` fails.

- [x] closes #30289
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
groupby.quantile(<arraylike>) fails with AssertionError
#### Code Sample, a copy-pastable example if possible

```python
# Your code here
df = pd.DataFrame(np.array([10*[_%4] for _ in range(100)]))            

df.groupby(0).quantile(0.5)                                            
# Out[19]: 
#     1    2    3    4    5    6    7    8    9
# 0                                             
# 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
# 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
# 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
# 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0

df.groupby(0).quantile([0.5,0.99])                                     
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-20-21c92d2481c9> in <module>
----> 1 df.groupby(0).quantile([0.5,0.99])

~/PycharmProjects/netsim_stats/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py in quantile(self, q, interpolation)
   1950 
   1951             indices = np.concatenate(arrays)
-> 1952             assert len(indices) == len(result)
   1953             return result.take(indices)
   1954 

AssertionError: 

df.quantile([0.5,0.99])                                                
#        0    1    2    3    4    5    6    7    8    9
# 0.50  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5  1.5
# 0.99  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
                                                              
df.groupby(0)[1].quantile(0.5) 
# 0
# 0    0.0
# 1    1.0
# 2    2.0
# 3    3.0
# Name: 1, dtype: float64

df.groupby(0)[1].quantile([0.5,0.99])

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-24-ebf6ade716ff> in <module>
----> 1 df.groupby(0)[1].quantile([0.5,0.99])

~/PycharmProjects/netsim_stats/venv/lib/python3.7/site-packages/pandas/core/groupby/groupby.py in quantile(self, q, interpolation)
   1950 
   1951             indices = np.concatenate(arrays)
-> 1952             assert len(indices) == len(result)
   1953             return result.take(indices)
   1954 

AssertionError: 

```
#### Problem description

The above is a constructed minimal example.
I am not sure how much I should elaborate on the "why this is a problem".

groupby.quantile() fails with an assertion error for "larger" dataframes, smaller dataframes seem to work fine.

#### Expected Output

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.5.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.3.15-200.fc30.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : de_DE.UTF-8
LOCALE           : de_DE.UTF-8

pandas           : 0.25.3
numpy            : 1.17.4
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.1.1
setuptools       : 40.8.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.2.5
html5lib         : 1.0.1
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.2.0
pandas_datareader: None
bs4              : 4.6.3
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.2.5
matplotlib       : 3.1.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.1.0
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None


</details>

Hints from issue discussion:
The problem is in the call to `reorder_levels`

```python
In [11]: df.groupby(0).quantile([0.5,0.99])
> /Users/taugspurger/sandbox/pandas/pandas/core/groupby/groupby.py(1941)quantile()
-> order = np.roll(list(range(result.index.nlevels)), -1)
(Pdb) n
> /Users/taugspurger/sandbox/pandas/pandas/core/groupby/groupby.py(1942)quantile()
-> result = result.reorder_levels(order)
(Pdb) result
          1    2    3    4    5    6    7    8    9
     0
0.50 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
     1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
     2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
     3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
0.99 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
     1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
     2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
     3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
(Pdb) order
array([1, 0])
(Pdb) result.reorder_levels(order)
       1    2    3    4    5    6    7    8    9
0 0
0 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
2 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
3 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
0 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1 1  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0  1.0
2 2  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0  2.0
3 3  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0  3.0
```

We're expecting to use level positions there, but one of the index levels is named `0`, the value `0` is interpreted as the label rather than the position.
I don't see a clean way to say that we really mean index levels when the index level names are integers.

```python
In [19]: idx = pd.MultiIndex.from_product([['a', 'b'], [0, 1]], names=['a', 0])

In [20]: idx.reorder_levels([0, 1])
Out[20]:
MultiIndex([(0, 0),
            (1, 1),
            (0, 0),
            (1, 1)],
           )
```

Should we add a keyword to `reorder_levels` to control how that's interpreted?

```python
def reorder_levels(self, order, positional=None):
    """
    positional : bool, optional
        How to interpret integer values in `order`.

          * None (default): prefer treating the values as labels,
            but fall back to positional if no label with that
            value is value.
          * True : only treat integer values as positions.
          * False : only treat integer values as labels.
    """
```

Hi, can I look into this? I will add the "positional" parameter and probably refactor MultiIndex._get_level_number a bit
I added the `positional` parameter in both `DataFrame.reorder_levels` and `MultiIndex.reorder_levels` methods. In `GroupBy.quantile` this `positional` argument is set to True. In all other existing codes `positional` is defaulted to None.

Commit message: CLN: removed unnecessary list comprehension (GH30289)

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 9023cf2ab1b4f..ac4c612f7b019 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -848,6 +848,7 @@ Groupby/resample/rolling
 - Bug in :meth:`DataFrameGroupBy.agg` with timezone-aware datetime64 column incorrectly casting results to the original dtype (:issue:`29641`)
 - Bug in :meth:`DataFrame.groupby` when using axis=1 and having a single level columns index (:issue:`30208`)
 - Bug in :meth:`DataFrame.groupby` when using nunique on axis=1 (:issue:`30253`)
+- Bug in :meth:`GroupBy.quantile` with multiple list-like q value and integer column names (:issue:`30289`)
 
 Reshaping
 ^^^^^^^^^
diff --git a/pandas/core/groupby/groupby.py b/pandas/core/groupby/groupby.py
index 529d123d256e8..227547daf3668 100644
--- a/pandas/core/groupby/groupby.py
+++ b/pandas/core/groupby/groupby.py
@@ -1937,21 +1937,22 @@ def post_processor(vals: np.ndarray, inference: Optional[Type]) -> np.ndarray:
             #  >>> result.stack(0).loc[pd.IndexSlice[:, ..., q], :]
             #  but this hits https://github.com/pandas-dev/pandas/issues/10710
             #  which doesn't reorder the list-like `q` on the inner level.
-            order = np.roll(list(range(result.index.nlevels)), -1)
-            result = result.reorder_levels(order)
-            result = result.reindex(q, level=-1)
+            order = list(range(1, result.index.nlevels)) + [0]
+
+            # temporarily saves the index names
+            index_names = np.array(result.index.names)
 
-            # fix order.
-            hi = len(q) * self.ngroups
-            arr = np.arange(0, hi, self.ngroups)
-            arrays = []
+            # set index names to positions to avoid confusion
+            result.index.names = np.arange(len(index_names))
+
+            # place quantiles on the inside
+            result = result.reorder_levels(order)
 
-            for i in range(self.ngroups):
-                arr2 = arr + i
-                arrays.append(arr2)
+            # restore the index names in order
+            result.index.names = index_names[order]
 
-            indices = np.concatenate(arrays)
-            assert len(indices) == len(result)
+            # reorder rows to keep things sorted
+            indices = np.arange(len(result)).reshape([len(q), self.ngroups]).T.flatten()
             return result.take(indices)
 
     @Substitution(name="groupby")
diff --git a/pandas/tests/groupby/test_function.py b/pandas/tests/groupby/test_function.py
index 4ca23c61ba920..51564154755f4 100644
--- a/pandas/tests/groupby/test_function.py
+++ b/pandas/tests/groupby/test_function.py
@@ -1398,6 +1398,38 @@ def test_quantile_array_multiple_levels():
     tm.assert_frame_equal(result, expected)
 
 
+@pytest.mark.parametrize("frame_size", [(2, 2), (100, 10)])
+@pytest.mark.parametrize("groupby", [[0], [0, 1]])
+@pytest.mark.parametrize("q", [[0.5, 0.6], [0.1, 0.9]])
+def test_groupby_quantile_with_arraylike_q_and_int_columns(frame_size, groupby, q):
+    # GH30289
+    nrow, ncol = frame_size
+    if len(groupby) >= ncol or any(by >= ncol for by in groupby):
+        pytest.skip(f"Invalid argument groupby={groupby}")
+
+    df = pd.DataFrame(
+        np.array([ncol * [_ % 4] for _ in range(nrow)]), columns=range(ncol)
+    )
+
+    idx_levels = [list(range(min(nrow, 4)))] * len(groupby) + [q]
+    idx_codes = [[x for x in range(min(nrow, 4)) for _ in q]] * len(groupby) + [
+        list(range(len(q))) * min(nrow, 4)
+    ]
+    expected_index = pd.MultiIndex(
+        levels=idx_levels, codes=idx_codes, names=groupby + [None]
+    )
+    expected_values = [
+        [float(x)] * (ncol - len(groupby)) for x in range(min(nrow, 4)) for _ in q
+    ]
+    expected_columns = [x for x in range(ncol) if x not in groupby]
+    expected = pd.DataFrame(
+        expected_values, index=expected_index, columns=expected_columns
+    )
+    result = df.groupby(groupby).quantile(q)
+
+    tm.assert_frame_equal(result, expected)
+
+
 def test_quantile_raises():
     df = pd.DataFrame(
         [["foo", "a"], ["foo", "b"], ["foo", "c"]], columns=["key", "val"]


Code review comments:
diff_hunk: @@ -1398,6 +1398,38 @@ def test_quantile_array_multiple_levels():
     tm.assert_frame_equal(result, expected)
 
 
+@pytest.mark.parametrize("frame_size", [(2, 2), (100, 10)])
+@pytest.mark.parametrize("groupby", [[0], [0, 1]])
+@pytest.mark.parametrize("q", [[0.5, 0.6], [0.1, 0.9]])
+def test_groupby_quantile_with_arraylike_q_and_int_columns(frame_size, groupby, q):
+    # GH30289
+    nrow, ncol = frame_size
+    if len(groupby) >= ncol or any(by >= ncol for by in groupby):
+        pytest.skip(f"Invalid argument groupby={groupby}")
path: pandas/tests/groupby/test_function.py
text: @user1:
hmm, what case hits this? can we just make the cases so this doesn't happen

@author:
it hits the skip when the groupby columns are not found or the test used all columns to groupby and nothing left for content (i.e when frame_size=(2, 2) and groupby=[0, 1]). Have cleaned up the test cases a bit so the skip is no longer there