ENH: Implement multi-column `DataFrame.quantiles`
State: closed

Rough attempt at implementing cuDF's `DataFrame.quantiles`; shares a lot of common logic with `sort_values`, as the indexer that sorts the dataframe by all columns is ultimately what is used to grab the desired quantiles.

cc @quasiben @rjzamora

- [ ] closes #43881
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Enhancement

Comments:
@TomAugspurger this is an attempt at trying to get multi-column sorting working in Dask which requires a multi-column quantile
I'm not sure about the name... I worry about having both a `.quantile` and `quantiles`, and I don't think that quantile**s** really conveys what's different about this method. To me the plural version sounds like you're able to compute multiple quantiles at once, which is already supported by passing a list to `quantile`.

So if we do this, I'd suggest a name like `multi_column_quantile` or `multi_quantile` or `quantile_table` (the last one mirrors a proposed API for `groupby.agg` on a table). Or perhaps we make this a keyword for `quantile`, like `table=True` or `columnar=False`? Because IIUC the output type (Series or Frame) and labels will be identical `.quantile`, the only difference is the values?




 
@shwina, do you think we could alias `quantiles` in cuDF if we do end up changing the name here ?
Yes -- that should be OK for cuDF. I also like `multi_quantile` or `quantile_table` over `quantiles` 
Cool! In that case I'm going to rename this method `multi_quantile`, and we can follow up with a rename / aliasing on cuDF?
@charlesbluca this hasn't received any scruity yet. -1 on adding methods directly like this.
> -1 on adding methods directly like this.

@jreback - Is your preference to add a new option to the existing `quantile` method. Or are you saying that down-stream libraries should be implementing logic like this themselves?
i would add the argument `method='single|table'` which is what we do for `.rolling` and friends
Currently blocked on handling for sparse arrays - I am using `BlockManager.take` to perform the actual quantile operation, which retains sparse dtypes where `BlockManager.quantile` would not. This causes issues when handling non-list-like `qs` as an `iloc` across columns is required, which does not seem to be possible with sparse dtypes:

```python
    def test_quantile_sparse(self, df, expected):
        # GH#17198
        # GH#24600
>       result = df.quantile(interpolation="nearest", method="table")

pandas/tests/frame/methods/test_quantile.py:591: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/core/frame.py:10475: in quantile
    return res.iloc[0]
pandas/core/indexing.py:957: in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
pandas/core/indexing.py:1509: in _getitem_axis
    return self.obj._ixs(key, axis=axis)
pandas/core/frame.py:3483: in _ixs
    new_values = self._mgr.fast_xs(i)
pandas/core/internals/managers.py:974: in fast_xs
    result = cls._empty((n,), dtype=dtype)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'pandas.core.arrays.sparse.array.SparseArray'>, shape = (2,), dtype = Sparse[int64, 0]

    @classmethod
    def _empty(cls, shape: Shape, dtype: ExtensionDtype):
        """
        Create an ExtensionArray with the given shape and dtype.
        """
        obj = cls._from_sequence([], dtype=dtype)
    
        taker = np.broadcast_to(np.intp(-1), shape)
        result = obj.take(taker, allow_fill=True)
        if not isinstance(result, cls) or dtype != result.dtype:
>           raise NotImplementedError(
                f"Default 'empty' implementation is invalid for dtype='{dtype}'"
            )
E           NotImplementedError: Default 'empty' implementation is invalid for dtype='Sparse[int64, 0]'

pandas/core/arrays/base.py:1492: NotImplementedError
```

A few follow up questions here:

- Do we want `quantile(method="table")` to not retain sparse dtypes to match `quantile(method="single")`?
- If not, are there any blockers to `iloc`ing across sparse columns that can be addressed to unblock this PR?
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Still interested in working on this, currently blocked by the handling for sparse arrays, specifically if want to retain sparse dtypes for `quantile(method="table")` 
@TomAugspurger if you get a chance could I get a follow-up review on this? Also a little confused by the test failures here - in trying to resolve the single test failures with sparse dtypes in runs like [this](https://github.com/pandas-dev/pandas/runs/4807168239?check_suite_focus=true), I seem to create a [lot more failures](https://github.com/pandas-dev/pandas/runs/4822740902?check_suite_focus=true) that I'm unable to replicate locally.
@charlesbluca if you can merge master and we'll have a look
@jreback done
@jreback @TomAugspurger could I get a follow up review on this if either of you get the chance?
thanks @charlesbluca really nice! (and @mroeschke for pushing over the line)