qcut: Option to return -inf/inf as lower/upper bound
State: closed

- [x] closes #17282
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=h1) Report
> Merging [#22185](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/6d3565a5181b3d0fb8342e69dd1d21a98ac3d380?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/22185/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #22185      +/-   ##
==========================================
+ Coverage   92.38%   92.38%   +<.01%     
==========================================
  Files         166      166              
  Lines       52363    52380      +17     
==========================================
+ Hits        48377    48393      +16     
- Misses       3986     3987       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.81% <100%> (ø)` | :arrow_up: |
| #single | `42.91% <16.66%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape/tile.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS90aWxlLnB5) | `95.08% <100%> (+0.13%)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `88% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvb3BzLnB5) | `94.28% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `95.27% <0%> (ø)` | :arrow_up: |
| [pandas/core/arrays/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2ludGVydmFsLnB5) | `93.12% <0%> (+0.03%)` | :arrow_up: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/22185/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.97% <0%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=footer). Last update [6d3565a...c2f194c](https://codecov.io/gh/pandas-dev/pandas/pull/22185?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Checking in on this. How do I move this forward?
@dberenbaum : So sorry!  Didn't realize this one fell through the cracks.

cc @jreback @jorisvandenbossche @TomAugspurger : This one has been ready to go IMO **for over a month**.  Could one of you take a look?
I am not sure I like this, we are growing too many keywords here. Now we have 2 ways of specifying the bins. Is this really a problem to use np.inf in the bins specification?
I'm not following how there are 2 ways of specifying the bins. I only see a way to specify quantiles in `qcut` (there is a bins arg in `cut`). I'm not sure how else to return `np.inf` as a bin edge using `qcut`.

The use cases I have are very similar to the one raised by @prcastro in #17282. When I need to 1) compare distributions or 2) bin numerical values into known categories, I use `qcut` to bin by quantiles and then bin the new distribution by those categories. This is problematic when the new distribution contains out of bounds values. The best alternative I've found is to avoid using `qcut`:

```
bins = np.quantile(x, [0, 0.25, 0.5, 0.75, 1])
bins[0] = -np.inf
bins[-1] = np.inf
pd.cut(x, bins)
```
@dberenbaum so I this for ``.cut()`` this is not needed at all as a user can just specify ``np.inf`` as a bound, for ``.qcut()`` its between 0 and 1, the quantiles. So can you give a case where this is actually useful?
Please see #17282 for an example of how this is useful. That example is very similar to the cases where I would have found this useful. Does that example clarify the intended usage?
@dberenbaum did you see my comment. ``qcut`` is for ``quantile`` cutting, not arbitrary bounds, where ``cut`` suffices.
@jreback I saw your comment. Did you look at the example in #17282?

Let's say I use `qcut` to bin a sample of data collected into quartiles:

```
>>>q = pd.qcut([1,2,3,4,5,6,7,8,9], 4)
>>>q
[(0.999, 3.0], (0.999, 3.0], (0.999, 3.0], (3.0, 5.0], (3.0, 5.0], (5.0, 7.0], (5.0, 7.0], (7.0, 9.0], (7.0, 9.0]]
Categories (4, interval[float64]): [(0.999, 3.0] < (3.0, 5.0] < (5.0, 7.0] < (7.0, 9.0]]
```

I subsequently collect a new data sample, which contains values > 9. I want to compare the new sample to my original data using the returned quartiles. There is no quartile for the new values > 9. With the `bounded` option, I could do the following:

```
>>>q = pd.qcut([1,2,3,4,5,6,7,8,9], 4, bounded=False)
>>>q
[(-np.inf, 3.0], (0.999, 3.0], (0.999, 3.0], (3.0, 5.0], (3.0, 5.0], (5.0, 7.0], (5.0, 7.0], (7.0, 9.0], (7.0, np.inf]]
Categories (4, interval[float64]): [(-np.inf, 3.0] < (3.0, 5.0] < (5.0, 7.0] < (7.0, np.inf]]
```

Now, I have useful quartiles for any subsequent code that uses the output of `qcut`, even if I'm applying those quartiles to new data that falls outside the range of the original sample.

The `bounded` behavior might not always be desirable, but I've found it to at least be a useful option.
@dberenbaum ok I looked at this again and it is ok, can you merge master
Closing this PR per my comment above.