API: add return_inverse to pd.unique
State: closed

- [x] splits off first chunk of #24108; progress towards #4087 / #21357 / #21720 / #22824
- [x] tests expanded / parametrized / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is the first part I'm splitting off of #24108, but now with full test coverage. For the moment, I've added `return_inverse` to `pd.unique` and to `Categorical.unique`, but it's not trivial because of inconsistencies like the following:
```
>>> import pandas as pd
>>> idx = pd.Index([0, 1, 1, 0])
>>> pd.unique(idx)
array([0, 1], dtype=int64)
>>>
>>> # So pd.unique(Index) yields an array, except if the Index is categorical...?
>>> idx = idx.astype('category')
>>> pd.unique(idx)
CategoricalIndex([0, 1], categories=[0, 1], ordered=False, dtype='category')
```

I'd be open to further split off the change for `Categorical.unique`, or just return `NotImplemented` for all `ExtensionArray` types. As mentioned in #24108 already, I believe that the possibility for `return_inverse` (or maybe even kwargs in general??) is something that should be added to the EA interface. @TomAugspurger @jreback @jbrockmendel 



Labels:
name: Enhancement
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @h-vetinari! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-10-11 14:19:02 UTC
Failure is (thankfully) only a flaky hypothesis test.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=h1) Report
> Merging [#24119](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/bb43726e1f52a0ddee45fcf485690719f262870d?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24119/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24119      +/-   ##
==========================================
- Coverage   92.37%    92.2%   -0.18%     
==========================================
  Files         166      162       -4     
  Lines       52420    51720     -700     
==========================================
- Hits        48423    47688     -735     
- Misses       3997     4032      +35
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.6% <91.66%> (-0.2%)` | :arrow_down: |
| #single | `43.01% <25%> (+0.13%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.43% <100%> (-0.55%)` | :arrow_down: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.9% <87.5%> (+0.12%)` | :arrow_up: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `0% <0%> (-86.37%)` | :arrow_down: |
| [pandas/io/sas/sasreader.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3Nhcy9zYXNyZWFkZXIucHk=) | `86.2% <0%> (-9.95%)` | :arrow_down: |
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `76.92% <0%> (-7.7%)` | :arrow_down: |
| [pandas/io/clipboard/clipboards.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NsaXBib2FyZC9jbGlwYm9hcmRzLnB5) | `28.23% <0%> (-2.36%)` | :arrow_down: |
| [pandas/core/arrays/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2Jhc2UucHk=) | `96.77% <0%> (-1.49%)` | :arrow_down: |
| [pandas/core/computation/check.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY2hlY2sucHk=) | `90.9% <0%> (-1.4%)` | :arrow_down: |
| [pandas/core/arrays/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lbGlrZS5weQ==) | `96.35% <0%> (-1.33%)` | :arrow_down: |
| [pandas/core/indexes/datetimelike.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZWxpa2UucHk=) | `97.29% <0%> (-1.23%)` | :arrow_down: |
| ... and [87 more](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=footer). Last update [bb43726...006d7ad](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=h1) Report
> Merging [#24119](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d8f9be7e396032acbd11de94dad78c459f78e4ba?src=pr&el=desc) will **decrease** coverage by `0.69%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24119/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #24119     +/-   ##
=========================================
- Coverage   93.07%   92.37%   -0.7%     
=========================================
  Files         192      166     -26     
  Lines       49551    52439   +2888     
=========================================
+ Hits        46119    48441   +2322     
- Misses       3432     3998    +566
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.79% <91.66%> (-1.04%)` | :arrow_down: |
| #single | `42.86% <25%> (+0.35%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `96% <100%> (-1.31%)` | :arrow_down: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.58% <87.5%> (-0.94%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-75%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `57.91% <0%> (-36.96%)` | :arrow_down: |
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.68% <0%> (-26.18%)` | :arrow_down: |
| [pandas/io/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2NvbW1vbi5weQ==) | `72.86% <0%> (-21.25%)` | :arrow_down: |
| [pandas/io/gcs.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2djcy5weQ==) | `80% <0%> (-20%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `86.36% <0%> (-13.64%)` | :arrow_down: |
| [pandas/io/formats/console.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2Zvcm1hdHMvY29uc29sZS5weQ==) | `66.66% <0%> (-11.46%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `88.68% <0%> (-8.84%)` | :arrow_down: |
| ... and [198 more](https://codecov.io/gh/pandas-dev/pandas/pull/24119/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=footer). Last update [d8f9be7...786159f](https://codecov.io/gh/pandas-dev/pandas/pull/24119?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Thinks changed around a bit here with `DatetimeArray` etc., but this commit should work.

As mentioned in the OP, this splits off first chunk of #24108 and makes some progress towards #4087 / #21357 / #21720 / #22824. I'm sure there'll still be lots of discussion, but having an implementation is a good start (even though there's not much happening - the cython backend is already there since a few months).

The diff in `test_algos.py` is completely busted, but I painstakingly created some nicely modular commits to step through the changes in a sane fashion.
@jreback @jorisvandenbossche @TomAugspurger 
Is it possible to give this PR some initial review? It's been lying around for ~3 months...
I won't have time in the near-term.

On Sun, Mar 10, 2019 at 7:55 AM h-vetinari <notifications@github.com> wrote:

> @jreback <https://github.com/jreback> @jorisvandenbossche
> <https://github.com/jorisvandenbossche> @TomAugspurger
> <https://github.com/TomAugspurger>
> Is it possible to give this PR some initial review? It's been lying around
> for ~3 months...
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/24119#issuecomment-471287508>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIpQHnXJQ1HqQj0zox1oTTiECzGoYks5vVQDZgaJpZM4ZFeYy>
> .
>

so again why should we add a method to do this, when we already have one?

```
In [16]: idx = pd.Index([0, 1, 1, 0])                                                                                                                                                                                                   

In [17]: pd.factorize(idx)                                                                                                                                                                                                              
Out[17]: (array([0, 1, 1, 0]), Int64Index([0, 1], dtype='int64'))

In [18]: idx.take(pd.factorize(idx)[0])                                                                                                                                                                                                 
Out[18]: Int64Index([0, 1, 1, 0], dtype='int64')
```

true numpy calls this ``return_inverse`` in ``.unique()`` but they also don't have ``.factorize()``. I really don't like having multiple ways of doing things. I am not convinced this is actually that useful. Can you show the usecase that is not *possible* or (claimed) not performant that this cannot be done now?

closing as stale
@jreback, this is not stale, but I do see that I overlooked to answer your comment a bit further up.

I've made the case in #22824 several times, but among other things, `factorize` and `unique` do *different* things, namely in the treatment of missing values (and there are several cases where that difference can be crucial).

Then there's the fact that most people who are not knee-deep in stats or R won't know `factorize`, but `unique` is incredibly intuitive and matches with what `numpy` offers.

Please reopen this and lets have this discussion (bearing in mind that the actual goal would be #24108; this PR is just a stepping stone).
Thanks for reopening! Will try to merge master soon.
@TomAugspurger @jorisvandenbossche @jreback 
This is updated and green. Since the diff in the tests basically unreadable, I'll recall the [following](https://github.com/pandas-dev/pandas/pull/24119#issuecomment-459917557):

> The diff in test_algos.py is completely busted, but I painstakingly created some nicely modular commits to step through the changes in a sane fashion.
I am highly u likely to change my -1 on this

i view this as duplicative and confusing api

if you want to document in an example great
> @jreback: i view this as duplicative and confusing api

`.unique` is central API, and extremely well-established (on top of being very intuitive; plus: the method exists in numpy with the inverse). You yourself [suggested](https://github.com/pandas-dev/pandas/issues/4087) to add a way of doing `return_inverse=True` for `unique`.

You had also already agreed to the utility of having an inverse [here](https://github.com/pandas-dev/pandas/issues/21357#issuecomment-399043089) (at the time you suggested to add the inverse to `duplicated`, which I did in #21645, only to be redirected to add the inverse to `.unique` where it fits more properly).

And as I won't tire of repeating: `unique` != `factorize`. The methods have different goals and differ in several key points.

I feel the conversation keeps going in circles - maybe this could be a nice example case for writing a fully fledged [enhancement proposal](https://github.com/pandas-dev/pandas/issues/28568)?
closing. I don't think anyone has the bandwith to work with you on this.