API/ENH/DEPR: Series.unique returns Series
State: closed

 I know this PR is too big, but I need it to initiate discussion before `v.0.24` cutoff.

I've been working on adding a `return_inverse` to `unique` since mid June (as fast as possible). I know that changing the return type of `Series.unique` is potentially a controversial issue, but I truly believe that this should be done for v.0.24 (as otherwise the behavior is locked in past 1.0 and who knows if it ever changes then).

IMO, it's even a necessity if pandas ever wants to support `return_inverse` for `unique`. Here's a few arguments from working on this for a while:

* This is both a long-standing issue (#4087 milestoned since 0.14) and *very* hard for a user to do any other way.
* `.unique` is the obvious place for an inverse (as opposed to `.duplicated`, which I was directed to work on first, #21645)
* An inverse for `Series` only works well if the return type of `Series.unique` is a `Series`, see https://github.com/pandas-dev/pandas/issues/22824#issuecomment-432841642. This is not even the strongest reason IMO to change the type, as `.unique` is a patchwork currently (see rest OP of #22824):
  * `Index.unique` is already an `Index` (and changing that back to `ndarray` would be equally disruptive, and wouldn't lead anyhwere re:reconstruction)
  * The current `Series.unique` already special-cases `Categorical` and (effectively) `DatetimeArray`.

The reason I'm pushing this WIP for discussion is that this is obviously needs a deprecation cycle, and I *really* think this should be part of `v.0.24`. I'm sorry for the late timing, but I've been working as fast as feedback speed allowed (as often as politeness allowed me to ping) - #21645 was lying around mostly finished for 2 month, the cython backend (#22986 / #23400) took about 2 months, and I haven't been able to get an answer at #22824 for about 5 weeks (e.g. @jorisvandenbossche seems to be very busy or simply not available)

As for the PR itself, I *only* wanted to change the return-type in this PR, but `Series.unique` touches several important paths:
1. `Series.unique`
1. `IndexOpsMixin.unique` and hence `Index.unique`
1. `Categorical.unique`
1. `EA.unique`

So, as a demo here, I've adapted the first three, and it's a separate issue that **the EA contract should support the possibility for `return_inverse`**. This is also something that should IMO needs to make it to `v.0.24`.

- [x] actually closes #4087 (milestoned since 0.14), makes several big steps towards #21357 / #22824
- [x] tests modified / pass
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
- [ ] this obviously still needs tests
- [ ] splitting this up could work as follows:
  - [ ] add `return_inverse` to `pd.unique`
  - [ ] add `return_inverse` to `Index.unique`
  - [ ] add `return_inverse` to `Categorical.unique`
  - [ ] add `raw` to `Series.unique`
  - [ ] **Alternatively**, I could avoid passing the `unique`-calls through class hierarchy and just directly call the cython methods from `Series.unique`, which would allow introducing the deprecation *without* introducing the `return_inverse` kwarg yet.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Hello @h-vetinari! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/algorithms.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/core/algorithms.py) !

 - There are no PEP8 issues in the file [`pandas/core/arrays/categorical.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/core/arrays/categorical.py) !

 - There are no PEP8 issues in the file [`pandas/core/base.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/core/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/indexes/base.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/core/indexes/base.py) !

 - There are no PEP8 issues in the file [`pandas/core/series.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/core/series.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/categorical/test_analytics.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/arrays/categorical/test_analytics.py) !

 - There are no PEP8 issues in the file [`pandas/tests/arrays/sparse/test_array.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/arrays/sparse/test_array.py) !

 - There are no PEP8 issues in the file [`pandas/tests/extension/base/methods.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/extension/base/methods.py) !

 - There are no PEP8 issues in the file [`pandas/tests/frame/test_indexing.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/frame/test_indexing.py) !

 - There are no PEP8 issues in the file [`pandas/tests/plotting/common.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/plotting/common.py) !

 - There are no PEP8 issues in the file [`pandas/tests/reshape/merge/test_merge.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/reshape/merge/test_merge.py) !

 - There are no PEP8 issues in the file [`pandas/tests/series/test_duplicates.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/series/test_duplicates.py) !

 - There are no PEP8 issues in the file [`pandas/tests/test_algos.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/test_algos.py) !

 - There are no PEP8 issues in the file [`pandas/tests/test_base.py`](https://github.com/h-vetinari/pandas/blob/6fd279a6b8d562bca9f4ddb6a44bd02c15320b27/pandas/tests/test_base.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=h1) Report
> Merging [#24108](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4b5f4d1a59a7fce09c47eb0c03ba99a6cea472c6?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `60%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24108/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24108      +/-   ##
==========================================
- Coverage   92.21%   92.16%   -0.05%     
==========================================
  Files         161      161              
  Lines       51684    51723      +39     
==========================================
+ Hits        47658    47673      +15     
- Misses       4026     4050      +24
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.57% <60%> (-0.04%)` | :arrow_down: |
| #single | `42.98% <23.63%> (-0.03%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `92.59% <45.45%> (-1.09%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.22% <50%> (-0.11%)` | :arrow_down: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `96.75% <66.66%> (-0.89%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.17% <75%> (-0.23%)` | :arrow_down: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.68% <77.77%> (-0.43%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=footer). Last update [4b5f4d1...6fd279a](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=h1) Report
> Merging [#24108](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4b5f4d1a59a7fce09c47eb0c03ba99a6cea472c6?src=pr&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `60%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24108/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24108      +/-   ##
==========================================
- Coverage   92.21%   92.16%   -0.05%     
==========================================
  Files         161      161              
  Lines       51684    51723      +39     
==========================================
+ Hits        47658    47673      +15     
- Misses       4026     4050      +24
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.57% <60%> (-0.04%)` | :arrow_down: |
| #single | `42.98% <23.63%> (-0.03%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `92.59% <45.45%> (-1.09%)` | :arrow_down: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.22% <50%> (-0.11%)` | :arrow_down: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `96.75% <66.66%> (-0.89%)` | :arrow_down: |
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.17% <75%> (-0.23%)` | :arrow_down: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/24108/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.68% <77.77%> (-0.43%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=footer). Last update [4b5f4d1...6fd279a](https://codecov.io/gh/pandas-dev/pandas/pull/24108?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback:
> the question about the return value of .unique needs to be answered first; 

This is what this PR is about, since the issue had stalled for >1 months despite several pings.

> return an .array of the result (for Index and Series) is probably the most reasonable change and is mostly backward compatible

I disagree with this quite strongly:
* Changing `Index.unique` from `Index`->`ndarray` is *as much* of a breaking change as changing `Series.unique` from `ndarray`->`Series` (but has no benefits for reconstruction)
* `Series.unique` already special-cases `Categorical` and `EA`. An `ndarray` fits even less as the return of a `Series` method for where pandas is heading.
* Since `.unique` strongly advertises that it does not sort, there's an implicit index mapping happening already, only that it's very hard to coax out.
* If it were to keep returning `ndarray`, having an inverse is basically impossible without running into several antipatterns.
* etc.

> however better to bring this up on the issue (for unique return value)

Would you mind chiming in there then?
good idea, but closing as stale
> @jreback: good idea, but closing as stale

Not stale, but will need precursors, like #24119. 
@jreback @gfyoung @simonjayhawkins 
Can we please reopen this PR, or at least the precursor #24119?
i am not averse to changing the return type of .unique but cannot be linked to return_inverse which likely has less support

PRs need to do exactly 1 thing as have been stated many times
this ape e too complex 
@jreback 
Thanks for the response. I know this PR is too big (see OP), but it is the goal towards which I'd chip off smaller PRs (e.g. #24119).

Unfortunately, the `return_inverse` parts will have to come first, because it is not possible to return a Series with the correct index without tracking the inverse internally. In fact, I would argue that this is the *main* reason that `Series.unique` historically returned an array, because the cython-backend did not support calculating the inverse (luckily that's already been added in #22986 / #23400; the rest is a relatively easy change - see #24119).
@h-vetinari maybe i still don’t see it

why is returning a Series from .unique() actually useful?
why not an .array? then doesn’t .factorize() exactly provide the ability to reconstruct (iirc your main goal); speaking of reconstruction; how is this useful to a user? what is the use case?

@jreback many of those questions have already seen some discussion in https://github.com/pandas-dev/pandas/issues/22824 (including example use cases), so I would suggest we keep the general discussion on overhauling unique there. Can you repeat your comment there?