DOC/TST: add pd.unique doc-string & buggy return of Categorical
State: closed

closes #9346


Labels:
name: Docs

Comments:
I also added the already implicit guarantees of ``pd.unique`` (and some tests, though this is used everywhere). xref #9346

cc @jorisvandenbossche @shoyer 
+1 on finally just adding the order of appearance to the docstring

Regarding the categorical unique, we have had quite some discussions about that before. Do you recall the outcome of that? (whether it should return an array or not)
This is from 0.19.2. So we *already* do this. The bug is that ``pd.unique`` didn't respect it.
```
In [2]: pd.Categorical(list('aabc')).unique()
Out[2]: 
[a, b, c]
Categories (3, object): [a, b, c]

In [3]: pd.unique(pd.Categorical(list('aabc')).unique())
Out[3]: array(['a', 'b', 'c'], dtype=object)
```
> This is from 0.19.2. So we already do this.

OK, then I misremember that (or maybe I remember the discussion about the datetime tz case ..). 
Then it indeed makes some sense to be consistent there. It should probably then a plain Categorical instead of a Series, though, to be consistent.

Something else, the datetime with tz example are not correct I think, I get also an array for them.
Ignore this one. Slightly wrong.

0.19.2
```
In [5]: Series(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:           pd.Timestamp('20160101', tz='US/Eastern')])).unique()
Out[5]: array([Timestamp('2016-01-01 00:00:00-0500', tz='US/Eastern')], dtype=object)

In [6]: pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:           pd.Timestamp('20160101', tz='US/Eastern')]).unique()
Out[6]: DatetimeIndex(['2016-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [7]: pd.unique(Series(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:           pd.Timestamp('20160101', tz='US/Eastern')])))
Out[7]: array(['2016-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

In [8]: pd.unique(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:           pd.Timestamp('20160101', tz='US/Eastern')]))
Out[8]: array(['2016-01-01T05:00:00.000000000'], dtype='datetime64[ns]')
```

master
```
In [1]: Series(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:    ...:           pd.Timestamp('20160101', tz='US/Eastern')])).unique()
Out[1]: array([Timestamp('2016-01-01 00:00:00-0500', tz='US/Eastern')], dtype=object)

In [2]: pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:    ...:           pd.Timestamp('20160101', tz='US/Eastern')]).unique()
   ...: 
Out[2]: DatetimeIndex(['2016-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [3]: pd.unique(Series(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:    ...:           pd.Timestamp('20160101', tz='US/Eastern')])))
   ...: 
Out[3]: DatetimeIndex(['2016-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [4]: pd.unique(pd.Index([pd.Timestamp('20160101', tz='US/Eastern'),
   ...:    ...:           pd.Timestamp('20160101', tz='US/Eastern')]))
   ...: 
Out[4]: DatetimeIndex(['2016-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)
```

I think these were completely non-tested before. I *think* that ``pd.unique`` (before my cleanup), was doing the wrong things for datetimetz aware.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=h1) Report
> Merging [#15939](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/3b53202d90e86a1bc0f5db7a9c2e66c164f909a9?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15939/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15939      +/-   ##
==========================================
+ Coverage   90.97%   90.97%   +<.01%     
==========================================
  Files         145      145              
  Lines       49475    49480       +5     
==========================================
+ Hits        45008    45013       +5     
  Misses       4467     4467
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.73% <100%> (ø)` | :arrow_up: |
| #single | `40.67% <37.5%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.57% <100%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=footer). Last update [3b53202...df7d073](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=h1) Report
> Merging [#15939](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f35209e2279154ed7060dd0e17d41da96f9c0186?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15939/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15939      +/-   ##
==========================================
+ Coverage      91%      91%   +<.01%     
==========================================
  Files         145      145              
  Lines       49576    49581       +5     
==========================================
+ Hits        45118    45123       +5     
  Misses       4458     4458
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.77% <100%> (ø)` | :arrow_up: |
| #single | `40.57% <44.44%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `95.51% <ø> (ø)` | :arrow_up: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `95.85% <ø> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.89% <100%> (ø)` | :arrow_up: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.57% <100%> (+0.04%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=footer). Last update [f35209e...dc81286](https://codecov.io/gh/pandas-dev/pandas/pull/15939?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
So I fixed this to be consistent. Though ``.unique`` itself is NOT very consistent (but we have known this for a while). 

Its now consistent between calling ``Series(....).unique()`` and ``pd.unique(Series)``.
@chris-b1 @jorisvandenbossche @shoyer 

FYI, not trying to break new ground here at all, just make things consistent.
Looks good, only question would on timezones, it looks you are taking the approach proposed in #15750 - returning an object array?  I suppose that's the right answer.
chris that's a good point

it is somewhat orthogonal to this one though (.unique already returns an object array of the stamps)


@jorisvandenbossche if you'd have a look.
One more thing that surprised me is that the unique of a Categorical changes the actual 'categories':

```
In [17]: pd.Series(pd.Categorical(list('aacb')))
Out[17]: 
0    a
1    a
2    c
3    b
dtype: category
Categories (3, object): [a, b, c]

In [18]: pd.Series(pd.Categorical(list('aacb'))).unique()
Out[18]: 
[a, c, b]
Categories (3, object): [a, c, b]   <----- not the original order, but taken the order of appearance
```

So the actual values have the order of appearance (like `unique` says to do), that is OK, but also the actual categories do the same, of which I don't really understand the reasoning. 
Note this is the current behaviour, so not related to this PR, just noticed it reviewing this. Did we have a good rationale for this behaviour? (otherwise can open a new issue for it)
@jorisvandenbossche from your last

> So the actual values have the order of appearance (like unique says to do), that is OK, but also the actual categories do the same, of which I don't really understand the reasoning.
Note this is the current behaviour, so not related to this PR, just noticed it reviewing this. Did we have a good rationale for this behaviour? (otherwise can open a new issue for it)


This is documented in ``Categorical.unique``. Essentially you get the order of the *codes* for an unordered categorical (an ordered categorical returns the category order).

This then preserves the ordering of *appearance* (rather than the category ordering, I *think* to preserve the ``pd.unique`` guarantee)

e.g.

same in 0.19.2 and now.
```
In [1]: pd.Categorical(list('bac')).unique()
Out[1]: 
[b, a, c]
Categories (3, object): [b, a, c]

In [2]: pd.Categorical(list('bac'), categories=list('abc')).unique()
Out[2]: 
[b, a, c]
Categories (3, object): [b, a, c]

In [3]: pd.Categorical(list('bac'), ordered=True).unique()
Out[3]: 
[b, a, c]
Categories (3, object): [a < b < c]

In [4]: pd.Categorical(list('bac'), categories=list('abc'), ordered=True).unique()
Out[4]: 
[b, a, c]
Categories (3, object): [a < b < c]
```

I can see how this works. Not really sure about it though. We can open an issue to discuss for 0.21.0. (this is unchanged in this PR)
@jorisvandenbossche updated if you'd have a quick relook. Updated docs & added some examples.
@jreback Looks good! Did some minor changes (pd -> pandas, I don't think sphinx is aware of the pd shortcut), and merged. Thanks!
thanks!
hmm links are not rendering in see also?

http://pandas-docs.github.io/pandas-docs-travis/generated/pandas.Series.unique.html?highlight=unique#pandas.Series.unique
Do we need to add `Categorical.unique` to https://raw.githubusercontent.com/pandas-dev/pandas/master/doc/source/api.rst?
hmm yes i think that's why the links don't work
Series and pd.unique seem good