ENH: duplicated and drop_duplicates now accept keep kw
State: closed

Closes #6511, Closes #8505.

Introduce `keep` kw to handle first / last / all duplicates based on the discussion above. If there is better API/kw, please lmk.
### `duplicated`
- `keep='first'` (default): mark duplicates as `True` except for the first occurrence.
- `keep='last'`: mark duplicates as `True` except for the last occurrence.
- `keep=False`: mark all duplicates as `True`.
### `drop_duplicates`
- `keep='first'` (default): drop duplicates except for the first occurrence.
- `keep='last'`: drop duplicates except for the last occurrence.
- `keep=False`: drop all duplicates.

``` python
import pandas as pd
s = pd.Series(['A', 'B', 'C', 'A', 'B', 'D'])

# mark duplicates except for the first occurrence
s.duplicated()
#0    False
#1    False
#2    False
#3     True
#4     True
#5    False
# dtype: bool

# mark duplicates except for the last occurrence
s.duplicated(keep='last')
#0     True
#1     True
#2    False
#3    False
#4    False
#5    False
# dtype: bool

# mark all duplicates
s.duplicated(keep=False)
#0     True
#1     True
#2    False
#3     True
#4     True
#5    False
# dtype: bool

# drop duplicates except for the first occurrence
s.drop_duplicates()
#0    A
#1    B
#2    C
#5    D
# dtype: object

# drop duplicates except for the last occurrence
s.drop_duplicates(keep='last')
#2    C
#3    A
#4    B
#5    D
# dtype: object

# drop all duplicates
s.drop_duplicates(keep=False)
#2    C
#5    D
# dtype: object
```

CC @shoyer, @wikiped, @socheon 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
I like this functionality... but I'm not sure about the name/keywords.

`take='first'` and `take='last'` indicate that the first/last item should be kept. Thus, if no items are kept, it would make more sense to use the value `take='none'` rather than `take='all'`.

Also, I'm not sure about the name `take`. I know this is similar to the existing keyword argument `take_last`, but I don't find it very suggestive. I don't know if I have a better alternative, though... maybe `keep`?

Much appreciated enhancement, but `take` does sound ambiguous to me as well (as in 'take-to-drop' or 'take-to-keep'). `keep` sounds like a clearer option. 

Thanks, agreed that `keep` is preferable to `take`. Changing the PR if there is no other suggestions within few days.

### `duplicated`
- `keep='first'` (default): mark duplicates as `True` except for the first occurrence.
- `keep='last'`: mark duplicates as `True` except for the last occurrence.
- `keep='none'`: mark all duplicates as `True`.

### `drop_duplicates`
- `keep='first'` (default): drop duplicates except for the first occurrence.
- `keep='last'`: drop duplicates except for the last occurrence.
- `keep='none'`: drop all duplicates.

`keep` lgtm. I only -0 on the `keep='none'` though as its not very pythonic.

Alternative could be `keep=False` or `keep=None`? 

I like, but I don't think will work `keep=None` (as it will then be the default), but `keep=False` looks ok to me

`keep=False` works for me, though there's something to be said for all arguments having the same type.

Thanks, updated PR and description on top. Should this wait for 0.17?

yep

status?

@jreback Fixed `duplicated_int64` to release the GIL. Please take a look when you have a time.

can u add this to the master deprecation issue so we know when to remove

couple of minor comments

looks good

merge when u r ready 

Thanks. Fixed and added to master tracker.