ENH: Support mask in duplicated algorithm
State: closed

- [x] xref #45776 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @jorisvandenbossche 

Benchmarks don't show any changes for duplicated.

```
ser = pd.Series([1, 2, pd.NA] * 1_000_000, dtype="Int64")
%timeit ser.duplicated(keep="first")
# 7.22 ms ± 42.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
ser = pd.Series((list(range(1, 1_000_000)) + [pd.NA]) * 3, dtype="Int64")
%timeit ser.duplicated(keep="first")
# 12.3 ms ± 402 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```



Labels:
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: duplicated
description: duplicated, drop_duplicates
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@mroeschke would you mind having a look?
Good point, assumed we had some but could not finde them 😂 will add some for all three cases
This is a much bigger change, but from looking at this PR, I started wondering: should we see multiple NAs as duplicated, or should we rather propagate NAs? 
In the past, propagating was never an option, since we didn't have a nullable boolean dtype. But similarly as NAs now propagate in comparison ops, they could also propagate here? 
> I started wondering: should we see multiple NAs as duplicated, or should we rather propagate NAs?

IMO if I was using `duplicated`, I would be a little more surprised If NAs persisted instead of indicating whether the row with NA was a duplicate with another row with NA (in the same position). I get though that comparison semantics with NA should propagate NA.

I think a natural follow up question with propagating NAs would be "How can i check whether this row with NA is a duplicate of this row with NA?"
Ah, good point, I wasn't considering the DataFrame case where you are checking for duplicated _rows_ (multiple columns at once), but I was only thinking about the more simple Series.duplicated (in which case the position doesn't matter, it's just which values are NA in that Series).

Yes, for DataFrames it certainly makes sense to consider NAs at the same location as equal, when considering whether two rows are equal (that's eg also what do in `equals()`)
Added the tests

I think this is a topic worth having a discussion, from a users perspective it would be a bit confusing If we consider them equal in one case and unequal in another case.

Would this impact operations like isin and merges and similar?
@mroeschke are you ok with merging this as is?
Small comment otherwise LGTM. Generally I am in favor of not propagating NA for  both `DataFrame/Series.duplicated` as this PR does. Is it worth discussing that in a separate issue @jorisvandenbossche before merging? 
> Is it worth discussing that in a separate issue @jorisvandenbossche before merging?

Given that this PR just preserves the current behaviour, that certainly doesn't need to block this PR from being merged.

If it would only be for Series.duplicated, I would take the time to open an issue to argue for propagating NAs, but since it's less clear how to deal with the DataFrame.duplicated case, I don't feel strongly enough about it.
Thanks @phofl