Repository: modin-project/modin

Issue:
drop_duplicates after pd.concat not working as expected
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS 12.3.1
- **Modin version** (`modin.__version__`): 0.14.1
- **Python version**: Python 3.9.12
- **Code we can use to reproduce**:

```python
import pandas as pd
import modin.pandas as mpd
import numpy as np

# Pandas
np.random.seed(0)
df1 = pd.DataFrame(np.random.randint(0, 1000, size=1000), columns=["A"])
np.random.seed(1)
df2 = pd.DataFrame(np.random.randint(0, 1000, size=1000), columns=["A"])

df = pd.concat([df1, df2])
print(len(df))  # 2000
print(len(df.drop_duplicates()))  # 874

# Modin
np.random.seed(0)
df1 = mpd.DataFrame(np.random.randint(0, 1000, size=1000), columns=["A"])
np.random.seed(1)
df2 = mpd.DataFrame(np.random.randint(0, 1000, size=1000), columns=["A"])

df = mpd.concat([df1, df2])
print(len(df))  # 2000
print(len(df.drop_duplicates()))  # 322 ?!?

# Issue caused by index? Use ignore_index=True to fix
df = mpd.concat([df1, df2], ignore_index=True)
print(len(df))  # 2000
print(len(df.drop_duplicates()))  # 874
```

### Describe the problem

After combining DataFrames using `mpd.concat()`, dropping duplicate values removes WAY too many items. I believe this is related to the indexing of the combined DataFrames, as using `ignore_index=True` in the `concat()` call seems to fix the issue.

In regular pandas, the result is the same whether or not `ignore_index=True` is used.

The above example is a toy one. I noticed this when combining large CSV files. Happy to share those, but this takes much less time to run!




Discussion:
Tested with both Ray and Dask as the engine. Both give the same results.
Think I found the issue in `BasePandasDataset` class in `base.py`

```python
    def drop_duplicates(
        self, keep="first", inplace=False, **kwargs
    ):  # noqa: PR01, RT01, D200
        """
        Return `BasePandasDataset` with duplicate rows removed.
        """
        inplace = validate_bool_kwarg(inplace, "inplace")
        subset = kwargs.get("subset", None)
        if subset is not None:
            if is_list_like(subset):
                if not isinstance(subset, list):
                    subset = list(subset)
            else:
                subset = [subset]
            duplicates = self.duplicated(keep=keep, subset=subset)
        else:
            duplicates = self.duplicated(keep=keep)
        indices = duplicates.values.nonzero()[0]
        return self.drop(index=self.index[indices], inplace=inplace)
```

The `return` statement is using the index values, but the indices are duplicated after the `concat`. This leads to extra rows getting dropped erroneously.

My solution may be overly simplistic, but instead of this:
```python
        indices = duplicates.values.nonzero()[0]
        return self.drop(index=self.index[indices], inplace=inplace)
```
could you do something like what [Pandas does](https://github.com/pandas-dev/pandas/blob/a160caff845db846516ff5f973c5c0a09a605d8d/pandas/core/frame.py#L6363) (shown below)?
```python
        result = self[-duplicated]
        if ignore_index:
            result.index = default_index(len(result))

        if inplace:
            self._update_inplace(result)
            return None
        else:
            return result
```

(sorry if this isn't useful -- I'm much better at breaking things than fixing them!)
@jrsacher thank you for reporting this bug! I was able to reproduce it at commit cf0eb393daa41abdd8cdf32b52ceee938cdcbe1a. If I see a quick fix, I'll assign the issue to myself and make a PR.
@jrsacher I'm sorry, but I missed your last message. I think you've diagnosed the bug correctly. I think making the Modin implementation more like the pandas implementation as you suggested should work. 

Would you be interested in contributing this fix yourself? Members of \@modin-project/modin-contributors, including me, can help you make the fix and get it merged. You could follow the guide [here](https://github.com/modin-project/modin/blob/cf0eb393daa41abdd8cdf32b52ceee938cdcbe1a/contributing/contributing.md) to get set up for Modin development.

Resolving patch:
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -1388,6 +1388,7 @@ def drop_duplicates(
         """
         inplace = validate_bool_kwarg(inplace, "inplace")
         subset = kwargs.get("subset", None)
+        ignore_index = kwargs.get("ignore_index", False)
         if subset is not None:
             if is_list_like(subset):
                 if not isinstance(subset, list):
@@ -1397,8 +1398,13 @@ def drop_duplicates(
             duplicates = self.duplicated(keep=keep, subset=subset)
         else:
             duplicates = self.duplicated(keep=keep)
-        indices = duplicates.values.nonzero()[0]
-        return self.drop(index=self.index[indices], inplace=inplace)
+        result = self[~duplicates]
+        if ignore_index:
+            result.index = pandas.RangeIndex(stop=len(result))
+        if inplace:
+            self._update_inplace(result._query_compiler)
+        else:
+            return result
 
     def eq(self, other, axis="columns", level=None):  # noqa: PR01, RT01, D200
         """
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -283,7 +283,7 @@ def drop_duplicates(
         Return ``DataFrame`` with duplicate rows removed.
         """
         return super(DataFrame, self).drop_duplicates(
-            subset=subset, keep=keep, inplace=inplace
+            subset=subset, keep=keep, inplace=inplace, ignore_index=ignore_index
         )
 
     @property


Test patch:
diff --git a/modin/pandas/test/dataframe/test_map_metadata.py b/modin/pandas/test/dataframe/test_map_metadata.py
--- a/modin/pandas/test/dataframe/test_map_metadata.py
+++ b/modin/pandas/test/dataframe/test_map_metadata.py
@@ -746,30 +746,43 @@ def test_droplevel():
     [None, "col1", "name", ("col1", "col3"), ["col1", "col3", "col7"]],
     ids=["None", "string", "name", "tuple", "list"],
 )
-def test_drop_duplicates(data, keep, subset):
+@pytest.mark.parametrize("ignore_index", [True, False], ids=["True", "False"])
+def test_drop_duplicates(data, keep, subset, ignore_index):
     modin_df = pd.DataFrame(data)
     pandas_df = pandas.DataFrame(data)
 
     try:
-        pandas_df.drop_duplicates(keep=keep, inplace=False, subset=subset)
+        pandas_df.drop_duplicates(
+            keep=keep, inplace=False, subset=subset, ignore_index=ignore_index
+        )
     except Exception as e:
         with pytest.raises(type(e)):
-            modin_df.drop_duplicates(keep=keep, inplace=False, subset=subset)
+            modin_df.drop_duplicates(
+                keep=keep, inplace=False, subset=subset, ignore_index=ignore_index
+            )
     else:
         df_equals(
-            pandas_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
-            modin_df.drop_duplicates(keep=keep, inplace=False, subset=subset),
+            pandas_df.drop_duplicates(
+                keep=keep, inplace=False, subset=subset, ignore_index=ignore_index
+            ),
+            modin_df.drop_duplicates(
+                keep=keep, inplace=False, subset=subset, ignore_index=ignore_index
+            ),
         )
 
     try:
         pandas_results = pandas_df.drop_duplicates(
-            keep=keep, inplace=True, subset=subset
+            keep=keep, inplace=True, subset=subset, ignore_index=ignore_index
         )
     except Exception as e:
         with pytest.raises(type(e)):
-            modin_df.drop_duplicates(keep=keep, inplace=True, subset=subset)
+            modin_df.drop_duplicates(
+                keep=keep, inplace=True, subset=subset, ignore_index=ignore_index
+            )
     else:
-        modin_results = modin_df.drop_duplicates(keep=keep, inplace=True, subset=subset)
+        modin_results = modin_df.drop_duplicates(
+            keep=keep, inplace=True, subset=subset, ignore_index=ignore_index
+        )
         df_equals(modin_results, pandas_results)
 
 
@@ -866,6 +879,14 @@ def test_drop_duplicates_after_sort():
     df_equals(modin_result, pandas_result)
 
 
+def test_drop_duplicates_with_repeated_index_values():
+    # This tests for issue #4467: https://github.com/modin-project/modin/issues/4467
+    data = [[0], [1], [0]]
+    index = [0, 0, 0]
+    modin_df, pandas_df = create_test_dfs(data, index=index)
+    eval_general(modin_df, pandas_df, lambda df: df.drop_duplicates())
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 @pytest.mark.parametrize("axis", axis_values, ids=axis_keys)
 @pytest.mark.parametrize("how", ["any", "all"], ids=["any", "all"])