Repository: modin-project/modin

Issue:
Series.drop() is 10x slower on Modin than on Pandas
### System information
- **OS Platform and Distribution**: MacBook Pro (16-inch, 2019) with MacOS BigSur 11.5.2
- **Memory**: 16 GB 2667 MHz DDR4
- **Modin version** (`modin.__version__`): 0.12.0+40.g7e85c5df
- **Python version**: 3.8.8
- **Code we can use to reproduce**:

```python
import modin.pandas as pd
matches = pd.Series([0] * 16000000)
%time repr(matches.drop([0]))
%time repr(matches.drop(matches.index))
```

### Describe the problem

I am using Ray with 16 partitions, and Dask with 12 workers.

From the above snippet, dropping the first entry in the series takes 9.64 seconds on modin[ray], 9.93 seconds on modin[dask], and .433 seconds on pandas. Dropping the entire series takes 24.4 seconds on modin[ray], 24.9 seconds on modin[dask], and 2.12 seconds on pandas.

This bug came up when I was investigating [this StackOverflow question](https://stackoverflow.com/questions/69785910/filter-rows-by-a-list-of-keywords).

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
It appeared that most of the time is spent at the [`PandasDataframe.mask()`](https://github.com/modin-project/modin/blob/db1f929e3045c2e7d02fdf201cd5f78ff7237876/modin/core/dataframe/pandas/dataframe/dataframe.py#L542) that is called by [`qc.drop`](https://github.com/modin-project/modin/blob/db1f929e3045c2e7d02fdf201cd5f78ff7237876/modin/core/storage_formats/pandas/query_compiler.py#L2277-L2279). 

I'm fixing the `.mask()` slowness in a separate issue (#4138).
@mvashishtha, #4140 was merged. Could you check the performance?
Dropping the first element of a long series is much faster than before, though still slower than pandas. Dropping the entire series is about as slow as before:

```python
import modin.pandas as pd
from modin.config import BenchmarkMode
BenchmarkMode.put(True)
matches = pd.Series([0] * 16000000)
# This takes about 26.4 seconds.
%time matches.drop(matches.index)
# This takes about 2.9 seconds, but used to take 9.18 seconds.
%time matches.drop([0])
```
There's probably something else to improve in mask. cc @dchigarev  
I checked `Series.drop` with different data sizes to find the main bottlenecks. All results are for Ray engine, 112 workers.

1. The first benchmark with drop by all index:
<details><summary>Code</summary>

```python
from time import time

import modin.pandas as mpd
from modin.config import BenchmarkMode

BenchmarkMode.put(True)

mdf = mpd.Series([0] * 100_000_000)
pdf = mdf._to_pandas()

t = time()
pdf.drop(pdf.index)
print(f"t_all_pd: {time() - t} s")

t = time()
mdf.drop(mdf.index)
print(f"t_all_md: {time() - t} s")
```

</details>

The results for this is the next:

| Shape(rows) | 10m, time s | 100m, time s |
| ----------- | ----------- | ----------- |
| **pandas**| 0.0471 |0.4774| 
| **modin** | 14.9805 |148.2909|
| modin hotspot 1 | 14.77 |140.97|
| modin hotspot 2 | 0.2 |7.2|

*modin hotspot 1* is: https://github.com/modin-project/modin/blob/ee2440c53a1e3bd47736776e7c643f05c4a0db70/modin/pandas/base.py#L1192

*modin hotspot 2* is: https://github.com/modin-project/modin/blob/ee2440c53a1e3bd47736776e7c643f05c4a0db70/modin/core/storage_formats/pandas/query_compiler.py#L2280-L2291

2. The second benchmark is for dropping only by [0]:

<details><summary>Code</summary>

```python
from time import time

import modin.pandas as mpd
from modin.config import BenchmarkMode

BenchmarkMode.put(True)

mdf = mpd.Series([0] * 100_000_000)
pdf = mdf._to_pandas()

t = time()
pdf.drop([0])
print(f"t_single_pd: {time() - t} s")

t = time()
mdf.drop([0])
print(f"t_single_md: {time() - t} s")
```

</details>

The results for this is the next:

| Shape(rows) | 10m, time s | 100m, time s |
| ----------- | ----------- | ----------- |
| **pandas**| 0.3512 |3.7244| 
| **modin** | 3.3007 |42.9572|
| modin hotspot 1 | 1.34 |20.29|
| modin hotspot 2 | 1.69 |20.33|

*modin hotspot 1* is: https://github.com/modin-project/modin/blob/ee2440c53a1e3bd47736776e7c643f05c4a0db70/modin/core/storage_formats/pandas/query_compiler.py#L2280-L2291

*modin hotspot 2* is: https://github.com/modin-project/modin/blob/c17dde71ae8114723a13279e36bf5532dcba328a/modin/core/dataframe/pandas/dataframe/dataframe.py#L628-L630

The same hotspot in `self._get_dict_of_block_index` is observed in https://github.com/modin-project/modin/issues/4268.

Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -2291,16 +2291,10 @@ def dropna(self, **kwargs):
 
     def drop(self, index=None, columns=None):
         if index is not None:
-            # The unique here is to avoid duplicating rows with the same name
-            index = np.sort(
-                self.index.get_indexer_for(self.index[~self.index.isin(index)].unique())
-            )
+            index = np.sort(self.index.get_indexer_for(self.index.difference(index)))
         if columns is not None:
-            # The unique here is to avoid duplicating columns with the same name
             columns = np.sort(
-                self.columns.get_indexer_for(
-                    self.columns[~self.columns.isin(columns)].unique()
-                )
+                self.columns.get_indexer_for(self.columns.difference(columns))
             )
         new_modin_frame = self._modin_frame.mask(
             row_positions=index, col_positions=columns
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -1229,44 +1229,25 @@ def drop(
                 "Need to specify at least one of 'labels', 'index' or 'columns'"
             )
 
-        # TODO Clean up this error checking
-        if "index" not in axes:
-            axes["index"] = None
-        elif axes["index"] is not None:
-            if not is_list_like(axes["index"]):
-                axes["index"] = [axes["index"]]
-            if errors == "raise":
-                non_existant = [obj for obj in axes["index"] if obj not in self.index]
-                if len(non_existant):
-                    raise ValueError(
-                        "labels {} not contained in axis".format(non_existant)
-                    )
-            else:
-                axes["index"] = [obj for obj in axes["index"] if obj in self.index]
-                # If the length is zero, we will just do nothing
-                if not len(axes["index"]):
-                    axes["index"] = None
-
-        if "columns" not in axes:
-            axes["columns"] = None
-        elif axes["columns"] is not None:
-            if not is_list_like(axes["columns"]):
-                axes["columns"] = [axes["columns"]]
-            if errors == "raise":
-                non_existant = [
-                    obj for obj in axes["columns"] if obj not in self.columns
-                ]
-                if len(non_existant):
-                    raise ValueError(
-                        "labels {} not contained in axis".format(non_existant)
+        for axis in ["index", "columns"]:
+            if axis not in axes:
+                axes[axis] = None
+            elif axes[axis] is not None:
+                if not is_list_like(axes[axis]):
+                    axes[axis] = [axes[axis]]
+                if errors == "raise":
+                    non_existent = pandas.Index(axes[axis]).difference(
+                        getattr(self, axis)
                     )
-            else:
-                axes["columns"] = [
-                    obj for obj in axes["columns"] if obj in self.columns
-                ]
-                # If the length is zero, we will just do nothing
-                if not len(axes["columns"]):
-                    axes["columns"] = None
+                    if len(non_existent):
+                        raise ValueError(f"labels {non_existent} not contained in axis")
+                else:
+                    axes[axis] = [
+                        obj for obj in axes[axis] if obj in getattr(self, axis)
+                    ]
+                    # If the length is zero, we will just do nothing
+                    if not len(axes[axis]):
+                        axes[axis] = None
 
         new_query_compiler = self._query_compiler.drop(
             index=axes["index"], columns=axes["columns"]


Test patch: