Repository: modin-project/modin

Issue:
Rewrite indexing benchmarks to make them more representative
Current indexing benchmarks are not so coverfull in terms of tracking regressions in common cases. There is a proposal to add more of them and change a bit the existing ones:
1. Use `np.ndarray` and a Series as a boolean mask instead of a python list https://github.com/modin-project/modin/blob/01a5de21c8fb3f6da0f9f887f6a659b7037136a0/asv_bench/benchmarks/benchmarks.py#L511-L512 When indexing with a boolean mask people usually pass a Series `df.iloc[df.field > 10]` or a numpy array `df.iloc[external_data_which_is_usally_a_numpy_array > 10]` rather than a python's list. Adding these cases would make us understand how is modin comparable with pandas in the most common ways of boolean indexing. Moreover, indexing with Series or a numpy array is way faster, so this change also would reduce the benchmarking time: 
<details><summary>Time difference example</summary>

```python
import pandas
import numpy as np
import timeit

df = pandas.DataFrame({f"col{i}": np.arange(30_000_000) for i in range(10)})
list_mask = [bool(i % 2) for i in range(len(df))]
np_array_mask = np.array(list_mask)

print(timeit.timeit(lambda: df.iloc[list_mask], number=10) / 10)     # 3.72s
print(timeit.timeit(lambda: df.iloc[np_array_mask], number=10) / 10) # 0.99s
```

</details>

2. Add a list-like indexer that grabs only a few elements, not the whole axis. https://github.com/modin-project/modin/blob/01a5de21c8fb3f6da0f9f887f6a659b7037136a0/asv_bench/benchmarks/benchmarks.py#L517-L518 The current [list-like indexer bench](https://modin.org/modin-bench/#benchmarks.TimeIndexing.time_loc?p-shape=%5B1000000%2C%2010%5D&machine=xeon-e5&p-indexer_type='list') runs for about a ~1.5 second, which is not representative enough since we don't know whether we're always slow in list indexing or it's only happening in the corner case like that, where we're taking the whole huge axis. (In addition, we might also convert list indexer into a numpy array to reduce overall benchmarking time)
3. At the current moment, we're only benchmarking indexing along the rows. It would be nice to also add some benchmarks for the indexing along the columns.
<details><summary>Example</summary>

```python
class TimeIndexingColumns:
    param_names = ["shape"]
    params = [
        get_benchmark_shapes("TimeIndexing")
    ]

    def setup(self, shape):
        self.df = generate_dataframe(ASV_USE_IMPL, "int", *shape, RAND_LOW, RAND_HIGH)
        self.numeric_indexer = [0, 1]
        self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()
    
    def time_iloc(self, shape):
        execute(self.df.iloc[:, self.numeric_indexer])
    
    def time_loc(self, shape):
        execute(self.df.loc[:, self.labels_indexer])
    
    def time___getitem__(self, shape):
        execute(self.df[self.labels_indexer])
```

</details>

All of these changes are highly desirable mostly for OmniSci benchs (for pandas backend it would also be a really good addition), because currently, in our top 25 funcs dashboard indexing benchmarks just showing many-second timings (for modin and for pandas) which is not representative at all.

The problem with adding all of this is that we will certainly lose the history of our indexing benchmarks. However, having representative numbers in a 25-funcs dashboard (at least for OmniSci) should cover this.

@anmyachev what do you think of these proposals?

cc @Garra1980 


Discussion:
1. Good idea. Do I understand correctly that now in `Modin` implementation there is no difference in the processing of `numpy` array and python list as an indexer?
2. Not against, but we need to pay attention to that If we take only a small part of the index, then small time numbers can be obtained, which makes benchmark times more difficult to compare.
3. Agree.

If necessary, it will be possible to run benchmarks with old `Modin` versions in order to restore history at least a little.
> Do I understand correctly that now in Modin implementation there is no difference in the processing of numpy array and python list as an indexer?

There is no difference in <i>how</i> it handles both of them but there is a difference in performance (the same as it shown at the head message of this issue)
> There is no difference in how it handles both of them but there is a difference in performance (the same as it shown at the head message of this issue)

ok

Resolving patch:
diff --git a/asv_bench/benchmarks/benchmarks.py b/asv_bench/benchmarks/benchmarks.py
--- a/asv_bench/benchmarks/benchmarks.py
+++ b/asv_bench/benchmarks/benchmarks.py
@@ -36,6 +36,7 @@
     execute,
     translator_groupby_ngroups,
     get_benchmark_shapes,
+    trigger_import,
 )
 
 
@@ -497,29 +498,73 @@ class TimeIndexing:
     param_names = ["shape", "indexer_type"]
     params = [
         get_benchmark_shapes("TimeIndexing"),
-        ["scalar", "bool", "slice", "list", "function"],
+        [
+            "bool_array",
+            "bool_series",
+            "scalar",
+            "slice",
+            "continuous_slice",
+            "numpy_array_take_all_values",
+            "python_list_take_10_values",
+            "function",
+        ],
     ]
 
+    indexer_getters = {
+        "bool_array": lambda df: np.array([False, True] * (len(df) // 2)),
+        # This boolean-Series is a projection of the source frame, it shouldn't
+        # be reimported or triggered to execute:
+        "bool_series": lambda df: df.iloc[:, 0] > 50,
+        "scalar": lambda df: len(df) // 2,
+        "slice": lambda df: slice(0, len(df), 2),
+        "continuous_slice": lambda df: slice(len(df) // 2),
+        "numpy_array_take_all_values": lambda df: np.arange(len(df)),
+        "python_list_take_10_values": lambda df: list(range(min(10, len(df)))),
+        "function": lambda df: (lambda df: df.index[::-2]),
+    }
+
     def setup(self, shape, indexer_type):
         self.df = generate_dataframe(ASV_USE_IMPL, "int", *shape, RAND_LOW, RAND_HIGH)
-        if indexer_type == "bool":
-            self.indexer = [False, True] * (shape[0] // 2)
-        elif indexer_type == "scalar":
-            self.indexer = shape[0] // 2
-        elif indexer_type == "slice":
-            self.indexer = slice(0, shape[0], 2)
-        elif indexer_type == "list":
-            self.indexer = [x for x in range(shape[0])]
-        elif indexer_type == "function":
-            self.indexer = lambda df: df.index[::-2]
+        trigger_import(self.df)
+
+        self.indexer = self.indexer_getters[indexer_type](self.df)
+        if isinstance(self.indexer, (pd.Series, pd.DataFrame)):
+            # HACK: Triggering `dtypes` meta-data computation in advance,
+            # so it won't affect the `loc/iloc` time:
+            self.indexer.dtypes
 
     def time_iloc(self, shape, indexer_type):
-        execute(self.df.iloc[self.indexer])
+        # Pandas doesn't implement `df.iloc[series boolean_mask]` and raises an exception on it.
+        # Replacing this with the semantically equivalent construction:
+        if indexer_type != "bool_series":
+            execute(self.df.iloc[self.indexer])
+        else:
+            execute(self.df[self.indexer])
 
     def time_loc(self, shape, indexer_type):
         execute(self.df.loc[self.indexer])
 
 
+class TimeIndexingColumns:
+    param_names = ["shape"]
+    params = [get_benchmark_shapes("TimeIndexing")]
+
+    def setup(self, shape):
+        self.df = generate_dataframe(ASV_USE_IMPL, "int", *shape, RAND_LOW, RAND_HIGH)
+        trigger_import(self.df)
+        self.numeric_indexer = [0, 1]
+        self.labels_indexer = self.df.columns[self.numeric_indexer].tolist()
+
+    def time_iloc(self, shape):
+        execute(self.df.iloc[:, self.numeric_indexer])
+
+    def time_loc(self, shape):
+        execute(self.df.loc[:, self.labels_indexer])
+
+    def time___getitem__(self, shape):
+        execute(self.df[self.labels_indexer])
+
+
 class TimeMultiIndexing:
     param_names = ["shape"]
     params = [get_benchmark_shapes("TimeMultiIndexing")]
diff --git a/asv_bench/benchmarks/omnisci/benchmarks.py b/asv_bench/benchmarks/omnisci/benchmarks.py
--- a/asv_bench/benchmarks/omnisci/benchmarks.py
+++ b/asv_bench/benchmarks/omnisci/benchmarks.py
@@ -29,6 +29,11 @@
     get_benchmark_shapes,
 )
 
+from ..benchmarks import (
+    TimeIndexing as TimeIndexingPandasExecution,
+    TimeIndexingColumns as TimeIndexingColumnsPandasExecution,
+)
+
 
 class TimeJoin:
     param_names = ["shape", "how"]
@@ -296,29 +301,18 @@ def time_value_counts(self, shape, ngroups):
         execute(self.series.value_counts())
 
 
-class TimeIndexing:
-    param_names = ["shape", "indexer_type"]
+class TimeIndexing(TimeIndexingPandasExecution):
     params = [
         get_benchmark_shapes("omnisci.TimeIndexing"),
-        ["scalar", "bool", "slice", "list", "function"],
+        *TimeIndexingPandasExecution.params[1:],
     ]
 
-    def setup(self, shape, indexer_type):
-        self.df = generate_dataframe(ASV_USE_IMPL, "int", *shape, RAND_LOW, RAND_HIGH)
-        trigger_import(self.df)
-        self.indexer = {
-            "bool": [False, True] * (shape[0] // 2),
-            "scalar": shape[0] // 2,
-            "slice": slice(0, shape[0], 2),
-            "list": list(range(shape[0])),
-            "function": lambda df: df.index[::-2],
-        }[indexer_type]
-
-    def time_iloc(self, shape, indexer_type):
-        execute(self.df.iloc[self.indexer])
-
-    def time_loc(self, shape, indexer_type):
-        execute(self.df.loc[self.indexer])
+
+class TimeIndexingColumns(TimeIndexingColumnsPandasExecution):
+    params = [
+        get_benchmark_shapes("omnisci.TimeIndexing"),
+        *TimeIndexingColumnsPandasExecution.params[1:],
+    ]
 
 
 class TimeResetIndex:
diff --git a/asv_bench/benchmarks/utils/common.py b/asv_bench/benchmarks/utils/common.py
--- a/asv_bench/benchmarks/utils/common.py
+++ b/asv_bench/benchmarks/utils/common.py
@@ -456,22 +456,20 @@ def trigger_import(*dfs):
     *dfs : iterable
         DataFrames to trigger import.
     """
-    assert ASV_USE_STORAGE_FORMAT == "omnisci"
+    if ASV_USE_STORAGE_FORMAT != "omnisci" or ASV_USE_IMPL == "pandas":
+        return
 
     from modin.experimental.core.execution.native.implementations.omnisci_on_native.omnisci_worker import (
         OmnisciServer,
     )
 
     for df in dfs:
-        if ASV_USE_IMPL == "modin":
-            df.shape  # to trigger real execution
-            df._query_compiler._modin_frame._partitions[0][
-                0
-            ].frame_id = OmnisciServer().put_arrow_to_omnisci(
-                df._query_compiler._modin_frame._partitions[0][0].get()
-            )  # to trigger real execution
-        elif ASV_USE_IMPL == "pandas":
-            pass
+        df.shape  # to trigger real execution
+        df._query_compiler._modin_frame._partitions[0][
+            0
+        ].frame_id = OmnisciServer().put_arrow_to_omnisci(
+            df._query_compiler._modin_frame._partitions[0][0].get()
+        )  # to trigger real execution
 
 
 def execute(


Test patch: