Repository: modin-project/modin

Issue:
BUG: the `execute()` function that is supposed to trigger execution actually doesn't do so
The current approach that is used in `execute()` function doesn't work as intended. The following code triggers execution only for the first partition, the rest of them stay unfinished:
https://github.com/modin-project/modin/blob/a7b6066f68ca47f23d8ee523cabed1a630214f71/asv_bench/benchmarks/utils/common.py#L500
This happens because the `all()` function halts iteration process once it meets first non-True value. The `.wait()` method that is supposed to be called against every partition returns `None`, the such return value causes `all()` function to stop iterating (and thus stop calling `.wait()`) right after evaluating the first partition.
https://github.com/modin-project/modin/blob/a7b6066f68ca47f23d8ee523cabed1a630214f71/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition.py#L197-L201
Here is a simple reproducer to illustrate this behavior:
```python
>>> def func(x):
...     print(f"Calling the 'func' with {x=}")
...
>>> res = all(map(func, [1, 2, 3, 4, 5]))
Calling the 'func' with x=1
>>> print(res)
False
```


Discussion:

Resolving patch:
diff --git a/asv_bench/benchmarks/utils/common.py b/asv_bench/benchmarks/utils/common.py
--- a/asv_bench/benchmarks/utils/common.py
+++ b/asv_bench/benchmarks/utils/common.py
@@ -496,8 +496,9 @@ def execute(
             df._query_compiler._modin_frame._execute()
             return
         partitions = df._query_compiler._modin_frame._partitions.flatten()
-        if len(partitions) > 0 and hasattr(partitions[0], "wait"):
-            all(map(lambda partition: partition.wait(), partitions))
+        mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
+        if len(partitions) and hasattr(mgr_cls, "wait_partitions"):
+            mgr_cls.wait_partitions(partitions)
             return
 
         # compatibility with old Modin versions


Test patch:
diff --git a/asv_bench/test/test_utils.py b/asv_bench/test/test_utils.py
--- a/asv_bench/test/test_utils.py
+++ b/asv_bench/test/test_utils.py
@@ -12,12 +12,13 @@
 # governing permissions and limitations under the License.
 
 import pytest
-from unittest.mock import patch, mock_open
+from unittest.mock import patch, mock_open, Mock
+import numpy as np
 
-from benchmarks.utils import data_shapes
-from benchmarks.utils import get_benchmark_shapes
+from benchmarks.utils import data_shapes, get_benchmark_shapes, execute
 
 from modin.config import AsvDataSizeConfig
+import modin.pandas as pd
 
 
 @pytest.mark.parametrize(
@@ -63,3 +64,13 @@ def test_get_benchmark_shapes_default(asv_config_content, result):
     AsvDataSizeConfig.put(None)
     with patch.object(data_shapes, "DEFAULT_CONFIG", new={"TimeJoin": result}):
         assert result == get_benchmark_shapes("TimeJoin")
+
+
+def test_execute():
+    df = pd.DataFrame(np.random.rand(100, 64))
+    partitions = df._query_compiler._modin_frame._partitions.flatten()
+    mgr_cls = df._query_compiler._modin_frame._partition_mgr_cls
+    with patch.object(mgr_cls, "wait_partitions", new=Mock()):
+        execute(df)
+        mgr_cls.wait_partitions.assert_called_once()
+        assert (mgr_cls.wait_partitions.call_args[0] == partitions).all()