Repository: modin-project/modin

Issue:
`MapFunction` is very slow at Ray engine
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: All, (Ray)
- **Modin version** (`modin.__version__`): Current master
- **Python version**: 3.7.5
- **Code we can use to reproduce**:
```python
import numpy as np
import string
import csv
import os
from timeit import default_timer as timer

RAND_LOW = -100
RAND_HIGH = 100
N = 30000
M = 1000
TEST_FILENAME = os.path.abspath(f"int_dataset-{N},{M},{RAND_LOW},{RAND_HIGH}.csv")


def generate_data_file(filename, row_n, col_n):
    letters = list(string.ascii_letters)
    columns = [f"col{i}" for i in np.arange(col_n)]

    with open(filename, "w") as file:
        writer = csv.writer(file, delimiter=",")
        writer.writerow(columns)
        for _ in np.arange(row_n):
            writer.writerow(np.random.randint(RAND_LOW, RAND_HIGH, col_n))


if __name__ == "__main__":
    import pandas
    import modin.pandas as pd

    if not os.path.exists(TEST_FILENAME):
        generate_data_file(TEST_FILENAME, N, M)

    md_df = pd.read_csv(TEST_FILENAME)
    print("file readed")
    t1_md = timer()
    md_res = print(md_df.abs())
    t2_md = timer()

    print("Modin:", t2_md - t1_md)

```
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
Related code will measure the working time of `abs`, which is implemented via `MapFunction`, that call `map_partitions` inside `partition_manager`. The problem is that, it works really slow at ray execution engine, but after changing `map_partitions` to `lazy_map_partitions` it starts to work fine:
```
112 cores:
	map_partitions: 2.6236
	lazy_map_partitions: 0.1817
```
I've run this code at python execution engine and didn't notice such a problem.
So, is just applying function to partitions is not async at `Ray`? Are there any limitations of using `lazy_map_partitions` instead of `map_partitions`? Can we just change `map_partitions` to `lazy_map_partitions` everywhere in modin?
Actually, the difference is not so critical for machines which have not many cores, because it has fewer partitions and so less places to possible blocks
```
8 cores:
	map_partitions: 0.2313
	lazy_map_partitions: 0.1215
```
@devin-petersohn please take a look



Discussion:
We can certainly test this for all operations. The `lazy_map_partitions` does make some assumptions, but it would be a good optimization for the interactive case.

`MapReduceFunction` might give some trouble, but I think we can work around that. Let's do this for the next release.

Resolving patch:
diff --git a/modin/engines/base/frame/data.py b/modin/engines/base/frame/data.py
--- a/modin/engines/base/frame/data.py
+++ b/modin/engines/base/frame/data.py
@@ -706,7 +706,9 @@ def astype(self, col_dtypes):
         def astype_builder(df):
             return df.astype({k: v for k, v in col_dtypes.items() if k in df})
 
-        new_frame = self._frame_mgr_cls.map_partitions(self._partitions, astype_builder)
+        new_frame = self._frame_mgr_cls.lazy_map_partitions(
+            self._partitions, astype_builder
+        )
         return self.__constructor__(
             new_frame,
             self.index,
@@ -1041,7 +1043,7 @@ def _map_reduce(self, axis, map_func, reduce_func=None, preserve_index=True):
         else:
             reduce_func = self._build_mapreduce_func(axis, reduce_func)
 
-        map_parts = self._frame_mgr_cls.map_partitions(self._partitions, map_func)
+        map_parts = self._frame_mgr_cls.lazy_map_partitions(self._partitions, map_func)
         reduce_parts = self._frame_mgr_cls.map_axis_partitions(
             axis, map_parts, reduce_func
         )
@@ -1079,7 +1081,7 @@ def _map(self, func, dtypes=None, validate_index=False, validate_columns=False):
         -------
             A new dataframe.
         """
-        new_partitions = self._frame_mgr_cls.map_partitions(self._partitions, func)
+        new_partitions = self._frame_mgr_cls.lazy_map_partitions(self._partitions, func)
         if dtypes == "copy":
             dtypes = self._dtypes
         elif dtypes is not None:
diff --git a/modin/engines/base/frame/partition_manager.py b/modin/engines/base/frame/partition_manager.py
--- a/modin/engines/base/frame/partition_manager.py
+++ b/modin/engines/base/frame/partition_manager.py
@@ -83,7 +83,7 @@ def groupby_reduce(cls, axis, partitions, by, map_func, reduce_func):
         new_partitions = np.array(
             [
                 [
-                    part.apply(
+                    part.add_to_apply_calls(
                         map_func,
                         other=by_parts[col_idx].get()
                         if axis
@@ -191,7 +191,7 @@ def broadcast_apply(cls, axis, apply_func, left, right):
         return np.array(
             [
                 [
-                    part.apply(
+                    part.add_to_apply_calls(
                         apply_func,
                         r=right_parts[col_idx].get()
                         if axis
@@ -451,7 +451,9 @@ def _apply_func_to_list_of_partitions_broadcast(
     ):
         preprocessed_func = cls.preprocess_func(func)
         return [
-            obj.apply(preprocessed_func, other=[o.get() for o in broadcasted], **kwargs)
+            obj.add_to_apply_calls(
+                preprocessed_func, other=[o.get() for o in broadcasted], **kwargs
+            )
             for obj, broadcasted in zip(partitions, other.T)
         ]
 
@@ -469,7 +471,9 @@ def _apply_func_to_list_of_partitions(cls, func, partitions, **kwargs):
             A list of BaseFramePartition objects.
         """
         preprocessed_func = cls.preprocess_func(func)
-        return [obj.apply(preprocessed_func, **kwargs) for obj in partitions]
+        return [
+            obj.add_to_apply_calls(preprocessed_func, **kwargs) for obj in partitions
+        ]
 
     @classmethod
     def apply_func_to_select_indices(


Test patch: