Repository: modin-project/modin

Issue:
Groupby dictionary aggregation regression
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 20.04
- **Modin version** : aa818f5a246703621d4a70edaa04b92a511faa46
- **Python version**: 3.8.6
- **Code we can use to reproduce**:
```
import os

os.environ["MODIN_ENGINE"] = "ray"
os.environ["MODIN_CPUS"] = "4"

import modin.pandas as pd
from timeit import default_timer as timer

data_filename = "h2o_10k.csv"

df = pd.read_csv(data_filename)


def q1(df):
    groupby_cols = ["id1"]
    agg_cols_funcs = {"v1": "sum"}

    return df.groupby(groupby_cols).agg(agg_cols_funcs)

def q10(df):
    groupby_cols = ["id1", "id2", "id3", "id4", "id5", "id6"]
    agg_cols_funcs = {"v3": "sum", "v1": "count"}

    return df.groupby(groupby_cols).agg(agg_cols_funcs)

def meas_func(func, iters, *args, **kwargs):
    t_meas = float("inf")
    for _ in range(iters):
        t0 = timer()
        ans = func(*args, **kwargs)
        ans.shape
        t = timer() - t0
        del ans
        t_meas = t if t < t_meas else t_meas
    
    return t_meas

t_q1 = meas_func(q1, iters=3, **{"df": df})
t_q10 = meas_func(q10, iters=3, **{"df": df})

# results comparison between before vs after ee39d1767efb363fe6c147fc701b41315eee3afa commit
print("t_q1:", t_q1) # 0.026 s vs 0.031s
print("t_q10:", t_q10) # 0.091 s vs 2.09 s
```

<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
Regression occurred after ee39d1767efb363fe6c147fc701b41315eee3afa commit.
data file [h2o_10k.txt](https://github.com/modin-project/modin/files/5988496/h2o_10k.txt)



### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->



Discussion:
The reason for the slow dict-groupby is pretty ridiculous. There is a dict, that contains map and reduce functions for every operation that supports GroupbyReduce. These functions are applied to groups in every partition.
https://github.com/modin-project/modin/blob/aa818f5a246703621d4a70edaa04b92a511faa46/modin/data_management/functions/groupby_function.py#L234-L267

The problem is that they're all lambdas, for pandas all of them are unknown python callables, so it can't execute aggregation with its fast C module, we force it to apply our callables. If we replace callables to the function names it performs much faster! 

Here is a comparison of groupby aggregation in pandas with function name and with callable:
```python
import pandas
from timeit import default_timer as timer

# 500mb H2O dataset
data_filename = "/localdisk/benchmark_datasets/h2o/G1_1e7_1e1_0_0.csv"
df = pandas.read_csv(data_filename)

t1 = timer()
df.groupby("id6").agg({"v1": "sum"})
print(f"str function: {timer() - t1}")

t1 = timer()
df.groupby("id6").agg({"v1": lambda df: df.sum()})
print(f"callable function: {timer() - t1}")

# Output:
# str function: 0.7193635171279311
# callable function: 38.2026267433539

```

Speaking about modin, when I replaced lambdas to function names in [`groupby_reduce_functions`](https://github.com/modin-project/modin/blob/aa818f5a246703621d4a70edaa04b92a511faa46/modin/data_management/functions/groupby_function.py#L234-L267) and ran the script from the [issue header](https://github.com/modin-project/modin/issues/2742#issue-809267781) I got this results:
```
Before ee39d17:
t_q1: 0.025292523205280304
t_q10: 0.08418524172157049

After ee39d17:
t_q1: 0.030419232323765755
t_q10: 1.9959391057491302

After ee39d17 + fix(lambdas -> str-names):
t_q1: 0.023051242344081402
t_q10: 0.0943737206980586

```

Going to create PR that fixes this soon


results of H2O queries with 500mb dataset and 112 partitions:
```
Before ee39d17:
t_q1: 3.1282251700758934
t_q10: 144.17810649238527

After ee39d17 + fix(lambdas -> str-names):
t_q1: 0.3956367587670684
t_q10: 140.4794331882149
```

Resolving patch:
diff --git a/modin/backends/pandas/query_compiler.py b/modin/backends/pandas/query_compiler.py
--- a/modin/backends/pandas/query_compiler.py
+++ b/modin/backends/pandas/query_compiler.py
@@ -2363,16 +2363,14 @@ def _callable_func(self, func, axis, *args, **kwargs):
     # nature. They require certain data to exist on the same partition, and
     # after the shuffle, there should be only a local map required.
 
-    groupby_count = GroupbyReduceFunction.register(*groupby_reduce_functions["count"])
-    groupby_any = GroupbyReduceFunction.register(*groupby_reduce_functions["any"])
-    groupby_min = GroupbyReduceFunction.register(*groupby_reduce_functions["min"])
-    groupby_prod = GroupbyReduceFunction.register(*groupby_reduce_functions["prod"])
-    groupby_max = GroupbyReduceFunction.register(*groupby_reduce_functions["max"])
-    groupby_all = GroupbyReduceFunction.register(*groupby_reduce_functions["all"])
-    groupby_sum = GroupbyReduceFunction.register(*groupby_reduce_functions["sum"])
-    groupby_size = GroupbyReduceFunction.register(
-        *groupby_reduce_functions["size"], method="size"
-    )
+    groupby_all = GroupbyReduceFunction.register("all")
+    groupby_any = GroupbyReduceFunction.register("any")
+    groupby_count = GroupbyReduceFunction.register("count")
+    groupby_max = GroupbyReduceFunction.register("max")
+    groupby_min = GroupbyReduceFunction.register("min")
+    groupby_prod = GroupbyReduceFunction.register("prod")
+    groupby_size = GroupbyReduceFunction.register("size", method="size")
+    groupby_sum = GroupbyReduceFunction.register("sum")
 
     def _groupby_dict_reduce(
         self, by, axis, agg_func, agg_args, agg_kwargs, groupby_kwargs, drop=False
diff --git a/modin/data_management/functions/groupby_function.py b/modin/data_management/functions/groupby_function.py
--- a/modin/data_management/functions/groupby_function.py
+++ b/modin/data_management/functions/groupby_function.py
@@ -19,7 +19,36 @@
 
 class GroupbyReduceFunction(MapReduceFunction):
     @classmethod
-    def call(cls, map_func, reduce_func, *call_args, **call_kwds):
+    def call(cls, map_func, reduce_func=None, **call_kwds):
+        """
+        Build GroupbyReduce function.
+
+        Parameters
+        ----------
+        map_func: str, callable or dict,
+            If 'str' this parameter will be treated as a function name to register,
+            so 'map_func' and 'reduce_func' will be grabbed from 'groupby_reduce_functions'.
+            If dict or callable then this will be treated as a function to apply to each group
+            at the map phase.
+        reduce_func: callable or dict (optional),
+            A function to apply to each group at the reduce phase. If not specified
+            will be set the same as 'map_func'.
+        **call_kwds: kwargs,
+            Kwargs that will be passed to the returned function.
+
+        Returns
+        -------
+        Callable,
+            Function that executes GroupBy aggregation with MapReduce algorithm.
+        """
+        if isinstance(map_func, str):
+
+            def build_fn(name):
+                return lambda df, *args, **kwargs: getattr(df, name)(*args, **kwargs)
+
+            map_func, reduce_func = map(build_fn, groupby_reduce_functions[map_func])
+        if reduce_func is None:
+            reduce_func = map_func
         assert not (
             isinstance(map_func, dict) ^ isinstance(reduce_func, dict)
         ) and not (
@@ -48,6 +77,8 @@ def map(
         groupby_args["as_index"] = True
         groupby_args["observed"] = True
         if other is not None:
+            # Other is a broadcasted partition that represents 'by' columns
+            # Concatenate it with 'df' to group on its columns names
             other = other.squeeze(axis=axis ^ 1)
             if isinstance(other, pandas.DataFrame):
                 df = pandas.concat(
@@ -63,7 +94,8 @@ def map(
         result = apply_func(
             df.groupby(by=by_part, axis=axis, **groupby_args), **map_args
         )
-        return result
+        # Result could not always be a frame, so wrapping it into DataFrame
+        return pandas.DataFrame(result)
 
     @classmethod
     def reduce(
@@ -101,8 +133,8 @@ def reduce(
         if not as_index:
             insert_levels = partition_idx == 0 and (drop or method == "size")
             result.reset_index(drop=not insert_levels, inplace=True)
-
-        return result
+        # Result could not always be a frame, so wrapping it into DataFrame
+        return pandas.DataFrame(result)
 
     @classmethod
     def caller(
@@ -231,37 +263,14 @@ def wrapper(df):
         return _map, _reduce
 
 
+# This dict is a map for function names and their equivalents in MapReduce
 groupby_reduce_functions = {
-    "all": (
-        lambda df, *args, **kwargs: df.all(*args, **kwargs),
-        lambda df, *args, **kwargs: df.all(*args, **kwargs),
-    ),
-    "any": (
-        lambda df, *args, **kwargs: df.any(*args, **kwargs),
-        lambda df, *args, **kwargs: df.any(*args, **kwargs),
-    ),
-    "count": (
-        lambda df, *args, **kwargs: df.count(*args, **kwargs),
-        lambda df, *args, **kwargs: df.sum(*args, **kwargs),
-    ),
-    "max": (
-        lambda df, *args, **kwargs: df.max(*args, **kwargs),
-        lambda df, *args, **kwargs: df.max(*args, **kwargs),
-    ),
-    "min": (
-        lambda df, *args, **kwargs: df.min(*args, **kwargs),
-        lambda df, *args, **kwargs: df.min(*args, **kwargs),
-    ),
-    "prod": (
-        lambda df, *args, **kwargs: df.prod(*args, **kwargs),
-        lambda df, *args, **kwargs: df.prod(*args, **kwargs),
-    ),
-    "size": (
-        lambda df, *args, **kwargs: pandas.DataFrame(df.size(*args, **kwargs)),
-        lambda df, *args, **kwargs: df.sum(*args, **kwargs),
-    ),
-    "sum": (
-        lambda df, *args, **kwargs: df.sum(*args, **kwargs),
-        lambda df, *args, **kwargs: df.sum(*args, **kwargs),
-    ),
+    "all": ("all", "all"),
+    "any": ("any", "any"),
+    "count": ("count", "sum"),
+    "max": ("max", "max"),
+    "min": ("min", "min"),
+    "prod": ("prod", "prod"),
+    "size": ("size", "sum"),
+    "sum": ("sum", "sum"),
 }


Test patch: