Repository: mindsdb/lightwood

Issue:
🔧 Implement a "most popular" ensemble for categorical tasks
## Introduction

Lightwood is an AutoML library that aims at abstracting common procedures in the Machine Learning workflow (e.g. data preparation, feature engineering, model building, model analysis) with an approach that can be fully automated but easily customizable and extended thanks to its flexibility. 

One of the key components in model building are "mixers", models that intake features and learn to predict the target column. For any given predictive task, there might be a number of valid mixers that will get trained.

After all mixers are trained, the [`BaseEnsemble`](https://github.com/mindsdb/lightwood/blob/staging/lightwood/ensemble/base.py) abstraction enables ML practitioners to aggregate the predictions of all mixers as they deem convenient.

For example [`BestOf`](https://github.com/mindsdb/lightwood/blob/staging/lightwood/ensemble/best_of.py), the only currently available ensemble, is a simple procedure that evaluates all models and selects the best one based on validation accuracy. 

## Task

In this task, we seek to implement a `ModeEnsemble` for categorical targets. Simply put, it should obtain the predictions of all internal mixers, then output the most popular predicted label. In case of a tie (might happen with an even amount of mixers), you can propose an initial approach and we can discuss further in this issue's comment section.

## Steps :male_detective: :female_detective: 

- Fork the Lightwood repository, checkout the `staging` branch and from it create a new one.
- Implement the `ModeEnsemble` class. Any files should be added inside the `lightwood/ensemble/` directory.
- Implement a unit test where your ensemble is successfully used with a sample categorical task, like [HDI](https://github.com/mindsdb/lightwood/blob/staging/tests/data/hdi.csv). For guidance on how to trigger your custom ensemble, you can check out `Step 3` in the [custom mixer tutorial](https://lightwood.io/tutorials/custom_mixer/custom_mixer.html).
- Make the PR and address any comments that reviewers might make.

## Additional rewards :1st_place_medal: 

Accepted PRs for ensemble challenges bring :five: points for entry into the draw for a :computer:  Deep Learning Laptop powered by the NVIDIA RTX 3080 Max-Q GPU or other swag :shirt: :bear: .  For more info check out https://mindsdb.com/hacktoberfest/


Discussion:
Hey @paxcema I can work on this!!
Forgot to mention this bit @Santhoshkumard11: 

> For guidance on how to trigger your custom ensemble, you can check out `Step 3` in the [custom mixer tutorial](https://lightwood.io/tutorials/custom_mixer/custom_mixer.html).

It's the same idea, but you would modify the `.ensemble` key instead.
@paxcema Thanks, will have a look at it. Mostly will try to finish it off by this weekend.
@paxcema 
I have started to implement the ModeEnsemble, but just need this clarified.
What do you exactly mean by  **output the most popular predicted label**?? I'm using HDI for testing.
It has four labels (1,2,3,4)
Hey @Santhoshkumard11, that's great!

Let's go through an example. Imagine you have trained three mixers to predict "Development Index" in the HDI dataset. What I mean by "output the most popular predicted label" is, essentially, that for each `MostPopular.predict()` call you will get the predicted label that each mixer outputs (say, `l1`, `l2` and `l3`). From those three, there might be a label that is the most popular, e.g. if `[l1, l2, l3] = ['1', '2', '2']`, then the most popular label would be '2', and that should be the output of the `MostPopular` ensemble. As the issue says, you can propose an approach to deal with ties, and we can discuss it in the PR submission.

Hopefully this makes sense, let me know!
@paxcema 
Yup! Got it now.
I Will work on the same. We could have weights for the mixers.
Neural Net would obviously be more accurate (at least for most cases) so we will assign weights for each mixer

Resolving patch:
diff --git a/lightwood/ensemble/__init__.py b/lightwood/ensemble/__init__.py
--- a/lightwood/ensemble/__init__.py
+++ b/lightwood/ensemble/__init__.py
@@ -1,6 +1,6 @@
 from lightwood.ensemble.base import BaseEnsemble
 from lightwood.ensemble.best_of import BestOf
 from lightwood.ensemble.mean_ensemble import MeanEnsemble
+from lightwood.ensemble.mode_ensemble import ModeEnsemble
 
-
-__all__ = ['BaseEnsemble', 'BestOf', 'MeanEnsemble']
+__all__ = ['BaseEnsemble', 'BestOf', 'MeanEnsemble', "ModeEnsemble"]
diff --git a/lightwood/ensemble/mode_ensemble.py b/lightwood/ensemble/mode_ensemble.py
new file mode 100644
--- /dev/null
+++ b/lightwood/ensemble/mode_ensemble.py
@@ -0,0 +1,80 @@
+from typing import List, Optional, Dict
+
+import pandas as pd
+import numpy as np
+
+from lightwood.mixer.base import BaseMixer
+from lightwood.ensemble.base import BaseEnsemble
+from lightwood.api.types import PredictionArguments
+from lightwood.data.encoded_ds import EncodedDs
+from lightwood import dtype
+from lightwood.helpers.general import evaluate_accuracy
+from lightwood.helpers.numeric import can_be_nan_numeric
+from lightwood.helpers.log import log
+
+
+class ModeEnsemble(BaseEnsemble):
+    mixer_scores: Dict[str, float]
+
+    def __init__(self, target, mixers: List[BaseMixer], data: EncodedDs, dtype_dict: dict,
+                 accuracy_functions, args: PredictionArguments, ts_analysis: Optional[dict] = None) -> None:
+        super().__init__(target, mixers, data)
+        self.mixer_scores = {}
+
+        if dtype_dict[target] not in (dtype.binary, dtype.categorical, dtype.tags):
+            raise Exception(
+                'This ensemble can only be used in classification problems! ' +
+                f'Got target dtype {dtype_dict[target]} instead!')
+
+        for _, mixer in enumerate(mixers):
+            score_dict = evaluate_accuracy(
+                data.data_frame,
+                mixer(data, args)['prediction'],
+                target,
+                accuracy_functions,
+                ts_analysis=ts_analysis
+            )
+            avg_score = np.mean(list(score_dict.values()))
+            log.info(f'Mixer: {type(mixer).__name__} got accuracy: {avg_score}')
+
+            if can_be_nan_numeric(avg_score):
+                avg_score = -pow(2, 63)
+                log.warning(f'Change the accuracy of mixer {type(mixer).__name__} to valid value: {avg_score}')
+
+            self.mixer_scores[f'__mdb_mixer_{type(mixer).__name__}'] = avg_score
+
+    def _pick_mode_highest_score(self, prediction: pd.Series):
+        """If the predictions are unimodal, return the mode. If there are multiple modes, return the mode whose voting
+        mixers have the highest score."""
+        prediction_counts = prediction.value_counts()
+
+        # If there is a clear winner, i.e. only one prediction
+        if len(prediction_counts) == 1:
+            return prediction_counts.index[0]
+
+        counts = prediction_counts.values  # how many times all predictions have appeared
+        max_count = np.max(counts)  # how many times the most frequent predictions have apppeared
+
+        # most frequent predictions and how many times they appeared
+        modes = prediction_counts[prediction_counts == max_count]
+
+        modes_predictions = modes.index  # most frequent predictions
+
+        # For each mode, get the sum of the scores of the predictors who voted for it
+        modes_predictions_scores = {}
+        for mode_prediction in modes_predictions:
+            voting_mixers_name = prediction[prediction == mode_prediction].index.tolist()
+            modes_predictions_scores[mode_prediction] = np.sum(
+                [self.mixer_scores[mixer_name] for mixer_name in voting_mixers_name])
+
+        # Return the mode with the maximum sum of accuracies
+        return max(modes_predictions_scores, key=modes_predictions_scores.get)
+
+    def __call__(self, ds: EncodedDs, args: PredictionArguments) -> pd.DataFrame:
+        predictions_df = pd.DataFrame()
+        for mixer in self.mixers:
+            predictions_df[f'__mdb_mixer_{type(mixer).__name__}'] = mixer(ds, args=args)['prediction']
+
+        mode_df = predictions_df.apply(func=self._pick_mode_highest_score, axis='columns')
+
+        return pd.DataFrame(mode_df, columns=['prediction'])


Test patch:
diff --git a/tests/integration/basic/test_ensembles.py b/tests/integration/basic/test_ensembles.py
--- a/tests/integration/basic/test_ensembles.py
+++ b/tests/integration/basic/test_ensembles.py
@@ -1,6 +1,6 @@
 import unittest
 import pandas as pd
-from sklearn.metrics import r2_score
+from sklearn.metrics import r2_score, accuracy_score
 
 from lightwood.api.high_level import code_from_json_ai, json_ai_from_problem, predictor_from_code
 from lightwood.api.types import ProblemDefinition
@@ -31,3 +31,30 @@ def test_0_mean_ensemble(self):
         predictions = predictor.predict(df)
 
         self.assertTrue(r2_score(df[target], predictions['prediction']) > 0.5)
+
+    def test_1_mode_ensemble(self):
+
+        df = pd.read_csv('tests/data/hdi.csv')
+
+        target = 'Development Index'
+
+        json_ai = json_ai_from_problem(df, ProblemDefinition.from_dict({
+            'target': target,
+            'time_aim': 5
+        }))
+
+        json_ai.outputs[target].ensemble = {
+            'module': 'ModeEnsemble',
+            "args": {
+                'dtype_dict': '$dtype_dict',
+                "args": "$pred_args",
+                "accuracy_functions": "$accuracy_functions",
+            }
+        }
+
+        code = code_from_json_ai(json_ai)
+        predictor = predictor_from_code(code)
+        predictor.learn(df)
+        predictions = predictor.predict(df)
+
+        self.assertTrue(accuracy_score(df[target].astype(int), predictions['prediction'].astype(int)) > 0.5)