Repository: skekre98/simple_learn (language: python)

Issue:
Create SimpleRegressor class
**Description of feature/enhancement**
This class should handle generating the most accurate model for for any given regression problem.

**Description of implementation**
The class should have a function called `fit` which creates the most accurate regression model for any given data set.

**Why this is a good feature/enhancement**
This will allow users to simply pass in data and be returned the most accurate model without having to worry about fine tuning the hyper-parameters themselves.

**Additional context**
n/a

Add logging for failed models
**Description of feature/enhancement**
Currently some models fail on certain types of data. Instead of the entire class throwing an error I think this should be handled in a `try/except` block and log the failed models at the end of the function.

**Description of implementation**
The implementation should be something along the following lines:
```python
for model in models:
    try:
        gridsearch(model)
        log(model.name + " succeeded")
    except ModelError:
        log(model.name + " failed")
```

**Why this is a good feature/enhancement**

This will do two things:
1. Allow working models to still be created
2. Allow advanced users to better understand their data

**Additional context**
Try logging in more of an alerting fashion than an absolute error. I recommend using the `logging` library.



Discussion / hints:
Hey @skekre98 I would like to help with this issue and writing tests for it while i'm writing the class ( #11 ).
I thought about adding the following regressors to `param_grid` (in another dictionary) for a start : 

"SGDRegressor"
"KNeighborsRegressor"
"DecisionTreeRegressor"
"RandomForestRegressor"
"GradientBoostingRegressor"
"AdaBoostRegressor"

They seem to cover easier to harder cases of data. Then We can see how they work out and improve from there.

Let me know what you think.


Hi @eranns, sorry about my absence. I think it would be a great idea to start with these regressors. Instead of adding the regressor `param_map` to the current file, let's go ahead and create a separate directory for regressors and add another file called `param_grid.py` there. 

I'll go ahead and assign this and #11 over to you 😄 


Fix patch:
diff --git a/scripts/custom.sh b/scripts/custom.sh
index 45500fa..ef9525f 100755
--- a/scripts/custom.sh
+++ b/scripts/custom.sh
@@ -85,4 +85,6 @@ function build_and_release {
 function run_tests {
     python -W ignore -m unittest tests/classifiers/simple_classifier_tests.py -v
     python -W ignore -m unittest tests/classifiers/simple_classifier_list_tests.py -v
+    python -W ignore -m unittest tests/regressors/simple_regressor_tests.py -v
+
 }
diff --git a/simple_learn/regressors/__init__.py b/simple_learn/regressors/__init__.py
new file mode 100644
index 0000000..e10dd57
--- /dev/null
+++ b/simple_learn/regressors/__init__.py
@@ -0,0 +1,21 @@
+# Copyright (c) 2020 Sharvil Kekre skekre98
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+from simple_learn.regressors.simple_regressor import SimpleRegressor
diff --git a/simple_learn/regressors/param_grid.py b/simple_learn/regressors/param_grid.py
new file mode 100644
index 0000000..b68c563
--- /dev/null
+++ b/simple_learn/regressors/param_grid.py
@@ -0,0 +1,70 @@
+# Copyright (c) 2020 Sharvil Kekre skekre98
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import numpy as np
+
+model_param_map = {
+    "SGDRegressor": {
+        "loss": [
+            "squared_loss",
+            "huber",
+            "epsilon_insensitive",
+            "squared_epsilon_insensitive",
+        ],
+        "penalty": ["l1", "l2", "elasticnet"],
+        "alpha": [0.0001, 0.001],
+        "learning_rate": ["constant", "optimal", "invscaling", "adaptive"],
+        "eta0": [0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2],
+        "max_iter": [10000],
+    },
+    "KNeighborsRegressor": {
+        "n_neighbors": np.arange(3, 15),
+        "weights": ["uniform", "distance"],
+        "algorithm": ["ball_tree", "kd_tree", "brute"],
+        "p": [1, 2],
+    },
+    "DecisionTreeRegressor": {
+        "criterion": ["mse", "friedman_mse", "mae", "poisson"],
+        "splitter": ["best", "random"],
+        "max_depth": np.arange(3, 15),
+        "max_features": ["auto", "sqrt", "log2"],
+    },
+    "RandomForestRegressor": {
+        "n_estimators": [200, 500],
+        "criterion": ["mse", "mae"],
+        "max_depth": [4, 16, 32, 64, 128],
+    },
+    "GradientBoostingRegressor": {
+        "loss": ["ls", "lad", "huber", "quantile"],
+        "learning_rate": [0.1, 0.01, 0.001],
+        "n_estimators": [50, 100],
+        "criterion": ["friedman_mse", "mse"],
+        "max_depth": [3, 6, 9],
+        "max_features": ["auto", "sqrt", "log2"],
+    },
+    "HistGradientBoostingRegressor": {
+        "loss": ["least_squares", "least_absolute_deviation", "poisson"],
+        "learning_rate": [0.01, 0.025, 0.05, 0.075, 0.1, 0.15, 0.2],
+    },
+    "AdaBoostRegressor": {
+        "n_estimators": [50, 100],
+        "loss": ["linear", "square", "exponential"],
+    },
+}
diff --git a/simple_learn/regressors/simple_regressor.py b/simple_learn/regressors/simple_regressor.py
new file mode 100644
index 0000000..f8ff037
--- /dev/null
+++ b/simple_learn/regressors/simple_regressor.py
@@ -0,0 +1,147 @@
+# Copyright (c) 2020 Sharvil Kekre skekre98
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import json
+import logging
+import time
+
+import numpy as np
+from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
+from sklearn.model_selection import GridSearchCV
+from sklearn.utils import all_estimators
+
+from simple_learn.regressors.param_grid import model_param_map
+
+
+class SimpleRegressor:
+    """
+        A class used to simplify the creation of regression models
+        ...
+        Attributes
+        ----------
+        name : str
+            the optimal model algorithm for given dataset
+        sk_learn : str
+            the sklearn model used for prediction
+        attributes : dict
+            a dictionary used to keep track of model hyper-parameters
+        metrics : dict
+            a dictionary to keep track of scoring metrics
+        gridsearch_duration : time.time
+            the duration of the gridsearch being used in hyper-parameter tuning
+        train_duration : time.time
+            the duration of model training
+        Methods
+        -------
+        fit(train_x, train_y, folds=3)
+            Fits a given dataset onto SimpleRegressor
+        predict(pred_x)
+            Predicts label of samples in prediction array
+        """
+
+    def __init__(self):
+        self.name = "Empty Model"
+        self.sk_model = None
+        self.attributes = dict()
+        self.metrics = dict()
+        self.gridsearch_duration = None
+        self.train_duration = None
+        self.failed_models = []
+        self.logger = logging.getLogger()
+
+    def __str__(self):
+
+        for k in self.attributes:
+            if type(self.attributes[k]) == np.int64:
+                self.attributes[k] = int(self.attributes[k])
+
+        attr = {
+            "Type": self.name,
+            "Training Duration": "{}s".format(self.train_duration),
+            "GridSearch Duration": "{}s".format(self.gridsearch_duration),
+            "Parameters": self.attributes,
+            "Metrics": self.metrics,
+        }
+
+        return json.dumps(attr, indent=4)
+
+    def fit(self, train_x, train_y, folds=3):
+        """Trains the optimal regression model
+                 on given dataset by running model algorithm search.
+                 If the argument folds isn't passed, the default
+                 value(3) is used.
+                 Parameters
+                 ----------
+                 train_x : numpy.ndarray
+                     The features for training classification model
+                 train_y : numpy.ndarray
+                     The corresponding label for feature array
+                 folds : int, optional
+                     The number of folds for cross validation
+                 """
+
+        estimators = all_estimators(type_filter="regressor")
+        for name, RegressionClass in estimators:
+            if name in model_param_map:
+                param_grid = model_param_map[name]
+                grid_clf = GridSearchCV(
+                    RegressionClass(),
+                    param_grid,
+                    cv=folds,
+                    scoring="neg_root_mean_squared_error",
+                    verbose=0,
+                    n_jobs=-1,
+                    error_score="raise",
+                )
+                start = time.time()
+                try:
+                    grid_clf.fit(train_x, train_y)
+                except ValueError as value_error:
+                    self.failed_models.append(name)
+                    self.logger.warning(f"{name} failed, Error : {value_error}")
+                    continue
+                end = time.time()
+                if self.metrics.get(
+                    "Training Score"
+                ) is None or -grid_clf.best_score_ < self.metrics.get("Training Score"):
+                    self.metrics["Training Score"] = -grid_clf.best_score_
+                    pred_y = grid_clf.predict(train_x)
+                    self.metrics["Mean Absolute Error"] = mean_absolute_error(
+                        train_y, pred_y
+                    )
+                    self.metrics["Mean Square Error"] = mean_squared_error(
+                        train_y, pred_y
+                    )
+                    self.metrics["R-Squared"] = r2_score(train_y, pred_y)
+                    self.sk_model = grid_clf.best_estimator_
+                    self.name = name
+                    self.attributes = grid_clf.best_params_
+                    self.train_duration = grid_clf.refit_time_
+                    self.gridsearch_duration = end - start
+
+    def predict(self, pred_x):
+        """Predicts class label based on input
+                feature array
+                Parameters
+                ----------
+                pred_x : numpy.ndarray
+                    The feature array for predicting class labels
+                """
+        return self.sk_model.predict(pred_x)


Test patch:
diff --git a/tests/regressors/simple_regressor_tests.py b/tests/regressors/simple_regressor_tests.py
new file mode 100644
index 0000000..8e41f31
--- /dev/null
+++ b/tests/regressors/simple_regressor_tests.py
@@ -0,0 +1,84 @@
+# Copyright (c) 2020 Sharvil Kekre skekre98
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+import unittest
+import warnings
+
+warnings.filterwarnings("ignore")
+import numpy as np
+from scipy.stats import levene
+from sklearn import datasets
+from sklearn.datasets import make_regression
+
+from simple_learn.regressors import SimpleRegressor
+
+
+class TestSimpleRegressor(unittest.TestCase):
+    def test_init(self):
+        clf = SimpleRegressor()
+        self.assertEqual(clf.name, "Empty Model")
+
+    def test_fail_models(self):
+        """
+        testing warnings of failed models with negative data value in y
+        """
+        true_x, true_y = (
+            [
+                np.random.standard_normal(10),
+                np.random.standard_normal(10),
+                np.random.standard_normal(10),
+            ],
+            np.arange(-3, 0),
+        )
+        clf = SimpleRegressor()
+        with self.assertLogs(clf.logger, level="WARNING") as cm:
+            clf.fit(true_x, true_y)
+            self.assertEqual(len(cm.output), len(clf.failed_models))
+
+    def test_regression(self):
+        true_x, true_y = make_regression(
+            n_features=4, n_informative=2, random_state=10, shuffle=False
+        )
+
+        clf = SimpleRegressor()
+        clf.fit(true_x, true_y)
+        self.assertIsNotNone(clf.sk_model)
+        self.assertTrue(clf.metrics["Training Score"] > 0.0)
+        pred_y = clf.predict(true_x)
+
+        stat, p = levene(true_y, pred_y)
+        self.assertTrue(p > 0.05)
+
+    def test_boston(self):
+        boston = datasets.load_boston()
+        true_x = boston.data
+        true_y = boston.target
+
+        clf = SimpleRegressor()
+        clf.fit(true_x, true_y)
+        self.assertIsNotNone(clf.sk_model)
+        self.assertTrue(clf.metrics["Training Score"] > 0.0)
+        pred_y = clf.predict(true_x)
+        stat, p = levene(true_y, pred_y)
+        self.assertTrue(p > 0.05)
+
+
+if __name__ == "__main__":
+    unittest.main()