Creating Machine Learning Models. CREATING AND TESTING MULTIPLE MODELS... | by Rising Odegua | Towards Data Science

Now that we have successfully done the hard part-data cleaning and wrangling- of every data science project, we will move on to the interesting part, modelling!
Although this article can stand alone, it is advisable to go through part 1 and part 2 first.
Modeling in machine learning is an iterative phase where a data scientist continually train and test machine learning models to discover the best one for the given task.
Different machine learning models exist and the choice of which one to use generally depends on the problem at hand. No machine learning model works best for all kind of problems. So, your job in this stage is test multiple models and fine tune parameters to squeeze out every bit of accuracy.
Note: While we generally strive for higher performing models, it is generally wiser and better to go for simpler models that perform almost as well as complex models.
Save the .0001 increase in performance for your machine learning competitions because most of the time in the real world those minute difference do not really matter because we need simpler interpretable models.
With that in mind, let’s get to work. It is always good to start with a simple base model so you can get a base line to measure performance against.
Since our data set is small, we’ll use cross validation and for performance metric, we’ll use the mean absolute error. Cross Validation helps measure our model’s true performance.
We create a cross Validation function as shown below.
We start our modelling with a simple linear regression, cross validate and print it’s score.
Linear regression is simple and as such does not have much parameter to tune. So we simply use the default parameters.
MAE Sccore:  0.42615411746170206
Next, we use another simple but efficient algorithm called decision tree
The decision tree contains many hyperparameter, but the most important ones we can quickly tune are:
max_depth: The max_depth parameter indicates how deep the tree can be. The deeper the tree, the more complex the model gets and the higher it can capture more information about the data. If your model is too simple, try increasing the max_depth and vice-versa.min_samples_leaf: This is the minimum number of samples required to be at a leaf node. Higher leaf number result in simpler model and may help in overfitting, whereas smaller numbers could result in underfiting.min_samples_split: This is the minimum number of samples required to split an internal node. Increase in this parameter result in a more simpler/constrained model.max_features: This is the number of features the model considers before splitting a node.
max_depth: The max_depth parameter indicates how deep the tree can be. The deeper the tree, the more complex the model gets and the higher it can capture more information about the data. If your model is too simple, try increasing the max_depth and vice-versa.
min_samples_leaf: This is the minimum number of samples required to be at a leaf node. Higher leaf number result in simpler model and may help in overfitting, whereas smaller numbers could result in underfiting.
min_samples_split: This is the minimum number of samples required to split an internal node. Increase in this parameter result in a more simpler/constrained model.
max_features: This is the number of features the model considers before splitting a node.
MAE Sccore:  0.4280371717119684
Next, we try a popular distance based algorithm called k-nearest neighbor
The KNN contains a few hyperparameter. The most important one to tune is:
n_neighbors: This is the number of neighbors to use when voting. The higher the number, the more accurate it gets (sometimes though) but at an expense of speed.
n_neighbors: This is the number of neighbors to use when voting. The higher the number, the more accurate it gets (sometimes though) but at an expense of speed.
MAE Sccore:  0.431624658307474
Looking at the MAE(mean absolute error) score of the simple models, we see that the decision tree is the best so far. This will be our base line model. Any model that performs better becomes the next base line to beat.
Now, we’ll move from simple to advanced models. Many high performing machine learning models are usually based on ensembles. Ensembling is a method that combines multiple machine learning algorithms to obtain better predictive performance than could be achieved by any single one.
We’ll use some popular ensemble method below and compare their performance.
The Random forest algorithm is a popular and effective implementation of bagging, that builds multiple trees from a bootstrap sample. It has high predictive power and works very well out of box.
Random Forest contains may hyperparameter, but the most important ones to tune are:
n_estimators: This is the number of decision trees to be created. As we might guess, a higher number of trees will result in a better model but at a cost of higher training time.max_depths: This is the maximum depth of the individual trees. Higher number result in simpler trees and hence simpler random forest ensemble.max_features: This defines the maximum number of features that is allowed for splitting.min_samples_split: This is the minimum number of samples required in a leaf node before a splitting.
n_estimators: This is the number of decision trees to be created. As we might guess, a higher number of trees will result in a better model but at a cost of higher training time.
max_depths: This is the maximum depth of the individual trees. Higher number result in simpler trees and hence simpler random forest ensemble.
max_features: This defines the maximum number of features that is allowed for splitting.
min_samples_split: This is the minimum number of samples required in a leaf node before a splitting.
MAE Sccore:  0.41512597643232896
The Extra Tree algorithm is another popular implementation of bagging, it is similar to the random forest, but builds its base trees differently and is generally faster. It also has high predictive power sometimes outperforming Random Forest.
Extra Trees contains similar hyperparameter to Random Forest. Important parameters to tune are:
n_estimators: This is the number of decision trees to be created. As we might guess, a higher number of trees will result in a better model but at a cost of higher training time.max_depths: This is the maximum depth of the individual trees. Higher number result in simpler trees and hence simpler random forest ensemble.max_features: This defines the maximum number of features that is allowed for splitting.min_samples_split: This is the minimum number of samples required in a leaf node before a splitting.
n_estimators: This is the number of decision trees to be created. As we might guess, a higher number of trees will result in a better model but at a cost of higher training time.
max_depths: This is the maximum depth of the individual trees. Higher number result in simpler trees and hence simpler random forest ensemble.
max_features: This defines the maximum number of features that is allowed for splitting.
min_samples_split: This is the minimum number of samples required in a leaf node before a splitting.
MAE Sccore:  0.4131678014407999
The Bagging estimator in sklearn allows you to create bagging ensembles from any chosen model. I.e you can create bagging ensembles from single models like Linear regression or even ensembles like Random Forest or Extra Trees.
The important hyperparameter to tune here are:
base_estimator: This is the estimator to use in performing bagging. The bagging meta-estimator builds multiple models from the specified base estimator.n_estimators: This is the number of base estimator to be created. A higher number of estimators will result in a better model but also at a cost of higher training time.max_samples: This specifies the maximum number of samples to train each base estimator on.max_features: This is the maximum number of features to draw from the data set when training each base estimator
base_estimator: This is the estimator to use in performing bagging. The bagging meta-estimator builds multiple models from the specified base estimator.
n_estimators: This is the number of base estimator to be created. A higher number of estimators will result in a better model but also at a cost of higher training time.
max_samples: This specifies the maximum number of samples to train each base estimator on.
max_features: This is the maximum number of features to draw from the data set when training each base estimator
MAE Sccore:  0.4045400489482442
From the error metrics, our next best model is the bagging regressor, which is based on the Extra Tree Regressor. This becomes our new baseline model.
Next, let’s try some boosting algorithms.
Boosting is another popular and effective ensembling technique. In Boosting, multiple models are trained sequentially. The goal is to train models that do better than their predecessors. This means we have to take into account the areas where the previous models performed poorly and improve on those areas.
We start with the vanilla implementation of boosting called the AdaBoost. This is a old but widely used boosting technique that performs well on regression task, although it is known to overfit alot.
The important hyperparameter to tune here are:
n_estimators: This is the number of base estimators to use. A higher value normally gives better performance.learning_rate:This parameter controls the training weight update. It is usually tuned together with the n_estimators. A popular rule of thumb is that “When you increase the n_estimator by a factor of 10, you decrease the learning rate by a factor of 10 too.base_estimator:This specifies the base model to boost.max_depth: This specifies the depth of the base estimators.
n_estimators: This is the number of base estimators to use. A higher value normally gives better performance.
learning_rate:This parameter controls the training weight update. It is usually tuned together with the n_estimators. A popular rule of thumb is that “When you increase the n_estimator by a factor of 10, you decrease the learning rate by a factor of 10 too.
base_estimator:This specifies the base model to boost.
max_depth: This specifies the depth of the base estimators.
MAE Sccore:  0.4243339946387083
Another popular boosting algorithm is the Gradient Boosting.
The important hyperparameter to tune here are:
n_estimators: This is the number of base estimators to use. A higher value normally gives better performance.max_depth: This specifies the depth of the base estimators.min_samples_split: This specifies the minimum number of samples required to split an internal node.max_features: The maximum number of features to draw from the data set when training each base estimator
n_estimators: This is the number of base estimators to use. A higher value normally gives better performance.
max_depth: This specifies the depth of the base estimators.
min_samples_split: This specifies the minimum number of samples required to split an internal node.
max_features: The maximum number of features to draw from the data set when training each base estimator
MAE Sccore:  0.4063934439500354
“When in doubt use XGBoost” has been one of the most popular quotes from the kaggle platform in recent years.
XGBoost is efficient, fast and works well out of the box. It is an advance implementation of Gradient Boosting and a very efficient one indeed.
The important hyperparameter to tune here are:
eta: Just a different name for learning rate. It helps in model weight update.min_child_weight: This specifies the minimum sum of weights of all features required in a child node. It is mostly used to control overfitting.max_depth: This is used to define the maximum depth of each tree node. Higher depth will allow the model to be more complex and vice versa.max_leaf_nodes: This specifies the maximum number of leaves in a tree.Gamma: The Gamma value specifies the minimum loss reduction required to make a split. A node is split only when the resulting split gives a positive reduction in the loss function. The values can vary depending on the loss function and should be tuned.Subsample: Specifies the fraction of features to be randomly selected for each tree. Lower values make the algorithm more simple and prevent overfitting.
eta: Just a different name for learning rate. It helps in model weight update.
min_child_weight: This specifies the minimum sum of weights of all features required in a child node. It is mostly used to control overfitting.
max_depth: This is used to define the maximum depth of each tree node. Higher depth will allow the model to be more complex and vice versa.
max_leaf_nodes: This specifies the maximum number of leaves in a tree.
Gamma: The Gamma value specifies the minimum loss reduction required to make a split. A node is split only when the resulting split gives a positive reduction in the loss function. The values can vary depending on the loss function and should be tuned.
Subsample: Specifies the fraction of features to be randomly selected for each tree. Lower values make the algorithm more simple and prevent overfitting.
MAE Sccore:  0.4099352064907469
LightGBM is a boosting algorithm from Microsoft. It is one of the fastest boosting algorithm out there; efficient out-of-the-box and currently the go-to when it comes to large structured data.
The important hyperparameter to tune here are:
num_iterations: This specifies the number of boosting iterations to be performed.num_leaves : Specifies the number of leaves to be formed in a tree. Note: In Light GBM, since splitting takes place leaf-wise rather than depth-wise, num_leaves must be smaller than 2^(max_depth), otherwise, it may lead to overfitting.min_data_in_leaf : It is one of the most important parameters in dealing with overfitting.max_depth: Specifies the maximum depth to which a tree can grow. A very high value for this parameter can cause overfitting.bagging_fraction: It is used to specify the fraction of data to be used for each iteration. This parameter is generally used to speed up the training.
num_iterations: This specifies the number of boosting iterations to be performed.
num_leaves : Specifies the number of leaves to be formed in a tree. Note: In Light GBM, since splitting takes place leaf-wise rather than depth-wise, num_leaves must be smaller than 2^(max_depth), otherwise, it may lead to overfitting.
min_data_in_leaf : It is one of the most important parameters in dealing with overfitting.
max_depth: Specifies the maximum depth to which a tree can grow. A very high value for this parameter can cause overfitting.
bagging_fraction: It is used to specify the fraction of data to be used for each iteration. This parameter is generally used to speed up the training.
MAE Sccore:  0.4084352064907469
Catboost is a recent boosting algorithm. It is fast, efficient and handles categorical features easily (One of its best feature).
CatBoost can automatically deal with categorical variables and does not require extensive data preprocessing like other machine learning algorithms
The important hyperparameter to tune here are:
loss_function: Specifies the metric to be used for training.Iterations: The maximum number of trees that can be built.learning_rate: Specifies the learning rate which is used for reducing the gradient step.depth: Specifies the depth of the trees to be created.
loss_function: Specifies the metric to be used for training.
Iterations: The maximum number of trees that can be built.
learning_rate: Specifies the learning rate which is used for reducing the gradient step.
depth: Specifies the depth of the trees to be created.
MAE Score: 0.4123613834195344
In this post, we’ve learnt how to take a preprocessed data set and try out different machine learning models. Although this article was based on a regression task, the same steps can be applied for a classification task. Some key takeaways from this post are:
Pick a good validation strategy. (Cross validation for small datasets, Splitting for large data sets)Start simple, get a baseline model.Continually improve the model by tuning and measuring performance.
Pick a good validation strategy. (Cross validation for small datasets, Splitting for large data sets)
Start simple, get a baseline model.
Continually improve the model by tuning and measuring performance.
More advance technique after hand tuning your best model is to use an automatic hyper parameter searching algorithm such as Grid or Random search.
And remember, keep your model as simple and interpretable as possible.
If you learnt from these series of posts, don’t forget to clap and share.
Link to full Notebook with code and explanations on GitHub.
Connect with me on Twitter.
Connect with me on LinkedIn.