Building an Automated Machine Learning Pipeline: Part Two | by Ceren Iyim | Towards Data Science

Part 1: Understand, clean, explore, process data
Part 2: Set metric and baseline, select and tune model (you are reading now)
Part 3: Train, evaluate and interpret model (live!)
Part 4: Automate your pipeline using Docker and Luigi (live!)
In this article series, we set our course to build a 9-step machine learning (ML) pipeline (we are calling it the wine rating predictor) and automate it. Eventually, we will observe how each step congregates and runs in production systems.
We are working on a supervised regression problem. We want to develop a performant, understandable and good wine rating predictor that can predict the points, a quality measure of wine.
In the first article, we defined the problem and our motivation behind building the wine rating predictor. Then, we had a detailed look at the data by visualizing the relationships between the features and the target along with the Understand & Clean & Format Data and Exploratory Data Analysis steps.
In the Feature Engineering & Pre-processing step, we added new and more useful features. Besides, we prepared the training and test datasets to use during training and evaluation of the model. As the final step of the first article, we created validation dataset out of the training dataset for model selection.
In this article, we are going to complete the following steps:
4. Set Evaluation Metric & Establish Baseline
5. Select an ML Model based on the Evaluation Metric
6. Perform Hyperparameter Tuning on the Selected Model
The code behind this article can be found in this notebook. The complete project is available on GitHub:
github.com
Feel free to share, fork, and utilize this repo for your projects!
The datasets that we are going to use is available in notebooks/transformed
X_train consisting of features (country, province, region_1, variety, price, year, taster_name, is_red, is_white, is_rose, is_dry, is_sweet, is_sparkling, is_blend) and y_train consisting of the target (points) to train the model
X_valid consisting of features and y_valid consisting of target to validate the model
We are going to need the following libraries of Python in this notebook:
Let’s load the datasets into dataframes and convert to an array using below functions:
X_train = pd.read_csv("transformed/X_train.csv")X_train_array = convert_features_to_array(X_train)X_valid = pd.read_csv("transformed/X_valid.csv")X_valid_array = convert_features_to_array(X_valid)y_train = pd.read_csv("transformed/y_train.csv")y_train_array = convert_target_to_array(y_train)y_valid = pd.read_csv("transformed/y_valid.csv")y_valid_array = convert_target_to_array(y_valid)
If this was a Kaggle competition, we would skip this step of the pipeline because we would be given with the evaluation metric.
However, in real-world applications of data science/machine learning, the evaluation metric is set by data scientists in line with the stakeholder’s expectations from the ML model. That is why this is an important step.
After we decide on our evaluation metric, to quantify our initial motive — building a good wine rating predictor and to compare our model’s performance against, we are going to form a baseline.
The mean square error (MSE) is set as the evaluation metric. It is the average of the sum of squared residuals where a ​residual is the subtraction of the actual values from the predicted values of the target variable.
In other words, evaluation of the model is done by looking at the measure of how large the squared errors (residuals) are spread out.
We selected MSE because it is interpretable and analogous to variance, and it is a widely-used optimization criterion among ML models. (E.g. linear regression, random forest)
A ​baseline​ can be explained as generating a naive guess of the target value by using expert knowledge or a few lines of code. It also helps to measure the performance of the ML model.
If the built-model (wine rating predictor) cannot beat this baseline, then the selected ML algorithm may not be the best approach to solve this problem or we might want to revisit the previous steps of the pipeline.
Recall that the points (target) has a normal distribution between 80 and 100. The mean is 88.45 and the variance is 9.1.
We are going to form the baseline by:
using these statistics and
calculating differences between each point in the validation set and the mean of the training dataset, then taking the average of the sum of the squared differences (similar approach of computing MSE):
# set baseline as mean of training set's target valuebaseline = (np            .mean(                y_train_array))# calculate MSE baselinemse_baseline = (np                .mean(                    np.square(                        baseline - y_valid_array)))
It is not a coincidence that the variance of the points and baseline error is almost equal. You can think of this baseline MSE as the manually calculated variance with a smaller set from our dataset.
This number (9.01) will accompany us in the next step — Select an ML Model based on the Evaluation Metric while we are testing different ML algorithms.
I always find useful trying out several algorithms with different rationales behind, because I believe this whole process involves experiment as well! (“scientist” part of being a data scientist makes sense now 🙃)
While searching for the best algorithm, we are going to observe both the improvement of the MSE and the run time of the algorithms (with %%time magic at the beginning of the cells) and compare the algorithm’s MSE to our baseline MSE. At the same time, we will keep in mind the understandable and performant requirements of the wine rating predictor.
We are going to experiment with one linear algorithm, two distance-based algorithm and two tree-based algorithms, ordered from the simplest to most complex:
Linear regression
K-nearest neighbors regressor
Support vector regressor
Random forest regressor
Light gradient boosting regressor
We are going to train them with the training set, and compare their generalization performances with the validation set. Below function will do the work for us.
At the end of this step, we are going to elaborate on how the selected algorithm works.
Linear regression slightly decreased the baseline metric, showing that it is not a candidate to be a good predictor.
Distance-based models use the euclidian distance (or other distance measures) for training, thus varying ranges causes distance-based models to generate inaccurate predictions. To apply distance-based algorithms, we scaled the datasets with normalization before here in the notebook.
K-nearest neighbors regressor performed better than the linear regression. However, MSE is still high, showing that this algorithm is not a good predictor as well.
Support vector regressor performed better than the k-nearest neighbors regressor at a higher run-time. All in all, MSE decreased 35% showing that this algorithm might be a candidate for building a good predictor.
Random forest regressor performed better than the support vector regressor at lesser run-times. It decreased the MSE 44% and replaced support vector regressor in the good-predictor list.
Light gradient boosting regressor (Light GBM) showed the best performance of all tried models. It also lowered the baseline MSE 45% and showed that it is a potential candidate for a good predictor at a lower run time.
As a summary:
It might not be a fair selection of algorithms since we only train them with the default hyperparameters. Nevertheless, this is the experimental step of the pipeline, that is why both light GBM and random forest algorithm has been given a chance for further improvements in the Perform Hyperparameter Tuning on the Selected Model step. (You can find it here in the notebook) In this article, we will use random forest regressor due to the understandable and performant requirements and we will only elaborate on it.
Before moving on to the next step, let’s understand how random forest regressor works:
Random forest regressor ​is an ensemble algorithm that builds multiple decision trees once and trains them on various sub-samples and various subsets of the features of the dataset. Its random selection of the dataset and feature subsamples makes this algorithm more robust.
A Decision tree​ uses a tree-like structure to make predictions. It breaks down a dataset into smaller and smaller subsets while at the same time an associated decision tree is incrementally developed. The final result is a tree with decision nodes and leaf nodes. A decision node has two or more branches, each representing values for the feature tested. Leaf node represents a final decision on the target.
A hyperparameter is the set of defined parameter(s) by the data scientist or ML engineer whose values are not affected by the training process of the model. On the other hand, a parameter of a model is searched and optimized by the model during the training process and affected by the dataset.
I loved this quotation from Jason Brownlee on Machine Learning Mastery to prevent confusion of the two:
“If you have to specify a model parameter manually then, it is probably a model hyperparameter.”
An example of a parameter from our simplest model: Coefficients of the linear regression model, which are optimized through the model training.
An example of a hyperparameter from our selected model: The number of trees constructed in a random forest model, which is specified by us or by scikit-learn.
Hyperparameter tuning is the process of defining, searching and perhaps improving the performance of the model further. We are going to search for the best set of parameters with the random search and k-fold cross-validation.
Random search is the process of searching for the combination of the defined parameters randomly and comparing the defined score (mean squared error, for this problem) in each iteration. It is fast and run-time-efficient, but you may not always find the most optimal set of hyperparameters due to the searching of a random combination of defined hyperparameters.
We are going to search for the below hyperparameters of random forest regressor with the hyperparameter_grid dictionary:
n_estimators: number of trees to be used in the model, default is 100.
min_samples_split: minimum number of samples required to split an internal node, default value is 2.
min_samples_leaf: minimum number of samples required to be at a leaf node, default value is 1.
max_features: number of features to consider when looking for the best split, default value is auto.
# define search parameters n_estimators = [100, 200, 300, 500, 1000] min_samples_split = [2, 4, 6, 10] min_samples_leaf = [1, 2, 4, 6, 8] max_features = ['auto', 'sqrt', 'log2', None]# Define the grid of hyperparameters to search hyperparameter_grid = {    "n_estimators": n_estimators,                   "min_samples_split": min_samples_split,                           "min_samples_leaf": min_samples_leaf,                             "max_features": max_features}
K-fold cross-validation is the method used to assess the performance of the model on the complete training dataset. Rather than splitting the dataset into two static subsets of training and validation set, the dataset is divided equally for the given K. Then the model is trained with K-1 subsets and tested on Kth subset iteratively. This process makes the model more robust to overfitting — more on that at the end of the article.
To perform hyperparameter tuning with random search and k-fold cross-validation we are going to add training and validation datasets and continue with one training set from now on.
# add dataframes back for to perform random search and cross-validationX = pd.concat([X_train, X_valid])y = pd.concat([y_train, y_valid])X_array = convert_features_to_array(X)y_array = convert_target_to_array(y)
Since we have less than 10.000 rows in the training dataset, we are going to perform 4 fold cross-validation to have sufficient number of data points in each fold. We are going to consolidate random search and k-fold cross-validation in the RandomizedSearchCV object:
rf_random_cv = RandomizedSearchCV(    estimator=rf,    param_distributions=hyperparameter_grid,    cv=4,     n_iter=25,    scoring='neg_mean_squared_error',    n_jobs=-1,     verbose=1,    return_train_score=True,    random_state=42)rf_random_cv.fit(X_array, y_array)
With the fit method, we initiate the search for the random combination of the values defined for each hyperparameter in the hyperparameter_grid. At the same time, "neg_mean_squared_error" is calculated for each fold in each iteration. We can observe the determined best set of parameters by calling the best_estimator_ method:
After hyperparameter tuning, the best set of hyperparameters are determined as:
n_estimators: 200
min_samples_split: 4
min_samples_leaf: 2
max_features: ‘sqrt’
Let’s see if those hyperparameters will help us to improve the MSE of the random forest regressor further.
rf_random_cv_model = rf_random_cv.best_estimator_
The MSE is decreased from 5.41 to 4.99 and tuned model’s run-time resulted in 1.12 seconds which is lower than the run-time of the initial random forest model (1.89 seconds). Hyperparameter tuning has not only improved the evaluation metric but also lowered the run-time in our case.
Given the sample dataset, the determined set of features and the tuned random forest regressor we have successfully built a good wine rating predictor without falling into the underfitting and overfitting areas!
Although this topic is another article by itself, I think it is important to address here since we mentioned these concepts.
Overfitting happens when the ML model perfectly fits (or memorizes) the training dataset rather than grasping the common patterns between the features and the target. An overfit model will have a high variance, I find a resemblance between this situation and a fragile glass home. It is perfectly built for its current conditions, but it is less likely to survive if the conditions are changed.
Did you remember that cross-validation makes our model more robust to overfitting?
The reason behind is that our model sees a different dataset in each fold during training — like a house already encounters different weather conditions. This improves the generalization performance which makes the model more resistant to overfitting.
Underfitting happens when the ML model fails to grasp the relationships between the features and the target due to not having enough data points or features. It may perform as bad as a random guess about the predictions. An underfit model will have a high bias, I think of this situation as a half-built shed. It should have to be undergone some construction to serve its purpose.
Did you remember the linear regression model that only showed a slight improvement on the baseline MSE?
That was an example of an underfitting. Linear regression model resulted in almost the same MSE as the baseline MSE and failed to understand the relationship between the features and the target.
All in all, both overfitting and underfitting decrease the generalization performance of the machine learning models, and results in unsatisfactory levels of evaluation metric. Although we did not test the tuned random forest regressor specifically the overfitting case, the 45% improvement shows that we are at the fine line between underfitting and overfitting.
Before moving on to the conclusions let’s save our model in the directory so that we can pick up where we left off in the last article and notebook.
filename = 'random_forests_model.sav'pickle.dump(rf_random_cv_model,            open(filename, 'wb'))
In this article, we completed the intermediate steps of the machine learning pipeline. After some quick recap on the first article and objectives, we
Set the evaluation metric as mean square error due to its wide-use in ML algorithms and explainability.Established a baseline aligning with the mean square error calculation which resulted in 9.01.Tried several different ML algorithms, and selected random forest regressor which reported MSE of 5.4 at 1.93 seconds run time.Fine-tuned hyperparameters of the random forest regressor and achieved 8% performance improvement compared to the initial random forest model (MSE: 4.99 at 1.12 seconds run time).
Set the evaluation metric as mean square error due to its wide-use in ML algorithms and explainability.
Established a baseline aligning with the mean square error calculation which resulted in 9.01.
Tried several different ML algorithms, and selected random forest regressor which reported MSE of 5.4 at 1.93 seconds run time.
Fine-tuned hyperparameters of the random forest regressor and achieved 8% performance improvement compared to the initial random forest model (MSE: 4.99 at 1.12 seconds run time).
If we consider our initial motivation (building a good wine predictor) and the baseline MSE (9.01), we are going on the right path as we approach the end of the pipeline.
We have significantly lowered the baseline MSE, from 9.01 to 4.99 which results in 45% improvement!
The third article will start by loading the fine-tuned random forest regressor model and will focus on the evaluation of the model with the test set, as well as the results of it. (steps 7, 8, and 9).
towardsdatascience.com
The last article will automate this pipeline with Docker and Luigi.
towardsdatascience.com
Thanks for reading 😊
For comments or constructive feedback, you can reach out to me on responses, Twitter or Linkedin!
Stay safe and healthy until then 👋