Supervised Machine Learning: Model Validation, a Step by Step Approach
By: Shreyas Jothish
Tags: Supervised Learning, Machine Learning, Model Validation, Gridsearchcv

Supervised Machine Learning: Model Validation, a Step by Step Approach

Model validation is the process of evaluating a trained model on test data set. This provides the generalization ability of a trained model. Here I provide a step by step approach to complete first iteration of model validation in minutes. Shreyas Jothish Feb 7, 2019·5 min read

The basic recipe for applying a supervised machine learning model are: Choose a class of model Choose model hyper parameters Fit the model to the training data Use the model to predict labels for new data From Python Data Science Handbook by Jake VanderPlas

Jake VanderPlas, gives the process of model validation in four simple and clear steps. There is also a whole process needed before we even get to his first step. Like fetching all the information we need from the data to make a good judgement for choosing a class model. Also providing finishing touches to confirm the results after. I will get into depth about these steps and break it down further.

Data cleansing and wrangling.

Split the data into training and test data sets.

Define the metrics for which model is getting optimized.

Get quick initial metrics estimate.

Feature engineering to optimize the metrics. (Skip this during first pass).

Data pre-processing.

Feature selection.

Model selection.

Model validation.

Interpret the results.

Get the best model and check it against test data set.

I will be using data set from UCI Machine Learning Repository. Data set is from the Blood Transfusion Service Center in Hsin-Chu City in Taiwan. This is a classification problem. The idea behind this extends to regression problems as well.

Data cleansing and wrangling.

Blood Transfusion Service Center Data Set is a clean data set. This will not be the case for most other data sets. So this is the step to inspect and clean up the data for example handling missing values…

Split the data into training and test data sets.

There are many ways to get the training and test data sets for model validation like:

3-way holdout method of getting training, validation and test data sets.

k-fold cross-validation with independent test data set.

Leave-one-out cross-validation with independent test data set.

Sebastian Raschka’s blog has a good reference for determining this.

I will make use of 5-fold cross-validation with independent test data set. So split the data into training and test data sets.

Define the metrics for which model is getting optimized.

The metrics used in classification and regression problems vary. Classification problem uses metrics like accuracy, precision, recall... Regression problem uses metrics like mean absolute error, R2 score...

Classification metrics in scikit-learn

Regression metrics in scikit-learn

Since this is a classification problem, I will use accuracy score here.

Get quick initial metrics estimate.

The main idea behind this step is to get the baseline estimate of metrics which is being optimized. This baseline will work as reference in further steps of model validation. There are several ways to get the baseline estimate for classification problem. I am using the majority class for prediction. The baseline accuracy score is approximately 77%.

Feature engineering to optimize the metrics.

Feature engineering is the process of using domain knowledge of the data to create features that make machine learning algorithms work. Feature engineering is fundamental to the application of machine learning, and is both difficult and expensive. From Wikipedia

This means identifying the relationships between independent and dependent features. This is with the help of graphs like pair plots or correlation matrix. Then the identified relationships we can add as polynomial or interaction features.

Feature engineering step is the point of entry for successive iterations. This is a critical step and plays a greater role in predictions as compared to model validation.

As a quick solution we can throw in some polynomial features using PolynomialFeatures in sci-kit learn.

Domain knowledge on the problem in hand will be of great use for feature engineering. This is a bigger topic in itself and requires extensive investment of time and resource.

Data pre-processing.

Data pre-processing converts features into format that is more suitable for the estimators. In general, machine learning model prefer standardization of the data set. I will make use of RobustScaler for our example.

Refer to sci-kit learn’s Preprocessing data section for detailed information.

Feature selection.

Feature selection or dimensionality reduction on data sets helps to

Either to improve models’ accuracy scores

or

To boost their performance on very high-dimensional data sets.

I will use SelectKBest, univariate feature selection method. The scoring function used for classification and regression problems will vary.

Refer to sci-kit learn’s Feature selection section for detailed information.

Model selection.

Since we are working with classification problem, I will be using LogisticRegression model. For regression problems we can use LinearRegression or Ridge models. The selection of model is dependent on the problem in hand. Based on the complexity of data we can use either simple linear model or advanced models.

Refer to sci-kit learn’s Generalized Linear Models section for detailed information.

There is a elegant way to combine above three steps using sci-kit learn’s Pipeline. Pipeline applies a list of transformations before the final model.

Model validation.

Hyper-parameters are parameters that are not learnt within model by itself. Hyper-parameters are passed as arguments to the constructor of the steps in pipeline. Based on the cross validation score it is possible to fetch the best possible parameters.

For hyper-parameter tuning GridSearchCV is one of the options. It performs exhaustive search over specified parameter values for the model. Here we are passing the hyper-parameters to steps in pipeline using param_grid. cv is set to 5 since we are to perform 5-fold cross-validation. scoring is set to accuracy since we want to predict accuracy of the model.

Refer to sci-kit learn’s Tuning the hyper-parameters of an estimator section for detailed information.

Interpret the results.

GridSearchCV returns few important information like:

best_estimator_ , estimator which gave highest score. Using this we can get the information about the selected features.

, estimator which gave highest score. Using this we can get the information about the selected features. best_score_, Mean cross-validated score of the best_estimator. For our problem set we got a best score of 78% for our very first iteration. This seems not much but we have already beaten our baseline accuracy score.

Mean cross-validated score of the best_estimator. For our problem set we got a best score of for our very first iteration. This seems not much but we have already beaten our baseline accuracy score. best_params_ , parameter setting that gave the best results on the hold out data. For the hyper-parameters that passed to the model, below setting gave the best results.

‘logisticregression__C’: 1.0 ‘logisticregression__class_weight’: None ‘selectkbest__k’: 4

This ends the first iteration of model validation. Now we can go back and repeat the process starting from feature engineering step. At the end of each iteration we can check the change in the accuracy score. Repeat the procedure till reaching the desired accuracy score or chosen metrics.

Get the best model and check it against test data set.

After choosing the final model, check its performance using the test data set. Here we get a accuracy score of 75% on the test data set.

Note this blog is to provide a quick introduction on supervised machine learning model validation. Idea here is to not get best metrics score in the very first iteration. Use this approach to set baseline metrics score. Improve our model validation score each iteration.

Get complete notebook here.

You might find my other blogs interesting. Do check them out.

Analyse retweet ratio to determine social influence.

Why are we letting Trump’s tweet affect our mood?