Supervised Machine Learning: Feature Engineering and Hyper Parameter Tuning
By: Shreyas Jothish
Tags: Supervised Learning, Machine Learning, Feature Engineering, Hyperparameter Tuning

We can either drop columns with high cardinality and NaN values if needed. Also for numerical features, we can fill the NaN values with its mean, median or mode instead of dropping it. In the same way for categorical features, we can categorize the NaN as a separate category.

Also based on the water table data set, we can engineer few features like:

NaN values in longitude and latitude updated with its mean.

Binary feature to identify if funder has at least funded 5 pumps. In the same was identify if installer has installed at least 5 pumps.

Group categorical features data with high cardinality based on prefix. Better way is to analyse the data set in detail and try to group them based on some logical construct.

Update construction_year based on median value of other pumps in the same ward.

Split the date_recorded into year_recorded and month_recorded. Even group in different bins.

Compute the age of pump based on construction_year and year_recorded.

Hyper-parameter tuning

Hyper-parameter optimization or tuning is the problem of choosing a set of optimal hyper-parameters for a learning algorithm. The same kind of machine learning model can require different constraints, weights or learning rates to generalize different data patterns. These measures are called hyper-parameters, and have to be tuned so that the model can optimally solve the machine learning problem. Hyper-parameter optimization finds a tuple of hyper-parameters that yields an optimal model which minimizes a predefined loss function on given independent data. The objective function takes a tuple of hyper-parameters and returns the associated loss. Cross-validation is often used to estimate this generalization performance.

Source: Wikipedia

In short, Hyper-parameter tuning means finding the best value to parameters. As part of this we chose a range of values to tuned parameter. Then based on cross validation result determine which value is more apt. Then move onto next set of model parameters. Hyper-parameter tuning is a delicate and tedious process of comparison and elimination.

Here we use XGBClassifier for classification. GridSearchCV for cross validation with independent test data set. For brevity, let us skip the details on pipeline and param_grid creation. Refer to previous blog for details. Here let me illustrate tuning of parameters max_depth and min_child_weight of XGBClassifier.

max_depth (int) — Maximum tree depth for base learners.

Used to control over-fitting as higher depth will allow model to learn relations very specific to a particular sample.

Typical values: 3–9

min_child_weight (int) — Minimum sum of instance weight(hessian) needed in a child.

Used to control over-fitting. Higher values prevent a model from learning relations which might be highly specific to the particular sample selected for a tree.

Too high values can lead to under-fitting hence, it should be tuned using CV. Typical values: 1–5

Output from first iteration of parameter tuning:

Cross Validation Score: 0.7924354657687991

0.7924354657687991 Best Parameters: {‘xgbclassifier__max_depth’: 5, ‘xgbclassifier__min_child_weight’: 1}

During first iteration we incremented the parameter value in steps of two. So in the second iteration let us narrow range of parameters further and check.

During second iteration best value for max_depth was 5. Now it is 6. So in the third iteration let us reconfirm the value of max_depth. Now let us keep min_child_weight parameter fixed to 1.

Output from third iteration of parameter tuning:

Cross Validation Score: 0.793625140291807

0.793625140291807 Best Parameters: {‘xgbclassifier__max_depth’: 6, ‘xgbclassifier__min_child_weight’: 1}

From third iteration we can confirm the values of max_depth as 6. In the same way, we can pickup other parameters and tune for their optimized values.

Note this blog is to provide a quick introduction on feature engineering and hyper-parameter tuning. Idea here is to understand the process so the code is not optimized. Use this approach to set baseline metrics score. Improve our model validation score each iteration.

Get complete notebook here.

You might find my other blogs interesting. Do check them out.

Analyse retweet ratio to determine social influence.

Why are we letting Trump’s tweet affect our mood?

References: