mlmachine - Hyperparameter Tuning with Bayesian Optimization
By: Tyler Peterson
Tags: Mlmachine, Machine Learning, Data Science, Python

Feature Importance Summary

As a second preparatory step, we want to perform feature selection for each of our classifiers:

Exhaustively Iterative Feature Selection

For our final preparatory step, we use this feature selection summary to perform iterative cross-validation on smaller and smaller subsets of features for each of our estimators:

From this result, we extract our dictionary of optimum feature sets for each estimator:

The keys are estimator names, and the associated values are lists containing the column names of the best performing feature subset for each estimator. Here are the key/value pairs for XGBClassifier() , which used only 10 of the available 43 features to achieve the best average cross-validation accuracy on the validation dataset:

With our processed dataset and optimum feature subsets in hand, it’s time to use Bayesian optimization to tune the hyperparameters of our 4 estimators.

Outline Our Feature Space

First, we need to establish our feature space for each parameter for each estimator:

The outermost keys of the dictionary are names of classifiers, represented by strings. The associated values are also dictionaries, where the keys are parameter names, represented as strings, and the values are hyperopt sampling distributions from which parameter values will be chosen.

Run the Bayesian Optimization Job

Now we’re ready to run our Bayesian optimization hyperparameter tuning job. We will use a built-in method belonging to our Machine() object called exec_bayes_optim_search() . Let’s see mlmachine in action:

Let’s review the parameters:

estimator_parameter_space : The dictionary-based feature space we setup above.

: The dictionary-based feature space we setup above. data : Our observations.

: Our observations. target : Our target data.

: Our target data. columns : An optional parameter that allows us to subset the input dataset features. Accepts a list of feature names, which will apply equally to all estimators. Also accepts a dictionary, where the keys represent estimator class names and values are lists of feature names to be used with the associated estimator. In this example, we use the latter by passing in the dictionary returned by cross_val_feature_dict() in the FeatureSelector() workflow above.

: An optional parameter that allows us to subset the input dataset features. Accepts a list of feature names, which will apply equally to all estimators. Also accepts a dictionary, where the keys represent estimator class names and values are lists of feature names to be used with the associated estimator. In this example, we use the latter by passing in the dictionary returned by in the workflow above. scoring : The scoring metric to be evaluated.

: The scoring metric to be evaluated. n_folds : Number of folds to use in cross-validation procedure.

: Number of folds to use in cross-validation procedure. iters : Total number of iterations to run the hyperparameter tuning process. In this example, we run the experiment for 200 iterations.

: Total number of iterations to run the hyperparameter tuning process. In this example, we run the experiment for 200 iterations. show_progressbar : Controls whether progress bar displays and actively updates during the course of the process.

Anyone familiar with hyperopt will be wondering where the objective function is. mlmachine abstracts away this complexity.