Hyperparameter Tuning — Always Tune your Models | by Zachary Warnes | Towards Data Science

Grid Search — Exhaustive and Computationally expensive, use when hyperparameter search space is restricted.
Random Search — Larger Search Spaces improved models compared to Grid Search, the go-to method built into scikit-learn.
Bayesian Optimization — Improved performance over random search & hyperparameter interpretation, use when possible.
Hyperparameter tuning is a vital aspect of increasing model performance. Given a complex model with many hyperparameters, effective hyperparameter tuning may drastically improve performance. For example, a gradient boosting classifier has many different parameters to fine-tune, each uniquely changing the model’s performance. Some may have little or no effect, while others could be critical to the model’s viability.
The most common approach is to try several user-defined hyper-parameter combinations, compare results, and select the overall best performer. This approach has some clear downsides as the selection is biased based on the initial options. While prior knowledge about the target problem may provide some direction to the ideal choice, there are potentially more optimal selections that go unchecked. Grid search follows this basic approach, but this is shown to be non-ideal even with prior knowledge.
The complete notebook for this post can be found here.
Each of these hyperparameter searches is incredibly easy to implement, so you should incorporate one into your model optimization process. For my example, I am using the breast cancer dataset. First, I load the data, relevant libraries, initialize some meta-parameters, and split the dataset. Next, I create a small pipeline, run each of the searches and print out the performance.
import pandas as pdfrom scipy.stats import uniform, geom, loguniform, randint, exponfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.model_selection import cross_val_score, GridSearchCV, train_test_split, RandomizedSearchCVfrom sklearn.metrics import roc_auc_scorefrom sklearn.pipeline import make_pipeline, Pipelinefrom sklearn.datasets import load_iris, load_boston, load_diabetes, load_digits, load_linnerud, load_wine, load_breast_cancerTEST_SIZE = 0.1 RANDOM_STATE = 10  data = load_breast_cancer() df = pd.DataFrame(data.data, columns=data.feature_names) df['target'] = data.target  X = df.drop(['target'], axis=1) y = df['target'].astype(float) X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=TEST_SIZE, random_state=RANDOM_STATE, stratify=y)
Grid search is one of the most common hyper-parameter selection techniques. This approach is effectively a brute force strategy, simply creating and testing a model for each hyper-parameter configuration — the approach benefits from the exhaustive search behavior. In addition, there is a certainty that each combination has been compared.
However, it should be clear that using a comprehensive search for hyper-parameters is expensive. Since these searches handle combinatorial complexity, the number of models that have to be tested increases dramatically, especially when the target model type has many parameters to tune. Because of this computational burden, Grid search is most effective when the optimal selection of some hyper-parameters is already known.
Random search is a common alternative to Grid Search. In contrast to an exhaustive search of each hyper-parameter configuration, random search is only run for a set number of samples. Each sample selects hyper-parameters at random, either from a list of possible parameters or the distribution of parameters. While this approach appears to be less optimal compared to grid search, it performs better given the same amount of computation.
The benefits of the method are noticeable when there are many negligible hyper-parameters. For example, in a gradient boosting model, the selection of features to be used for a model with few features may either be the square root of the total features or the log of the number of features. When the number of features is small, the difference between the two selections is limited. Adding both variants of the parameter to the search would double the search space in a grid search. In contrast, a random search would only include these parameter choices proportionally across all samples. The samples that include the variants of the negligible parameter would reveal the effect of this parameter and drastically save time in computation.
Consequently, a random search can optimize over a larger hyper-parameter space with the same computation as a grid search. Thus a random search further improves the optimal model performance compared to a model optimized through grid search.
Moreover, random search has another benefit over grid search. Because grid search creates models using discrete selections of hyper-parameters, the probability of missing an optimal selection of a non-negligible hyper-parameter is increased. In contrast, a random search can select configurations from either a continuous or the complete discrete distribution of possible hyper-parameters. This behavior has been shown to much more effective in identifying optimal model settings, especially when the search space for the hyper-parameters is significant.
The comparison between approaches is shown in the chart below. Assuming two hyper-parameters require tuning and the model’s true performance at each configuration is shown in blue for each parameter. It should be clear that the models produced from a grid search can ‘miss’ the optimal hyperparameter configuration for the model. In contrast, random search effectively ignores the first negligible hyperparameter and samples more varied values from the second hyperparameter. Thus, a random search can find a slightly more optimal choice of hyperparameters.
Grid search and random search are part of the standard scikit-learn package and work seamlessly with existing models produced from scikit-learn. Here is the rough setup for creating a pipeline that uses grid search and then random search. The model for this example is a GradientBoostingClassifier, but the model is interchangeable.
To switch to another model, update the model in the pipeline definition and update the parameter names in the hyper_parameter dictionary and the choices/distributions. The names of the parameters should have the prefix ‘model__’ to match the name ‘model’ in the pipeline. These names can be changed as needed.
N_JOBS = 1K_FOLDS = 10N_ITER_NO_CHANGE = 100SCORING_METRIC = 'roc_auc'# Gradient Boosting Parametershyper_parameters = {        'model__n_estimators': [50, 150, 300],        'model__learning_rate': [0.1, 0.03, 0.001],        'model__max_depth': [3, 5, 8],        'model__max_features': ['sqrt'],         'model__min_samples_split': [2],        'model__n_iter_no_change': [N_ITER_NO_CHANGE], # Note single elements should still be in a list        'model__random_state': [RANDOM_STATE],        'model__min_samples_leaf': [2, 10]}# Make pipeline - Add steps if needed, i.e. feature selectionpipe = Pipeline(steps=[    ('model', GradientBoostingClassifier())])# 'n_iter' no longer needed as all combinations are checkedsearch_space = GridSearchCV(    pipe, hyper_parameters,     cv=K_FOLDS, scoring=SCORING_METRIC, n_jobs = N_JOBS,     return_train_score=True, verbose = 1)search_space.fit(X_train, y_train) y_pred = search_space.best_estimator_.predict(X_train)y_pred_prob = search_space.best_estimator_.predict_proba(X_train)[:,1]y_pred_test = search_space.best_estimator_.predict_proba(X_test)[:,1]print(     'Best Training Score: ', search_space.cv_results_['mean_train_score'][search_space.best_index_],     '\nBest Test Score: ', search_space.best_score_,    '\nHold Out Test Score: ', roc_auc_score(y_test, y_pred_test))
And then the code for a random search, note the change from list options for parameters to distributions defined from the stats library.
N_ITER = 54N_JOBS = 1K_FOLDS = 10N_ITER_NO_CHANGE = 100SCORING_METRIC = 'roc_auc'hyper_parameters = {        'model__n_estimators': randint(50,500),        'model__learning_rate': loguniform(3e-4, 3e-1),        'model__max_depth': randint(3,10),        'model__max_features': ['sqrt'],         'model__min_samples_split': randint(2,20),        'model__n_iter_no_change': [N_ITER_NO_CHANGE], # Note single elements should still be in a list        'model__random_state': [RANDOM_STATE],        'model__min_samples_leaf': randint(1,10)}pipe = Pipeline(steps=[    ('model', GradientBoostingClassifier())])search_space = RandomizedSearchCV(    pipe, hyper_parameters,     n_iter = N_ITER, cv=K_FOLDS,     scoring=SCORING_METRIC, n_jobs = N_JOBS,     return_train_score=True, verbose = 1)search_space.fit(X_train, y_train) y_pred = search_space.best_estimator_.predict(X_train)y_pred_prob = search_space.best_estimator_.predict_proba(X_train)[:,1]y_pred_test = search_space.best_estimator_.predict_proba(X_test)[:,1]print(     'Best Training Score: ', search_space.cv_results_['mean_train_score'][search_space.best_index_],     '\nBest Test Score: ', search_space.best_score_,    '\nHold Out Test Score: ', roc_auc_score(y_test, y_pred_test))
Random Search proves to be more effective against Grid search. However, another approach proves to find even more optimal hyperparameter selections. Bayesian search is a hyper-parameter selection technique that models and optimizes the hyper-parameter search space. This modeling is done using Gaussian processes, which model the function produced from the hyperparameter space.
The search process is as follows. Samples are taken from the hyper-parameter space, creating multiple models, measure the model’s performance, and optimize using an argmax function to determine an optimal hyper-parameter configuration. This process is similar to a Random Search with the added benefit that patterns are extrapolated to find a more optimal model. In addition, Bayesian hyper-parameter search models each of the hyper-parameters compared to one another, which allows for a more robust interpretation of the model’s effectiveness.
Comparing Bayesian hyperparameter to grid search and random search, you can see that not only does Bayesian search sample randomly similar to random search but then, after many samples, generates a function to model the hyperparameter space, which is then optimized. This process results in the dotted line in the image with the optimal point indicated by the small red star. Note how this is still not the true optimal hyperparameter selection for the function but improves model performance.
The code for Bayesian optimization is unfortunately not part of the standard scikit-learn package. However, the package scikit-optimize provides a Bayesian hyperparameter optimization function that integrates with scikit-learn models. The code for this method follows a similar pattern to the grid and random searches, but the parameter distributions are defined differently.
from skopt import BayesSearchCVfrom skopt.space import Real, Categorical, IntegerN_ITER = 54N_JOBS = 1hyper_parameters = {        'n_estimators': Integer(50, 300),        'max_depth': Integer(3, 10),        'min_samples_split': Integer(2, 300),        'learning_rate': Real(3e-6, 3e-1,prior='log-uniform'),             'max_features': Categorical(['sqrt', None]),}search_space = BayesSearchCV(    estimator=GradientBoostingClassifier(        n_iter_no_change=N_ITER_NO_CHANGE,         random_state=RANDOM_STATE    ),    search_spaces=hyper_parameters,    scoring=SCORING_METRIC,    cv=K_FOLDS,    n_iter=N_ITER,    random_state=RANDOM_STATE,    return_train_score=True,    verbose = 1,    n_jobs=N_JOBS)# executes bayesian optimization_ = search_space.fit(X_train, y_train)y_pred = search_space.best_estimator_.predict(X_train)y_pred_prob = search_space.best_estimator_.predict_proba(X_train)[:,1]y_pred_test = search_space.best_estimator_.predict_proba(X_test)[:,1]# model can be saved, used for predictions or scoringprint(     'Best Training Score: ', search_space.cv_results_['mean_train_score'][search_space.best_index_],     '\nBest Test Score: ', search_space.best_score_,    '\nHold Out Test Score: ', roc_auc_score(y_test, y_pred_test))
Moreover, scikit-optimization provides methods for visualizing the functions generated for hyperparameter optimization. These plots can further guide the optimization process. Moreover, emergent patterns can be viewed, such as identifying negligible parameters and limited search spaces. These plots serve to refine the search space further to create an even more optimal model.
import matplotlib.pyplot as pltfrom skopt.plots import plot_objective, plot_histogram_ = plot_objective(search_space.optimizer_results_[0],                   n_minimum_search=int(1e8))plt.show()
You should be using hyperparameter optimization if you are not already and if your models have hyperparameters. Then, depending on your available resources and package dependencies, you can use any of the three methods described.
For example, Bayesian optimization is ideal for rapidly finding good hyperparameter settings, while grid search can further fine-tune a model when a small search space is defined. In general, random search is a happy medium between the two methods, and you should be (at minimum) incorporating this hyperparameter search into your existing models and pipelines for immediate performance gains.
If you’re interested in reading articles about novel data science tools and understanding machine learning algorithms, consider following me on Medium.
If you’re interested in my writing and want to support me directly, please subscribe through the following link. This link ensures that I will receive a portion of your membership fees.