Comparison of Hyperparameter Tuning algorithms: Grid search, Random search, Bayesian optimization
By: Lavanya Gupta
Tags: Crossvalidation, Hyperparameter Tuning, Randomizedsearchcv, Bayesian Optimization, Gridsearchcv

Hyperparameter Tuning Algorithms

1. Grid Search

This is the most basic hyperparameter tuning method. You define a grid of hyperparameter values. The tuning algorithm exhaustively searches this space in a sequential manner and trains a model for every possible combination of hyperparameter values.

For example, to train a SVM model we can define our hyperparameter grid space (C, gamma, kernel) as follows. The grid search algorithm trains multiple models (one for each combination) and finally retains the best combination of hyperparameter values.

{

'C': [0.1, 1, 10, 100, 1000],

'gamma': [0.1, 0.01 ,0.001, 0.0001],

'kernel': ['rbf', 'linear']

}

Grid search is not very often used in practice because the number of models to train grows exponentially as you increase the number of hyperparameters to train. This can be very inefficient — both in computing power and time.

Source — SigOpt

2. Random Search

Random search differs from grid search in that we no longer provide an explicit set of possible values for each hyperparameter; rather, we provide a statistical distribution for each hyperparameter from which values are sampled. Essentially, we define a sampling distribution for each hyperparameter to carry out a randomized search.

Using random search, we can also control or limit the number of hyperparameter combinations used. Unlike grid search, in which every possible combination is evaluated; in random search, we can specify to train only a fixed number of models and terminate the tuning algorithm post that. The number of search iterations can also be set based on time or resources.

For example, to train a SVM model we can define our hyperparameters (C, gamma) as log-uniform distributions. (Some other common distributions are exponential , normal , lognormal , uniform and random ). The random search algorithm samples a value for C and gamma from their respective distributions, and uses it to train a model. This process is repeated several times and multiple models are trained. The best combination of hyperparameter values are finally retained.

{

'C': loguniform(1e-1, 1e3),

'gamma': loguniform(1e-4, 1e-1),

'kernel': ['rbf', 'linear']

}

Source — SigOpt

3. Bayesian Optimization

In the previous two methods, we performed individual experiments by building multiple models with various hyperparameter values. All these experiments were independent of each other. Since each experiment was performed independently, we are not able to use the information from one experiment to improve the next experiment.

Bayesian optimization is a sequential model-based optimization (SMBO) algorithm that uses the results from the previous iteration to decide the next hyperparameter value candidates. So instead of blindly searching the hyperparameter space (like in grid search and random search), this method advocates the usage of intelligence to pick the next set of hyperparameters which will improve the model performance. We iteratively repeat this process until we converge to an optimum.

An interesting analogy is to compare this to Bagging Vs Boosting. If you think about it, the idea is very similar!

In bagging, we build a lot of trees in parallel and independent to each other. Boosting, on the other hand, is a sequential process where with each additional tree, we learn to correct the mistakes from its predecessor tree.

Bayesian optimization creates a probabilistic model, mapping hyperparameters to a probability of a score on the objective function. For more mathematical details, refer this.

Source — SigOpt

Summary

Bayesian optimization methods are efficient because they select hyperparameters in an informed manner. By prioritizing hyperparameters that appear more promising from past results, Bayesian methods can find the best hyperparameters in lesser time (in fewer iterations) than both grid search and random search.