A Practical Introduction to Grid Search, Random Search, and Bayes Search | by B. Chen | Towards Data Science

In Machine Learning, hyperparameters refer to the parameters that cannot be learned from data and need to be provided before training. The performance of machine learning models relies heavily on finding the optimal set of hyperparameters.
Hyperparameter tuning basically refers to tweaking the hyperparameters of the model, which is basically a length process. In this article, you’ll learn the 3 most popular hyperparameter tuning techniques: Grid Search, Random Search, and Bayes Search. This article is structured as follows:
Getting and preparing dataGrid SearchRandom SearchBayes SearchConclusion
Getting and preparing data
Grid Search
Random Search
Bayes Search
Conclusion
Please check out the Notebook for source code. More tutorials are available from Github Repo.
For demonstration, we’ll be using the built-in breast cancer data from Scikit Learn to train a Support Vector Classifier (SVC). We can get the data with the load_breast_cancer function:
from sklearn.datasets import load_breast_cancercancer = load_breast_cancer()
Next, let’s create df_X and df_y for features and target label as follows:
# Featuresdf_X = pd.DataFrame(cancer['data'], columns=cancer['feature_names'])# Target labeldf_y = pd.DataFrame(cancer['target'], columns=['Cancer'])
P.S. If you want to know more about the dataset, you can run print(cancer['DESCR']) to print out summary and feature information.
After that, let’s split the dataset into a training set (70%) and a test set (30%) using training_test_split():
# Train test splitfrom sklearn.model_selection import train_test_splitimport numpy as npX_train, X_test, y_train, y_test = train_test_split(df_X, np.ravel(df_y), test_size=0.3)
We will be training a Support Vector Classifier (SVC) model. The regularization parameter C and kernel coefficient gamma are the two most important hyperparameters in SVC:
The regularization parameter C determines the strength of the regularization.
The kernel coefficient gamma controls the width of the kernel. SVC uses radial basis function(RBF) kernel by default (also known as the Gaussian kernel).
We will be tuning these 2 parameters in the following tutorial.
It’s tricky to find the optimal value for C and gamma. The simplest solution is to try a bunch of combinations and see what works best. This idea of creating a “grid” of parameters and just trying out all the possible combinations is called a Grid Search.
This method is common enough that Scikit-learn has this functionality built-in with GridSearchCV. The CV stands for Cross-Validation which is another technique to evaluate and improve our Machine Learning model.
GridSearchCV takes a dictionary that describes the parameters that should be tried and a model to train. The grid of parameters is defined as a dictionary, where the keys are the parameters and the values are the settings to be tested. Let’s first define our candidate C and gamma as follows:
param_grid = {   'C': [0.1, 1, 10, 100, 1000],   'gamma': [1, 0.1, 0.01, 0.001, 0.0001]}
Next, let’s create a GridSearchCV object and fit it to the training data.
from sklearn.model_selection import GridSearchCVfrom sklearn.svm import SVCgrid = GridSearchCV(SVC(), param_grid, refit=True, verbose=3)grid.fit(X_train,y_train)
Once the training is completed, we can inspect the best parameters found by GridSearchCV in the best_params_ attribute, and the best estimator in the best_estimator_ attribute:
# Find the best paramters>>> grid.best_params_{'C': 1, 'gamma': 0.0001}# Find the best estimator>>> grid.best_estimator_SVC(C=1, gamma=0.0001)
Now take that grid model and create some predictions using the test set and create classification reports and confusion matrices for them.
Grid Search tries all combinations of hyperparameters hence increasing the time complexity of the computation and could result in an unfeasible computing cost. Providing a cheaper alternative, Random Search tests only as many tuples as you choose. The selection of the hyperparameter values is completely random.
This method is also common enough that Scikit-learn has this functionality built-in with RandomizedSearchCV. The function API is very similar to GridSearchCV.
First, let’s specify parameters C & gamma and distributions to sample from as follows:
import scipy.stats as statsfrom sklearn.utils.fixes import loguniform# Specify parameters and distributions to sample fromparam_dist = {  'C': stats.uniform(0.1, 1e4),  'gamma': loguniform(1e-6, 1e+1),}
Next, let’s create a RandomizedSearchCV object with argument n_iter_search and fit it to the training data.
n_iter_search = 20random_search = RandomizedSearchCV(    SVC(),     param_distributions=param_dist,    n_iter=n_iter_search,     refit=True,    verbose=3)random_search.fit(X_train, y_train)
Similarly, once the training is completed, we can inspect the best parameters found by RandomizedSearchCV in the best_params_ attribute, and the best estimator in the best_estimator_ attribute:
>>> random_search.best_params_{'C': 559.3412579902997, 'gamma': 0.00022332416796205752}>>> random_search.best_estimator_SVC(C=559.3412579902997, gamma=0.00022332416796205752)
Finally, we take that random search model and create some predictions using the test set and create classification reports and confusion matrices for them.
Bayes Search uses the Bayesian optimization technique to model the search space to arrive at optimized parameter values as soon as possible. It uses the structure of search space to optimize the search time. Bayes Search approach uses the past evaluation results to sample new candidates that are most likely to give better results (shown in the figure below).
Scikit-Optimize library comes with BayesSearchCV implementation.
First, let’s specify parameters C & gamma and distributions to sample from as follows:
from skopt import BayesSearchCV# parameter ranges are specified by one of belowfrom skopt.space import Real, Categorical, Integersearch_spaces = {  'C': Real(0.1, 1e+4),  'gamma': Real(1e-6, 1e+1, 'log-uniform'),}
Next, let’s create a BayesSearchCV object with argument n_iter_search and fit it to the training data.
n_iter_search = 20bayes_search = BayesSearchCV(    SVC(),     search_spaces,     n_iter=n_iter_search,     cv=5,    verbose=3)bayes_search.fit(X_train, y_train)
Similarly, once the training is completed, we can inspect the best parameters found by BayesSearchCV in the best_params_ attribute, and the best estimator in the best_estimator_ attribute:
>>> bayes_search.best_params_OrderedDict([('C', 0.25624177419852506), ('gamma', 0.00016576008531229226)])>>> bayes_search.best_estimator_SVC(C=0.25624177419852506, gamma=0.00016576008531229226)
Finally, we take that Bayes search model and create some predictions using the test set and create classification reports and confusion matrices for them.
In this article, we have covered the 3 most popular hyperparameter optimization techniques that are used to get the optimal set of hyperparameters leading to training a robust machine learning model.
In general, if the number of combinations is limited enough, we can use the Grid Search technique. But when the number of combinations increases, we should try Random Search or Bayes Search as they are not computationally expensive.
I hope this article will help you to save time in learning Machine Learning. I recommend you to check out their APIs [1, 2] and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
References:
[1] Scikit-Learn docs: https://scikit-learn.org/stable/auto_examples/model_selection/plot_randomized_search.html
[2] Scikit-Optimize docs: https://scikit-optimize.github.io/stable/modules/generated/skopt.BayesSearchCV.html