Hyperparameter Optimization or Tuning
By: Sivasai Yadav Mudugandla
Tags: Machine Learning, Artificial Intelligence, Python, Data Science, AI

4 Hyper-Parameter Tuning Techniques and Limitations

Popular hyper-parameter tuning techniques that every Data Scientist should know

Image from SigOpt

Introduction

Wikipedia states that “Hyperparameter optimization or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm”

One of the most challenging parts in ML workflow is finding the best hyperparameters for the model. Performance of ML models is directly related to Hyper-parameter’s. The more you tune the hyperparameters, the better model you get. Tuning Hyper-parameters could be tedious, complicated and is more of an art than science.

Hyper-parameters

Hyper-parameters are the parameters used to control the behaviour of the algorithm while building the model. These parameters cannot be learned from the normal training process. They need to be assigned before training of the model.

A sample list of Hyper-parameters by Dr.Mukesh Rao

Table of Content

Traditional or Manual Tuning Grid Search Random Search Bayesian Search

1. Traditional or Manual Tuning

In Traditional Tuning, we manually check with random sets of hyper-parameters by training the algorithm and select the best set of parameters that fits our objective.

Let’s see with code

Drawbacks:

No guarantee to get the best parameter combination. It’s a trial and error approach. So, it consumes more time.

2. Grid Search

Grid Search is a basic hyper-parameter tuning technique. It is similar to manual tuning where it builds a model for each permutation of all the given hyperparameter values specified in the grid, evaluates and selects the best model. Consider the above example where it has two hyperparameters k_value =[2,3,4,5,6,7,8,9,10] & algorithm = [‘auto’ , ’ball_tree’ , ’kd_tree’ , ’brute’], in this case it builds a total of 9*4 = 36 different models.

Image from SigOpt

Let’s understand how sklearn’s GridSearchCV works,

Drawback:

As it tries every combination of the hyper-parameters and chooses the best combination based on the cross-validation score, this makes GridsearchCV extremely slow.

3. Random Search

The motivation to use random search in place of grid search is that for many cases, all the hyperparameters may not be equally important. Random Search selects a random combination of parameters from the hyper-parameter space, parameters will be selected with fixed numbers of iterations given by n_iter. Random search has empirically been demonstrated to give better results than Grid Search.

Image from SigOpt

Let’s understand how sklearn’s RandomizedSearchCV works,

Drawback:

The problem with random search is it doesn’t guarantee to give the best parameter combination.

4. Bayesian Search

Bayesian optimization falls in a class of optimization algorithms called sequential model-based optimization (SMBO) algorithms. These algorithms use previous observations of the loss f , to determine the next (optimal) point to sample f for. The algorithm can roughly be outlined as follows.

Using previously evaluated points X1:n, compute a posterior expectation of what the loss f looks like. Sample the loss f at a new point Xnew, that maximizes some utility of the expectation of f. The utility specifies which regions of the domain of f are optimal to sample from.

These steps are repeated until some convergence criterion is met.

Image from SigOpt

Let’s understand this with scikit-optimize’s BayesSearchCV

Installation : pip install scikit-optimize

Another similar library to implement Bayesian Search is bayesian-optimization

Installation : pip install bayesian-optimization

Drawback:

It takes a dozen or so samples to get a good surrogate surface in 2 or 3 dimensions of search space; increasing dimensionality of the search space requires yet more samples.