Hyperparameter Tuning - Brief Theory and What you won’t find in the HandBook
By: Thiago Nunes
Tags: Machine Learning, Hyperparameter Tuning, Statistics

Parameters and Hyperparameters:

Parameters:

Probabilistic models are estimated by unknown quantities, called parameters. These are adjusted using an optimization technique so that in the training sample it is possible to find a pattern in the best possible way. In a simple way, parameters are estimated by the algorithm and the user has little / nothing control over them.

In a simple linear regression, the model parameters are betas (ẞ).

BAM!!! : In statistics jargon, parameters are defined as population characteristics. So strictly speaking, when we want to talk about sample characteristics, we use the term estimators. This makes little difference in this context, but it is important to note.

Hyperparameters:

Hyperparameters are sets of information that are used to control way of learn an algorithm. Their definitions impact parameters of the models, seen as a way of learning, change from the new hyperparameters. This set of values affects performance, stability and interpretation of a model.

Each algorithm requires a specific hyperparameters grid that can be adjusted according to the business problem.

Hyperparameters alter the way of a model learn trigger this training algorithm after parameters to generate outputs.

In a decision tree, one of the main hyperparameters is the depth of the tree and the number of samples in each leaf.

ALERT + BAM!!! : Hyperparameters should not be chosen in only Training + Test data cross validation strategy. We must be very careful about this.

DataSet Strategy:

When the objective is tuning and test hyperparameters configuration the data arrangement must be designed like:

Training: Set of data to train the algorithm with grid of hyperparameters; Validation: Dataset where performance metrics will be applied to validate hyperparameters; Test: Last set to effectively validate the performance of your model.

Hyperparameters Choice:

For choice let’s talk about 4 techniques :

Business Knowledge; Grid Search; Random Search; Bayesian Optimization.

Business Knowledge:

Hyperparameters choice with business knowledge is by far the most complex. It requires that you be able to abstract your model and fit assumptions and behaviors expected from previous studies or business knowledge for a more appropriate choice. At this stage, it is also very common to rely on scientific articles and publications that support the choice of sets of hyperparameters for greater success in solving the problem.

Obs!!! : In my point of view it’s the best way to start an optimization algorithm. However, it is more complex and requires a lot of experience and reading from the data scientist.

Grid Search:

Grid Search combines a selection of hyperparameters established by the scientist and runs through all of them to evaluate the model’s performance. Its advantage is that it is a simple technique that will go through all the programmed combinations. The biggest disadvantage is that it traverses a specific region of the parameter space and cannot understand which movement or which region of the space is important to optimize the model.

Random Search:

In Random Search, the hyperparameters are chosen at random within a range of values that it can assume. The advantage of this method is that there is a greater chance of finding regions of the cost minimization space with more suitable hyperparameters, since the choice for each iteration is random. The disadvantage of this method is that the combination of hyperparameters is beyond the scientist’s control.

Grid vs Random Search:

Bayesian Optimization:

Bayesian Hyperparameter optimization is a very efficient and interesting way to find good hyperparameters. In this approach, in naive interpretation way is to use a support model to find the best hyperparameters.

A hyperparameter optimization process based on a probabilistic model, often Gaussian Process, will be used to find data from data observed in the later distribution of the performance of the given models or set of tested hyperparameters.

As it is a Bayesian process at each iteration, the distribution of the model’s performance in relation to the hyperparameters used is evaluated and a new probability distribution is generated. With this distribution it is possible to make a more appropriate choice of the set of values that we will use so that our algorithm learns in the best possible way.

Summing Up!!! : It is a model to optimize the learning parameters.For a more technical explanation of the subject I recommend the following article:

Bayes Optim tecnique