Why Hyper parameter tuning is important ?
By: Niwratti Kasture
Tags: Artificial Intelligence, Machine Learning, Deep Learning, Google, AWS

It is rare that a model will perform at the level you need for production just in the first instance. To find the right solution for your business problem, often you have to go through an iterative cycle. There are multiple pieces that come together to solve the intended machine learning puzzle. You may need to train and evaluate multiple models that include different data setup and algorithms, perform feature engineering a few times or even augment more data. This cycle also involves tweaking your model’s hyperparameters.

Hyperparameters are the knobs or settings that can be tuned before running a training job to control the behavior of an ML algorithm. They can have a big impact on model training as it relates to training time, infrastructure resource requirements (and as a result cost), model convergence and model accuracy.

Model parameters are learnt as part of training process, whereas the values of hyperparameters are set before running the training job and they do not change during the training.

Different types of Hyperparameters

Hyperparameters can be broadly divided into 3 categories —

Model hyperparameters — defines the fundamental construct of a model itself for ex. attributes of a neural network architecture like filter size, pooling, stride, padding.

defines the fundamental construct of a model itself for ex. attributes of a neural network architecture like filter size, pooling, stride, padding. Optimizer hyperparameters — are related to how the model learn the patterns based on data. These types of hyperparameters include optimizers like gradient descent and stochastic gradient descent (SGD), Adam, RMSprop, Adadelta and so on. More details are available here on Keras Optimizer page.

are related to how the model learn the patterns based on data. These types of hyperparameters include optimizers like gradient descent and stochastic gradient descent (SGD), Adam, RMSprop, Adadelta and so on. More details are available here on Keras Optimizer page. Data hyperparameters — are related to the attributes of the data, often used when you don’t have enough data or enough variation in data. In such cases data augmentation techniques like cropping, resizing, binarization etc.. are involved.

How Hyperparameters are selected ?

Each learner is unique.

This human quality also reflects in the machine learning process. There aren’t a defined set of rules that are applied for hyper parameter selection. Generally, hyperparameter selection and tuning is done manually. One of ways involves seeking help from someone who has domain experience and can guide to select different hyperparameters with appropriate values. Post which, model training is carried out and checked against the validation data.

Typically, these 2 approaches are commonly followed —

Grid search — set up a grid made up of hyperparameters and their different values. For each possible combination, a model is trained and a score is produced on the validation data. With this approach, every single combination of the given possible hyperparameter values is tried. While the approach performs a extensive sweep on all the possible combinations, it can be very inefficient in terms of training time and cost.

Random search — similar to grid search, but instead of training and scoring on each possible hyperparameter combination, random combinations are selected. You can set the number of search iterations based on time and resource constraints.

These 2 approaches are still primitive and requires trial and error method until satisfactory results are achieved. Due to laborious nature of the manual methods, it is preferable in recent times to use automated hyperparameter tuning solutions available from various cloud service provides such as AWS, Google and Microsoft.

These Auto tuning solutions uses methods such as gradient descent, bayesian optimization, exploration-exploitation trade-off to conduct a guided search for the best hyperparameter settings. for ex. Amazon SageMaker or Google AutoML automatic model tuning finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best for the desired metric.

Common Hyperparameters

Each algorithm learns with a variety of hyper parameters. However, following are the most common when it comes to machine learning.

Learning Rate (LR) — determines the step size at each iteration (epoch) while moving toward a minimum of a loss function. Since this process influences to what extent the model can shed off old information and acquire new information, it symbolically represents the speed at which a machine learning model “learns”. Generally, model learning starts at some random point (as a result of initialization) and sample different weights over many epochs . It also represents how far apart these samples are is the learning rate. It is crucial to achieve right learning rate as a larger learning rate would mean that model might overshoot the optimal solution and a smaller learning rate will increase training time to find the optimal solution. Frameworks such as Keras provide ways to automatically modulate the learning rate during training process. This is achieved through learning rate decay functions to allow the optimizer to change learning rate over the period of time.

Batch Size — This value indicates number of training samples that are used within each epoch. With smaller batch sizes it is observed that it is easy to get out of “local minima”. However, larger batch size can end up getting stuck in the wrong solution.

Hyper parameter tuning (optimization) is an essential aspect of machine learning process. A good choice of hyperparameters can really make a model succeed in meeting desired metric value or on the contrary it can lead to a unending cycle of continuous training and optimization.

The content written in this article is a collection of personal learning and experience. To share any feedback or comments, please use the clap/response feature on Medium platform. You can also reach out to me on Linkedin.