Hyperparameter Tuning: A Practical Guide and Template | by Jiahao Weng | Towards Data Science

You have painstakingly performed your exploratory data analysis, data processing and finally 😎, it’s time for modeling! But you soon realized that you have quite a few hyperparameters to tune and it may take you a thousand years to brute-force search that large hyperparameter space for the optimal settings.
Life is short and so, the lazy me decided to write not just one, but two code templates to automate and quicken the hyperparameter tuning process for both machine learning and deep learning models. Click here for the links to the codes (or scroll to bottom), but do read on to find out more about hyperparameter tuning and understand what the code does.
Before we delve into the two code templates, it is necessary to first lay out the context in which we may face for hyperparameter tuning.
There are essentially two situations:
Number of trials > Number of hyperparameters; andNumber of trials < Number of hyperparameters,
Number of trials > Number of hyperparameters; and
Number of trials < Number of hyperparameters,
where each trial is the evaluation of a set of hyperparameters.
Situation 1 typically (though not always) applies to machine learning. Training time is not exorbitantly long, so we can afford to have a large number of trials to cover the hyperparameter space (i.e., explore many of the possible combinations of hyperparameters). The challenge is that if the hyperparameter space is huge, we may still take numerous trials and a long time to get to the optimal settings. Hence, if the algorithm can intelligently maneuver the hyperparameter space, that will help to quicken the optimization.
In deep learning however, training time usually takes much longer; possibly days or even weeks. We therefore cannot afford to experiment many trials and this is situation 2. In this case, we need to boost our capacity by parallel running many trials, and instead of training all trials for the full number of epochs, the algorithm should weed out the least promising trial candidates early during training to reallocate computing resources to other more promising ones.
With the above in mind, we now look at the various hyperparameter tuning algorithms and the ones chosen for our purposes.
The table below shows a high level overview of the hyperparameter tuning algorithms.
We would not be going through each and every one of the algorithms in detail but there are links below to other resources for your reference if you are interested to find out more.
Grid search methodically evaluates a model for each combination of hyperparameters specified in a grid, while random search samples each set of hyperparameters from a distribution over possible parameter values.
In the above famous diagram by James Bergstra and Yoshua Bengio in their 2012 paper, we see grid search on the left and random search on the right, both with nine trials (black dots) and two parameters. The green area above each square shows the gain in function by changing the values of the important parameter and the yellow area left of each square shows the gain attributed to the unimportant parameter. The diagram illustrated that random search is likely to more thoroughly explore the parameter space and lead to discovery of more optimal settings.
While random search is better than grid search, it is possible to do better.
Notice that in a hyperparameter space, points in the same region tend to return similar performance. Bayesian optimization makes use of this fact and samples regions of better performing points with higher likelihood. Tree of Parzen Estimators (TPE) is one of the algorithms that takes a Bayesian optimization approach.
The diagram above illustrates the way Bayesian optimization will search the hyperparameter space. Suppose the red region is where the optimal loss occurs and each of the white circles is a trial with the number representing the sequence of search. Initially, the hyperparameter search resembles a random search, but gradually the search starts to move towards the red region as the Bayesian optimization algorithm updates itself with the information provided by past trials. This is essentially the intuition why Bayesian optimization is better than random search.
In a paper also by James Bergstra et al., the TPE algorithm is shown to be able to find better hyperparameter configurations than random search.
Gray dots in the above diagram represent the lowest error among n. trials that used random search, while green dots are the lowest errors by the TPE algorithm for the corresponding n. trials. The paper thus showed in their experiments that TPE generally discovers hyperparameter configurations that return lower validation error than random search.
However, it is non-trivial to run Bayesian optimization in a parallel manner. And in deep learning where model training typically takes a long time, parallel computing is especially important and ASHA leverages on this.
To understand how ASHA works, there are two components we need to introduce. First, the successive halving algorithm (SHA) and second, the asynchronous aspect.
Successive Halving Algorithm (SHA)
Suppose there are 64 hyperparameter configurations to evaluate, SHA will strategize in the following manner.
Referring to the table above, SHA will first evaluate all 64 hyperparameter configurations for 1 epoch each. The performance of these 64 trials will be compared against each other and the top 1/reduction factor of trials will be “promoted”. In our example, we arbitrarily set reduction factor to 4, so 1/4 * 64 = 16 trials will be “promoted” to the next rung where another 3 epochs will be run. Similar comparison is performed at each of the next rungs until only 1 trial is left.
The key insight from SHA is that we do not need to fully evaluate all trials for a full 64 epochs in order to select the optimal hyperparameter configurations. More often than not, the least promising trials can be identified after a number of epochs and we can perform early stopping of these trials to reallocate our resources to other more promising ones.
Asynchronous algorithm
Unfortunately, as SHA needs to wait for all trials in a rung to complete before it can continue on to the next rung, it is difficult to parallelize optimally in the presence of stragglers, i.e., trials that run more slowly than the rest. And suppose we have 10 computing nodes and allocated one computing node per trial, SHA will also result in more and more leftover computing nodes that are not utilized as the number of trials decreases for higher rungs (e.g., 6 idle nodes at rung 3).
As such, ASHA tweaks the SHA algorithm by promoting configurations whenever possible, rather than waiting for all trials to finish in a rung. Referring to the same example in table above, ASHA will promote one trial to next rung for every 4 trials evaluated.
When compared to other existing algorithms that can be run in a parallel setting, ASHA generally performs as well as or even better according to this paper.
My intuition from the paper is that ASHA may perform better because it was able to make full use of parallel computing to evaluate as many configurations as possible given a time frame. For other algorithms, they may not be able to as fully utilise parallel computing in the presence of stragglers.
Other types of algorithms include evolutionary-based ones such as Population Based Training. For further details, please refer to DeepMind’s blog post.
Having reviewed the existing literature on hyperparameter tuning algorithms and recalling our two situations, I decided to have two separate code templates for each of the situations.
The first code template caters for the simpler situation 1, where we do not need parallel computing so I have written something that can be easily run in Google Colab. The second code template will cater for situation 2, leveraging Google Cloud Platform for distributed trials.
In terms of algorithm, the first code uses Bayesian Optimization while the second uses ASHA.
Hyperopt is a hyperparameter optimization library that implements TPE for Bayesian optimization. One of the main contributors of Hyperopt is James Bergstra.
Our code template uses the Hyperopt library and can be easily run in Google Colab with two main sections.
1. Configurations of models to explore
In this section, we set the configurations of the model that we would like to explore. For example, the code block below sets the hyperparameter space for XGBoost classifier.
# XGB Classifier parametersmodel_params = {'learning_rate': hp.choice('xgbc.learning_rate', np.arange(0.05, 0.31, 0.05)),'max_depth': hp.choice('xgbc.max_depth', np.arange(5, 16, 1, dtype=int)),'min_child_weight': hp.choice('xgbc.min_child_weight', np.arange(1, 8, 1, dtype=int)),'colsample_bytree': hp.choice('xgbc.colsample_bytree', np.arange(0.3, 0.8, 0.1)),'subsample': hp.uniform('xgbc.subsample', 0.8, 1),'n_estimators': 100,}
For your convenience, we have set the template for 3 main models, each with a classifier and regressor:
XGBoost
LightGBM
RandomForest
To adapt the code template for your purpose, simply amend the relevant code block accordingly.
2. Hyperopt class object
This section contains our main code.
To perform hyperparameter optimization, simply instantiate the HyperOpt class object and call the optimize method. Furthermore, you can also perform model search by including other models in the experiment_space dictionary.
# set dictionary containing hyperparameter space definedexperiment_space = {'model_type': hp.choice('model_type',[rf_clf_params, xgb_clf_params, lgbm_clf_params])}# Perform optimal model searchparam_search = HyperOpt(X, y, cv=5)param_search.optimize(space=experiment_space, max_evals=2)
In the above code, we have set max_evals to 2 for example, but you can also set it to other number of trials. Alternatively, set it to ‘auto’ to let the code search for the optimal settings till the loss metric does not improve for a predefined number of trials.
Ray Tune is a Python library for hyperparameter tuning at any scale, allowing us to easily perform multi-node distributed computing to evaluate various hyperparameter configurations at the same time.
Our code took references from Ray Tune’s GitHub and consists of two files that can be downloaded here.
cluster_config_cpu.yml (configurations for distributed cluster)tune_cifar10.py (script for hyperparameter tuning)
cluster_config_cpu.yml (configurations for distributed cluster)
tune_cifar10.py (script for hyperparameter tuning)
We will be leveraging Google Cloud for distributed computing and using CIFAR-10 as example. Note that Google Cloud has a free trial for new customers.
In our CIFAR-10 example, we were able to evaluate 100 configurations using 40 CPUs in an hour for what would usually take a day. All these are achieved in 5 main steps after creating a Google Cloud account.
1. Setup Google Cloud Authentication
Create service account and download JSON file that contains your key
Set environment variable to point to the directory of the JSON file downloaded
2. Enable the following APIs:
Cloud resource manager API
Identity and Access Management (IAM) API
Compute Engine API
3. Copy paste project ID to project_id in cluster_config_cpu.yml config file
4. Launch your cluster by running in terminal:
ray up -y cluster_config_cpu.yml
5. Start hyperparameter tuning trials by executing in terminal:
ray submit cluster_config_cpu.yml tune_cifar10.py# To trial run scripts, add argument smoke-test# ray submit cluster_config_cpu.yml tune_cifar10.py --smoke-test
While the hyperparameter tuning process is ongoing, you will see the status updates in terminal such as the screenshot below. Metrics (acc, total time(s)) will be reported for each trial after each iteration (iter).
In our example, we were tuning three hyperparameters:
Dropout layer’s probability of setting an element to zero (dropout_p)
Number of nodes in second fully connected layer (dense2_nodes)
Number of channels to output for second convolution layer (out_channels)
The hyperparameter space can be set in tune_cifar10.py, together with other main configurations, at the top of the script.
To adapt the code for your purpose, you may wish to modify the ConvNet class (for model architecture) and get_data_loaders function (for dataset loading).
To set the type and number of CPUs to use, please modify the head_node, worker_nodes and max_workers in cluster_config_cpu.yml. Alternatively, to launch GPUs, please see cluster_config_gpu.yml.
At the end of the hyperparameter tuning process, the best configuration will be reported and the corresponding model checkpoint can be downloaded from the head node of the Google Cloud cluster.
Quick tips:
If you are launching a large number of CPUs, do make sure that your head node is a sufficiently powerful machine type.
Google Cloud by default has quota limits for number of CPUs and GPUs that you can launch. You could however edit your quota.
Through the illustrations above, we hope to have demonstrated how hyperparameter tuning can be performed for both machine learning and deep learning models.
In case you missed the code template links, here they are again:
Google Colab notebook for situation 1
Github repository for situation 2
Thanks for reading and I hope the codes and article are useful. Please also feel free to comment with any questions or suggestions you may have.