7 Hyperparameter Optimization Techniques Every Data Scientist Should Know | by Satyam Kumar | Towards Data Science

Selecting the right machine learning model and the corresponding correct set of hyperparameters is essential to train a robust machine learning model. The performance of the machine learning model improves with hyperparameter tuning.
Hyperparameters refer to the parameters that the model cannot learn and need to be provided before training. Hyperparameter tuning basically refers to tweaking the parameters of the model, which is basically a lengthy process.
In this article, we will discuss 7 techniques for Hyperparameter Optimization along with hands-on examples.
Hyperparameter Optimization Checklist:1) Manual Search2) Grid Search3) Randomized Search4) Halving Grid Search5) Halving Randomized Search6) HyperOpt-Sklearn7) Bayes Search
Credit Card Fraud Detection dataset will be used to train a baseline logistic regression model. Further hyperparameters of the logistic regression model will be tuned using various techniques to improve the performance.
One does not need a dedicated library for Manual tuning of hyperparameters, instead, the developer needs to try the different combination of hyperparameters for the model and select the model the performs the best. max_depth, gamma, reg_lambda, scale_pos_weight are some of the hyperparameters that can be tuned for an XBGClassifer model.
One can try all combinations of hyperparameters values and train models for each, and pick the model with the best performance. Looping through different values of hyperparameters and evaluating for each combination can also be another manual search approach for hyperparameter optimization.
Manual Seach is a bit tedious process and hence requires unnecessary manpower.
Grid Seach can be termed as an automated version of manual search hyperparameter optimization. Scikit-Learn library comes with a GridSearchCV implementation. GridSearch is not computational friendly as it takes a lot of time to optimize, but one can be free from writing multiple lines of code.
One can feed the training model and list of hyperparameters in a dictionary format to the GridSeachCV function, and its returns with the performing model and its scoring metric.
Grid Search tries all combinations of hyperparameters hence increasing the time complexity of the computation. Randomized Search trains model on random hyperparameter combination. The total number of combinations on which the several models are trained is less for randomized search compared to grid search.
Scikit-Learn package also comes with RandomSearchCV implementation.
Halving Grid Search is an optimized version of Grid Search hyperparameter optimization. Halving Grid Search searches over a specified list of hyperparameters using a successive halving approach. The search strategy starts evaluating all the candidates on a small sample of the data and iteratively selects the best candidates using more and more larger samples.
Halving Grid Search is less computationally expensive than the Grid Search approach. Scikit-Learn library comes up with HalvingGridSearch implementation.
Read the below-mentioned article to know how Halving Grid Search CV can speed up hyperparameter optimization by 20x times.
towardsdatascience.com
Halving Randomized Search uses the same successive halving approach, and it is further optimized compared to Halving Grid Search. Unlike Halving Grid Search, it does not train on all combinations of hyperparameters instead randomly picks a set of combinations of hyperparameters. Scikit-Learn library also comes up with HalvingRandomizedSeachCV.
Hyperopt is an open-source Python library for Bayesian Optimization, designed for large-scale optimizations of models with hundreds of parameters. It allows the hyperparameter optimization to be scaled across multiple cores of the CPU.
Hyperopt-Sklearn is an extension of the Hyperopt library, that allows for the automatic search of machine learning algorithms, and model hyperparameters for classification and regression tasks.
Read this article to know more about the usage and implementation of the Hyperopt-Sklearn package.
Bayes Grid Search uses the Bayesian optimization technique to model the search space to arrive at optimized parameter values as soon as possible. It uses the structure of search space to optimize the search time. Bayes Search approach uses the past evaluation results to sample new candidates that are most likely to give better results.
Scikit-Optimize library comes with BayesSearchCV implementation.
In this article, we have covered 7 hyperparameter optimization techniques one can use to get the best set of hyperparameters leading to training a robust machine learning model. The trade-off between the best model performance and the most optimized optimization technique is a factor that influences someone’s choice.
For a few components, one can use the GridSearchCV technique. But when the number of components increases one can try Halving Grid Search CV or Randomized Search CV as they are not computationally expensive.
[1] Scikit-Learn Documentation: https://scikit-learn.org/stable/modules/grid_search.html
Thank You for Reading