Faster Hyperparameter Tuning with Scikit-Learn’s HalvingGridSearchCV | by Kyle Gilde | Towards Data Science

If you are a Scikit-Learn fan, Christmas came a few days early in 2020 with the release of version 0.24.0. Two experimental hyperparameter optimizer classes in the model_selection module are among the new features: HalvingGridSearchCV and HalvingRandomSearchCV.
Like their close cousins GridSearchCV and RandomizedSearchCV, they use cross-validation to find optimal hyperparameters. However, instead of independently searching the hyperparameter set candidates, their successive halving search strategy “starts evaluating all the candidates with a small number of resources and iteratively selects the best candidates, using more and more resources.” The default resource is the number of samples, but the user can set it to any positive-integer model parameter like gradient boosting rounds. Thus, the halving approach has the potential of finding good hyperparameters in less time.
I read through Scikit-Learn’s “Comparison between grid search and successive halving” example, but because takes a grand total of 11 seconds to run, I was still unclear about the real-world impact of using the halving versus exhaustive approach. So I decided to set up an experiment to answer the following questions:
How much faster is HalvingGridSearchCV compared to GridSearchCV?Does HalvingGridSearchCV still select the same hyperparameter set that GridSearchCV does?
How much faster is HalvingGridSearchCV compared to GridSearchCV?
Does HalvingGridSearchCV still select the same hyperparameter set that GridSearchCV does?
I’m going to run and compare 3 searches:
GridSearchCVHalvingGridSearchCV using the default “n_samples” resourceHalvingGridSearchCV using the CatBoost’s “n_estimators” as the resource
GridSearchCV
HalvingGridSearchCV using the default “n_samples” resource
HalvingGridSearchCV using the CatBoost’s “n_estimators” as the resource
The first step is to upgrade your version of Scikit to 0.24.0 and make sure you can import the correct version.
# !! pip install scikit-learn --upgradeimport sklearnprint(sklearn.__version__)0.24.0
I ran my tests using the Kaggle’s Ames, IA house prices dataset. It has 1,460 observations and 79 features. The dependent variable is SalePrice of the home. I recommend reading this notebook if you are interested in some exploratory data analysis on the dataset.
import numpy as np  import pandas as pd  DEP_VAR = 'SalePrice'train_df = pd.read_csv('../kaggle/input/house-prices-advanced-regression-techniques/train.csv')\           .set_index('Id')            y_train = train_df.pop(DEP_VAR)
I also wrote a script called pipeline_ames.py. It instantiates a Pipeline containing some feature transformations and the CatBoostRegressor. I’ve plotted its visual representation below. (You can read more about my approach to feature engineering in my previous post.)
from sklearn import set_config        from sklearn.utils import estimator_html_repr   from IPython.core.display import display, HTML     from pipeline_ames import pipeset_config(display='diagram')display(HTML(estimator_html_repr(pipe)))
The grid_search_paramsdictionary contains the control parameters that were used in the 3 searches. I performed 3-fold cross-validation on param_grid, which contains 4 CatBoost hyperparameters with 3 values each. The results were measured in root mean squared log error (RMSLE).
from sklearn.metrics import mean_squared_log_error, make_scorernp.random.seed(123) # set a global seedpd.set_option("display.precision", 4)rmsle = lambda y_true, y_pred:\         np.sqrt(mean_squared_log_error(y_true, y_pred))scorer = make_scorer(rmsle, greater_is_better=False)param_grid = {"model__max_depth": [5, 6, 7],              'model__learning_rate': [.01, 0.03, .06],              'model__subsample': [.7, .8, .9],              'model__colsample_bylevel': [.8, .9, 1]}grid_search_params = dict(estimator=pipe,                          param_grid=param_grid,                          scoring=scorer,                          cv=3,                          n_jobs=-1,                          verbose=2)
The baseline exhaustive grid search took nearly 33 minutes to perform 3-fold cross-validation on our 81 candidates. We will see if the HalvingGridSearchCV process can find the same hyperparameters in less time.
%%timefrom sklearn.model_selection import GridSearchCVfull_results = GridSearchCV(**grid_search_params)\               .fit(train_df, y_train)pd.DataFrame(full_results.best_params_, index=[0])\    .assign(RMSLE=-full_results.best_score_)Fitting 3 folds for each of 81 candidates, totalling 243 fitsWall time: 32min 53s
In the first halving grid search, I used the default ‘n_samples’ for the resource and set the min_resources to use 1/4 of the total resources, which was 365 samples. I did not use the default min_resources calculation of 22 samples because it produced terrible results.
For both halving searches, I used a factor of 2. This parameter determines the n_candidates and n_resources used in the successive iterations and indirectly determines the total number of iterations utilized in the search.
The reciprocal of the factor determines the proportion of n_candidates retained - in this case, one half. All other candidates are discarded. Hence, as you can see in the logs below, the 3 iterations in my search had 81, 41 and 21 candidates.The product of the factor and the previous iteration’s n_resources determines the n_resources. My 3-iteration search used 365, 730 and 1460 samples.The total number of iterations is determined by how many times n_resources can increase by the factor while not exceeding the max_resources. If you want the final iteration to use all of the resources, you will need to set min_resources and factor to be factors of max_resources.
The reciprocal of the factor determines the proportion of n_candidates retained - in this case, one half. All other candidates are discarded. Hence, as you can see in the logs below, the 3 iterations in my search had 81, 41 and 21 candidates.
The product of the factor and the previous iteration’s n_resources determines the n_resources. My 3-iteration search used 365, 730 and 1460 samples.
The total number of iterations is determined by how many times n_resources can increase by the factor while not exceeding the max_resources. If you want the final iteration to use all of the resources, you will need to set min_resources and factor to be factors of max_resources.
%%timefrom sklearn.experimental import enable_halving_search_cv  from sklearn.model_selection import HalvingGridSearchCVFACTOR = 2MAX_RESOURCE_DIVISOR = 4n_samples = len(train_df)halving_results_n_samples =\    HalvingGridSearchCV(resource='n_samples',                        min_resources=n_samples//\                        MAX_RESOURCE_DIVISOR,                        factor=FACTOR,                        **grid_search_params                        )\                        .fit(train_df, y_train)n_iterations: 3n_required_iterations: 7n_possible_iterations: 3min_resources_: 365max_resources_: 1460aggressive_elimination: Falsefactor: 2----------iter: 0n_candidates: 81n_resources: 365Fitting 3 folds for each of 81 candidates, totalling 243 fits----------iter: 1n_candidates: 41n_resources: 730Fitting 3 folds for each of 41 candidates, totalling 123 fits----------iter: 2n_candidates: 21n_resources: 1460Fitting 3 folds for each of 21 candidates, totalling 63 fitsWall time: 34min 46s
This first halving search did not produce good results. It actually took a little longer than the exhaustive search. Using my compare_cv_best_params function, we see that it found only the ninth optimal hyperparameter set.
from compare_functions import *compare_cv_best_params(full_results, *[halving_results_n_samples])\ .style.applymap(lambda cell: ‘background: pink’ if cell == 9 else)
In the second halving search, I used CatBoost’s n_estimators as the resource and set the first iteration’s min_resources to use a quarter of those estimators while keeping the factor set at 2.
%%timehalving_results_n_estimators =\    HalvingGridSearchCV(resource='model__n_estimators',                                                  max_resources=1000,                         min_resources=1000 // MAX_RESOURCE_DIVISOR,                         factor=FACTOR,                        **grid_search_params                        )\                        .fit(train_df, y_train)n_iterations: 3n_required_iterations: 7n_possible_iterations: 3min_resources_: 250max_resources_: 1000aggressive_elimination: Falsefactor: 2----------iter: 0n_candidates: 81n_resources: 250Fitting 3 folds for each of 81 candidates, totalling 243 fits----------iter: 1n_candidates: 41n_resources: 500Fitting 3 folds for each of 41 candidates, totalling 123 fits----------iter: 2n_candidates: 21n_resources: 1000Fitting 3 folds for each of 21 candidates, totalling 63 fitsWall time: 22min 59s
This halving search produced the results that we were hoping to see. It was finished 10 minutes earlier, so it was about 30% faster than the exhaustive grid search. Importantly, it also found the best set of hyperparameters.
compare_cv_best_params(full_results, *[halving_results_n_samples,                                        halving_results_n_estimators])\    .style.apply(lambda row: \     row.apply(lambda col: \     'background: lightgreen' if row.name == 2 else ''), \     axis=1)
The results of my HalvingGridSearchCV experiment were mixed. Using the default “n_samples” resource yielded slow and suboptimal results. If you are not using a large number of samples, limiting them may not save you any time.
However, using CatBoost’s n_estimators as the resource yielded the optimal results in less time. This tracks with my own experience manually tuning gradient boosting hyperparameters. I can usually tell pretty quickly from the validation logs whether the hyperparameter set is worth boosting for many more rounds.
Let me know if you found this post helpful. The original notebook for this blog post can be found here.
Follow me and stay tuned for further posts on training models with Scikit-Learn. Thanks!