20x times faster Grid Search Cross-Validation | by Satyam Kumar | Towards Data Science

To train a robust machine learning model, one must select the correct machine learning algorithm with the correct combination of hyperparameters. The process of choosing the optimal set of parameters is known as hyperparameter tuning.
One must train the dataset on all machine learning algorithms and on a different combination of its hyperparameters to improve the performance metric. The cross-validation technique can be used to train the dataset on various machine learning algorithms and choose the best out of it.
Cross-Validation is a resampling technique that can be used to evaluate and select machine learning algorithms on a limited dataset. k-fold cross-validation is a type of cross-validation, where the training data is split into k-folds and (k-1) folds is used for training and kth fold is used for validation of the model.
Read the scikit-learn documentation of cross-validation for better understanding.
Grid Search cross-validation is a technique to select the best of the machine learning model, parameterized by a grid of hyperparameters. Scikit-Learn library comes with grid search cross-validation implementation.
Grid Search CV tries all combinations of parameters grid for a model and returns with the best set of parameters having the best performance score. This can also serve as a disadvantage, as training the model of each combination of parameters increases the time complexity.
param_grid = {"max_depth": [3, 4, 7, 10, 25],"gamma": [0.5, 1, 5, 10, 25],"min_child_weight": [1, 3, 5, 10, 25],"reg_lambda": [5, 10, 50, 100, 300],"scale_pos_weight": [1, 3, 5, 10, 25]}# Grid Search CV implementationxgb_cl = xgb.XGBClassifier(objective="binary:logistic")grid_cv = GridSearchCV(xgb_cl, param_grid, scoring="roc_auc", n_jobs=-1, cv=3)grid_cv.fit(X_train, y_train)# Return set of parameters with the best performancegrid_cv.best_params_# Return the performance metric scoregrid_cv.best_score_
The above code snippet explains the usage of Grid Search CV, with cv=3 and it trains the X_train dataset on all combination of param_grid parameters.
For a large size dataset, Grid Search CV time complexity increases exponentially, and hence it’s not practically feasible.
One can shift to Random Search CV where the algorithm will randomly choose the combination of parameters. The performance of Random Search CV is somewhat equal or less than Grid Search CV but the time complexity decreases and it becomes practically feasible to apply Grid Seach of a large-size dataset.
In this article, we will discuss a new algorithm Halving Grid Search CV, that performs equally well as Grid Search CV with a large decrease in time complexity compared to Grid Search.
While both Grid Search CV and Random Search CV train all the components (combination of parameter grid) on the entire data, Halving Grid Search CV or Halving Random Search CV follows a successive halving approach.
The steps Halving Grid Search (HGS) CV follows:
HGS trains a subset of data to all the combinations of parameters.Top-performing candidates or combinations are selected.A larger subset of training data is trained on the top-performing candidates.The above 3 steps are repeated until the best set of hyperparameters are left standing.
HGS trains a subset of data to all the combinations of parameters.
Top-performing candidates or combinations are selected.
A larger subset of training data is trained on the top-performing candidates.
The above 3 steps are repeated until the best set of hyperparameters are left standing.
Using HGS, for each passing iteration, the parameter components are decreasing and the training dataset is increasing. Since the algorithm follows a successive halving approach, so the time complexity of the algorithm is comparatively very less compared to Grid Search CV.
Scikit-Learn library comes with the implementation of the Halving Grid Search CV algorithm. The below code snippet self explains the working and implementation of the algorithm.
param_grid = {"max_depth": [3, 4, 7, 10, 25],"gamma": [0.5, 1, 5, 10, 25],"min_child_weight": [1, 3, 5, 10, 25],"reg_lambda": [5, 10, 50, 100, 300],"scale_pos_weight": [1, 3, 5, 10, 25]}# Grid Search CV implementationxgb_cl = xgb.XGBClassifier(objective="binary:logistic")halving_cv = HalvingGridSearchCV(xgb_cl, param_grid, scoring="roc_auc", n_jobs=-1, min_resources="exhaust", factor=3)halving_cv.fit(X_train, y_train)# Return set of parameters with the best performancehalving_cv.best_params_# Return the performance metric scorehalving_cv.best_score_
Now let's compare the performance score and time complexity between Grid Search CV and Halving Search CV.
Credit Card Fraud dataset from Kaggle is used to compute the time numbers. This is a classification problem, and ROC-AUC score was used as the metric to compare the performance.
Only ~7.5k records were used for training with cv=3, and ~3k records for testing purpose.
Observing the above time numbers, for parameter grid having 3125 combinations, the Grid Search CV took 10856 seconds (~3 hrs) whereas Halving Grid Search CV took 465 seconds (~8 mins), which is approximate 23x times faster.
In this article, we have discussed an optimized approach of Grid Search CV, that is Halving Grid Search CV that follows a successive halving approach to improving the time complexity. One can also try Random Search CV or Halving Random Search CV that may further reduce the time complexity but sometimes with the cost of the performance score.
The best part is Halving Grid Search CV implementation comes with the scikit-learn package for usage.
[1] Halving Grid Search CV documentation: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.HalvingGridSearchCV.html
Thank You for Reading