Hyperparameter Tuning with Grid Search and Random Search | by Idil Ismiguzel | Towards Data Science

Hyperparameter tuning also known as hyperparameter optimization is an important step in any machine learning model training that directly affects model performance.
This article covers two very popular hyperparameter tuning techniques: grid search and random search and shows how to combine these two algorithms with coarse-to-fine tuning. By the end of the article, you will know their working principles and main differences that will help you to decide which one to use confidently.
While following the article, I encourage you to check out the Jupyter Notebook on my GitHub for the full analysis and code. 🌠
Hyperparameters are parameters that are defined before training to specify how we want model training to happen. We have full control over hyperparameter settings and by doing that we control the learning process.
For example in the random forest model n_estimators (number of decision trees we want to have) is a hyperparameter. It can be set to any integer value but of course, setting it to 10 or 1000 changes the learning process significantly.
Parameters, on the other hand, are found during the training. We have no control over parameter values as they are the result of model training. For example, in linear regression coefficients and intercept are parameters and found at the end of model training.
To learn model hyperparameters and their values we can simply call get_params in Python. 🔍
from sklearn.ensemble import RandomForestClassifier# Instantiate the modelrf_model = RandomForestClassifier()# Print hyperparametersrf_model.get_params
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion=’gini’, max_depth=None, max_features=’auto’, max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=None, oob_score=False, random_state=None, verbose=0, warm_start=False)
As you can see random forest classifier has many hyperparameters and if the model is instantiated without defining hyperparameters, then it will have default values. In the random forest by default n_estimators=100 which results in a moderately small forest. (only 100 trees 🌳)
You need to know that in any model (in this case random forest) some hyperparameters are more important than others such as:
n_estimators: Number of decision treesmax_features: Maximum number of features considered while splittingmax_depth: Max depth of the treemin_samples_leaf: Minimum number of data points in a leaf nodebootstrap: Sampling with or without replacement
And some hyperparameters do not affect model performance such as:
n_jobs: Number of jobs to run in parallelrandom_state: Seedverbose: Printing information while training continuesoob_score: Whether or not to use out-of-bag samples
Finally, hyperparameter tuning is finding the best combination of hyperparameters that gives the best performance according to the defined scoring metric.
In this article, we will be using Glass Identification data set from UCI, where we have 9 attributes to predict the type of glass (out of 7 discrete values).
Next, I will separate X and y, and generate train and test sets.
# Seperate X and yX = df.drop(columns=['Type'], axis=1)y = df['Type']# Generate training and test sets for X and yX_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=1)
After that, we simply run a random forest classifier with default values and get the predictions for the test set.
# Instantiate and fit random forest classifierrf_model = RandomForestClassifier()rf_model.fit(X_train, y_train)# Predict on the test set and call accuracyy_pred = rf_model.predict(X_test)accuracy = accuracy_score(y_test, y_pred)print(accuracy)
0.81
As you can see accuracy with the default model is 81%. Now we will see how to use grid search to tune selected hyperparameters.
Grid Search starts with defining a search space grid. The grid consists of selected hyperparameter names and values, and grid search exhaustively searches the best combination of these given values. 🚀
Let’s say we decided to define the following parameter grid to optimize some hyperparameters for our random forest classifier.
param_grid:n_estimators = [50, 100, 200, 300]max_depth = [2, 4, 6, 8, 10]min_samples_leaf = [1, 5, 10]max_features = ['auto', 'sqrt']bootstrap = [True, False]
We will use GridSearchCV class from Scikit-Learn library for this optimization. The first thing to mention is grid search will have to run and compare 240 models (=4*5*3*2*2, multiplication of the selected number of values).
Moreover, GridSearchCV class has the option to perform cross validation to resample the training and test data into multiple folds. By applying cross validation we use every record in data for training and testing instead of splitting the dataset at one time as training and testing. If we decide to use cross validation (let’s say with 5 folds) this means grid search will have to evaluate 1200 (=240*5) model performances.
Let’s have a look at all the input parameters of GridSearchCV class:
class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, refit=True, cv=None, return_train_score=False)
We start with defining a dictionary for the grid which we will be an input for GridSeachCv.
# Define the gridparam_grid = {'n_estimators': [50, 100, 200, 300],'min_samples_leaf': [1, 5, 10],'max_depth': [2, 4, 6, 8, 10],'max_features': ['auto', 'sqrt'],'bootstrap': [True, False]}# Instantiate GridSearchCVmodel_gridsearch = GridSearchCV(estimator=rf_model,param_grid=param_grid,scoring='accuracy',n_jobs=4,cv=5,refit=True,return_train_score=True)
As decided previouslyestimator is RandomForestClassifier (rf_model) and param_grid is the parameter grid we defined above. scoring is the desired evaluation metric such as accuracy for a classification task and n_jobs executes model evaluation in parallel, but be careful if you set n_jobs=-1 it uses all the processors! Since grid search is an uninformed tuning, we can take advantage of running models in parallel as their outcome does not affect other models’ runs.
Settingrefit=True refits the estimator with the best-found hyperparameter values in the end so we do not need to code them in an additional step. cv defines the cross validation strategy and setting return_train_score=True we can print the logs of model runs to make further analysis.
# Record the current time start = time()# Fit the selected modelmodel_gridsearch.fit(X_train, y_train)# Print the time spend and number of models ranprint("GridSearchCV took %.2f seconds for %d candidate parameter settings." % ((time() - start), len(model_gridsearch.cv_results_['params'])))
GridSearchCV took 247.79 seconds for 240 candidate parameter settings.
# Predict on the test set and call accuracyy_pred_grid = model_gridsearch.predict(X_test)accuracy_grid = accuracy_score(y_test, y_pred_grid)
0.88
As you can see, simply tuning some hyperparameters increased the initial accuracy from 81% to 88% spending 247 seconds to hyperparameter tuning.
Grid search always finds the best-performing model with hyperparameter values mentioned in the grid. It is also easy to implement and explain. However, with the increasing number of hyperparameters and values to test it can easily become computationally expensive because it models all of the combinations of hyperparameters. The drawback of not learning from already ran models makes grid search inefficient and time-consuming. In addition, parameter grid plays an extremely important role: even though grid search will always find the best combination if the parameter grid is selected poorly the best combination will not be performing well.
After running the GridSeachCV, we can return the following attributes for further investigation:
cv_results_
best_estimator_
best_score_
best_params_
Let’s look at some of them:
print(model_gridsearch.best_params_)
{‘bootstrap’: True, ‘max_depth’: 10, ‘max_features’: ‘sqrt’, ‘min_samples_leaf’: 1, ‘n_estimators’: 300}
print(model_gridsearch.best_estimator_)
RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None, criterion=’gini’, max_depth=10, max_features=’sqrt’, max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0, min_impurity_split=None, min_samples_leaf=1, min_samples_split=2, min_weight_fraction_leaf=0.0, n_estimators=300, n_jobs=None, oob_score=False, random_state=None, verbose=0, warm_start=False)
In random search, we define distributions for each hyperparameter which can be defined uniformly or with a sampling method. The key difference from grid search is in random search, not all the values are tested and values tested are selected at random.
For example, if there are 500 values in the distribution and if we input n_iter=50 then random search will randomly sample 50 values to test. By doing that random search optimizes time spend and not defining an absolute grid allows it to explore other values in the given distribution.
Since random search does not try every hyperparameter combination, it does not necessarily return the best performing values, but it returns a relatively good performing model in a significantly shorter time. ⏰
param_distributionsn_estimators = list(range(100, 300, 10))min_samples_leaf = list(range(1, 50))max_depth = list(range(2, 20)max_features = ['auto', 'sqrt']bootstrap = [True, False]
RandomizedSearchCV from Scikit-Learn has the following input parameters:
class sklearn.model_selection.RandomizedSearchCV(estimator, param_distributions, n_iter=10, scoring=None, n_jobs=None, refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', random_state=None, error_score=nan, return_train_score=False)
We start with defining a dictionary for parameter distributions which will be an input for RandomizedSearchCV.
# specify distributions to sample fromparam_dist = {'n_estimators': list(range(50, 300, 10)),'min_samples_leaf': list(range(1, 50)),'max_depth': list(range(2, 20)),'max_features': ['auto', 'sqrt'],'bootstrap': [True, False]}# specify number of search iterationsn_iter = 50# Instantiate RandomSearchCVmodel_random_search = RandomizedSearchCV(estimator=rf_model,param_distributions=param_dist,n_iter=n_iter)
Most parameters are similar to GridSearchCV, however, in RandomizedSearchCV we have param_distributions for defining the search distributions. n_iter is used to limit the total number of model runs (in other words parameter combinations that are sampled from the grid). Be careful with the trade-off here, since setting high n_iter increases search runtime, and setting low n_iter decreases model quality.
# Record the current time start = time()# Fit the selected modelmodel_random_search.fit(X_train, y_train)# Print the time spend and number of models ranprint("RandomizedSearchCV took %.2f seconds for %d candidate parameter settings." % ((time() - start), len(model_random_search.cv_results_['params'])))
RandomizedSearchCV took 64.17 seconds for 50 candidate parameter settings.
# Predict on the test set and call accuracyy_pred_random = model_random_search.predict(X_test)accuracy_random = accuracy_score(y_test, y_pred_random)
0.86
As you see only in 64 seconds we were able to increase accuracy of the initial model from 81% to 86%. Random search did not reach 88% accuracy of grid search, however, this is the tradeoff between two tuning methods.
Finally, let’s have a look at the best parameters found with random search.
print(model_random_search.best_params_)
{‘n_estimators’: 230, ‘min_samples_leaf’: 4, ‘max_features’: ‘auto’, ‘max_depth’: 13, ‘bootstrap’: False}
From the beginning of the article, we have seen how to apply grid search and random search for hyperparameter optimization.
Using grid search, we were able to test all of the hyperparameter values given in the grid to find the best among them. However, an increased number of hyperparameters easily becomes a bottleneck. Ideally, we can combine grid search with random search to prevent this inefficiency.
In coarse-to-fine tuning, we start with a random search to find the promising value ranges for each hyperparameter. For example, if the random search returns high performance for n_estimators between 150 and 200, this is the range we want grid search to focus on. After getting focus area for each hyperparameter using random search, we can define the grid accordingly for grid search to find the best values amongst them.
In the illustration above first, we are testing values at random to find a focus area with random search. Second, we are testing all values in this focus area with grid search and eventually find the optimal value.
Another useful strategy to fine-tune machine learning models is using ensemble learning techniques. Different from hyperparameter tuning, ensemble learning aims to improve model performance by combining multiple models into a group model. This group model aims to perform better than each model alone. The most used ensemble learning methods are voting, bagging, boosting and stacking and if you want to learn more or refresh your knowledge, you can read my article on the topic. 🐬
In this article, we used a random forest classifier to predict “type of glass” using 9 different attributes. Initial random forest classifier with default hyperparameter values reached 81% accuracy on the test. Using grid search we were able to tune selected hyperparameters in 247 seconds and increased accuracy to 88%. Next, we did the same job using random search and in 64 seconds we increased accuracy to 86%. Last but not least, we discussed coarse-to-fine tuning to combine these two methods and get advantages from both!
I hope you enjoyed reading about hyperparameter tuning and find the article useful for your analyses!
If you liked this article, you can read my other articles here and follow me on Medium. Let me know if you have any questions or suggestions.✨
Enjoy this article? Become a member for more!