Hyper-Parameter Tuning in Python. Grid Search vs Random Search | by Songhao Wu | Towards Data Science

When I have a result after running a model, I always want to ask myself. Is this the best result I can get?
Besides changing the algorithm or increasing the amount of training data, we can also utilize Hyper-Parameter tuning to make the model more robust for our dataset.
Firstly, let us be clear on the terms. What is a parameter and what is a hyper-parameter?
Parameter: Model parameter that can be trained and adjusted by training data. E.g. y=Ax+B. A and B are parameters that can be trained by the training data.
Hyper-parameter: Parameters from the models that are not affected by the training data. It defines the underlying model structure for that particular algorithm. E.g. Number of leave for the decision tree algorithm
This article will cover what are the general steps to do the hyper-parameter tuning and two frequently used packages for auto-tuning. For the demo purpose, I have created a classification dataset using the make_classification package.
#Create classification datasetfrom sklearn.datasets import make_classificationX, Y = make_classification(n_samples=10000, n_informative=10,n_redundant=0, n_classes=3,                               n_features=10,                               random_state=0)#split into training and testing setfrom sklearn.model_selection import train_test_splitX_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=0)
import timestart = time.time()import lightgbm as lgbfrom sklearn.metrics import accuracy_scoreclf = lgb.LGBMClassifier()clf.fit(X=X_train, y=Y_train)predicted=clf.predict(X_test)print('Classification of the result is:')print(accuracy_score(Y_test, predicted))end = time.time()print('Execution time is:')print(end - start)
I used Light GBM here as the algorithm. The baseline model has an accuracy of 88.8% and it takes only 0.3 seconds to complete the whole process.
Firstly, we need to identify the main goals that we trying to achieve for the tuning process. Usually, there are 3 common goals.
Increase AccuracyReduce OverfittingFaster Running Speed
Increase Accuracy
Reduce Overfitting
Faster Running Speed
The first two goals can be achieved by auto-tuning hyper-parameters. We can evaluate the model by testing it on an out-of-sample dataset (meaning we did not use it for training).
The last goal ‘running speed’ is usually required when dealing with very large amounts of data. If the data science model is deployed and running daily, we expect the model to give a result within a reasonable timeframe. Usually, we can tune this part manually at the last step.
For my example, I do not have a very large dataset and thus I am only concerned about the accuracy and overfitting issue(first two goals).
After we are clear on what we want to achieve, we should determine the parameters for each goal's tuning. Depending on the algorithm you choose, usually, you can find the documentation on all the possible parameters for tuning.
In this example, I am using Light GBM and you can find the whole list of parameters here.
Below are the 5 hyper-parameters that I chose for auto-tuning:
num_leaves: maximum number of leaves in one tree, main parameter to tune for a tree model
min_child_samples: Minimum number of data in one leave
max_depth: maximum depth of the tree
learning_rate: shrinkage rate, determine how fast the model can learn
reg_alpha: regularisation, dealing with overfitting
Num_leaves, min_child_samples, max_depth, and reg_alpha can affect both accuracy and overfitting. Higher the value of num_leaves and max_depth, higher the accuracy and also causing more overfitting. Higher the min_child_samples and the reg_alpha, lesser the overfitting but would lower the accuracy. Learning rate determines the speed of model learning but would also affect accuracy.
Usually, we are clueless on what value range to set for auto-tuning when we first start using certain algorithms. What I do is first find out the default value for that hyper-parameter and then set the value range around the default value.
I would usually run the model 2 rounds. For the first round, I would usually set the interval to be largely aiming to find a suitable range of parameter values. For example in my case, num_leaves is the most important parameter to tune. The default value is 31. Thus firstly, I set the range to be [10,20,30,60,90,120] which has a big range.
After the first round of results, I get the best parameter value for num_leaves to be 60. For the second round, I would set the interval again around 60 at a closer interval like [40,50,60,70,80].
You do not need to run the model too many times because best parameters are a combination of different parameters so there is no best value for a single parameter. Running the model 2 times is just an approach for you to narrow down the parameter value within the reasonable range.
Here, I am using the two commonly used methods GridSearch and RandomSearch for auto-tuning. Both approaches adopt the ‘fit and score’ method. This means it would select one set of hyper-parameter from the defined range, fit the model, produce the score, and in the end output the model with the highest score. ‘Score’ here is defined by us and it depends on what we are trying to maximize. E.g. it can be accuracy, log loss, F1 score, etc.
Grid Search vs Random Search
Grid Search: Exhaustive search over the pre-defined parameter value range. The number of trials is determined by the number of tuning parameters and also the range.
start = time.time()from sklearn.model_selection import GridSearchCVimport lightgbm as lgblgb=lgb.LGBMClassifier()#Define the parametersparameters = {'num_leaves':[20,40,60,80,100], 'min_child_samples':[5,10,15],'max_depth':[-1,5,10,20],             'learning_rate':[0.05,0.1,0.2],'reg_alpha':[0,0.01,0.03]}#Define the scoringclf=GridSearchCV(lgb,parameters,scoring='accuracy')clf.fit(X=X_train, y=Y_train)print(clf.best_params_)predicted=clf.predict(X_test)print('Classification of the result is:')print(accuracy_score(Y_test, predicted))end = time.time()print('Execution time is:')print(end - start)
Compared to the baseline model, Grid Search increases accuracy by around 1.2%. However, the running time is 4 plus hours!
Random Search: Take a random sample from the pre-defined parameter value range. The number of trials is determined by the ‘n_iter’ parameter so there is more flexibility. In addition, as the picture shows, when you are having a continuous value range, it can retrieve more randomly distributed cases rather than parameter sets from the ‘gridline’.
start = time.time()from sklearn.model_selection import RandomizedSearchCVimport lightgbm as lgblgb=lgb.LGBMClassifier()parameters = {'num_leaves':[20,40,60,80,100], 'min_child_samples':[5,10,15],'max_depth':[-1,5,10,20],             'learning_rate':[0.05,0.1,0.2],'reg_alpha':[0,0.01,0.03]}clf=RandomizedSearchCV(lgb,parameters,scoring='accuracy',n_iter=100)clf.fit(X=X_train, y=Y_train)print(clf.best_params_)predicted=clf.predict(X_test)print('Classification of the result is:')print(accuracy_score(Y_test, predicted))end = time.time()print('Execution time is:')print(end - start)
I specified the number of trials to be 100. The accuracy of the random search tuned model is just 0.3% less than the grid search. However, it only takes 3.5 minutes which is much faster than Grid Search.
Depend on the goal and tuned model result, you can choose to tune some hyper-parameter manually. For example, if the testing accuracy is still much lower than training accuracy, you can choose to set a higher value for regularisation to reduce overfitting. Or if you want to improve running speed you can set a higher learning rate or reduce num_leaves.
The main message for this article is not just the technical part on auto-tuning of hyper-parameters but also the thinking process of how should we tune the hyper-parameters.
Between Grid Search and Random Search, I would recommend using random search because of few reasons:
Random Search has more flexibility and control on how many trials you want to run.The running time for Random Search is usually much lower and so it gives you more time to try out more rounds of tuning.For continuous hyper-parameter, Random Search can explore a more distinct set of values
Random Search has more flexibility and control on how many trials you want to run.
The running time for Random Search is usually much lower and so it gives you more time to try out more rounds of tuning.
For continuous hyper-parameter, Random Search can explore a more distinct set of values
If you have any questions or recommendations or any new topics you want to learn about, please put them in the comment!
Thanks for reading and hopefully it helps you get a better understanding of hyper-parameter tuning.