GridSearch VS RandomizedSearch VS BayesSearch | by Jay Hui | Towards Data Science

You may know optimization algorithms (GridSearch, RandomizedSearch or even BayesianSearch) for tuning hyper-parameters in the machine learning model. So which one should you use it? And how to use it? In this post, I am going to give a tutorial with real code demo and comparison:
www.kaggle.com
We set up our tutorial as follow:
Noted that the GridSearchCV, RandomizedSearch (in sci-kit learn) and BayesSearchCV (in Skopt, sci-kit learn optimization) are modelling optimization implementations. We would like to compare these three algorithms by time (based on Kaggle’s kernel) and accuracy.
Here are the implementations of these three algorithms:
best score of Grid Search over 120 iterations: 0.504CPU times: user 823 ms, sys: 62.1 ms, total: 885 ms
best score of Randomized Search over 120 iterations: 0.708CPU times: user 876 ms, sys: 23.2 ms, total: 899 ms
Noted that the input range the ”uniform” is (loc, loc + scale).
best score of Bayes Search over 60 iterations: 0.728CPU times: user 2min 7s, sys: 2.11 s, total: 2min 9s
Noted that the input range of the ”Real” is [lower bound, upper bound].
So based on the result, you can see BayesSearch provides the best result, even it only needs 60 iterations, while GridSearch and RandomizedSearch are searching for 120 iterations, but BayesSearch needs much more time.
So, what can we learn from this simple experiment?
RandomizedSearch defeats GridSearch
RandomizedSearch defeats GridSearch
For GridSearch and RandomizedSearch, you should always use RandomSearch as RandomSearch performs better (search time should be similar). The reason is that RandomSearch can easily and efficiently find the optimized values across important parameters.
To illustrate it, consider the above picture with 1 important parameter and 1 unimportant parameter. With 3x3 = 9 combinations of GridSearch, actually, it only searches 3 different values for the important parameter in 9 iterations. However, for Randomized Search, it can search 9 different values for the 9 iterations. As a result, it is much easier for RandomizedSearch to search for the important parameters.
2. BayesSearch VS RandomizedSearch
As RandomizedSearch searches for the parameters randomly, what if we search it intentionally and directionally with the idea similar to Gradient Descent? So what we can further improve it is to consider the conditional probability (bases’ rule) to search for the parameters more wisely.
Since BasesSearch includes few computations for the conditional probability, for a small dataset, BayesSearch will take more time than the RandomizedSearch. Still, with a huge dataset, BayesSearch should have better performance than the RandomizedSearch given complex parameter search distributions and enough iterations.
Let us do one more experiment with a larger dataset:
best score of Randomized Search over 10 iterations: 0.91944CPU times: user 22.9 s, sys: 157 ms, total: 23.1 s
best score of Bayes Search over 10 iterations: 0.95512CPU times: user 1min 28s, sys: 749 ms, total: 1min 28s
In the first experiment with the small dataset, the time ratio between BayesSearch and RandomizedSearch is 129/0.9 = 143x. For the second experiment with the large dataset, the ratio now reduces to 88/23.1 = 3.81x.
In conclusion, based on the theory and computation experiments, we suggest the following:
You should choose RandomizedSearch over GridSearch. (Unless you just want to tune very few hyper-parameters.)For a huge dataset (≥ 100,000 records with more than 100 features), BayesSearch can search the optimal set of hype-parameters more efficiently.
You should choose RandomizedSearch over GridSearch. (Unless you just want to tune very few hyper-parameters.)
For a huge dataset (≥ 100,000 records with more than 100 features), BayesSearch can search the optimal set of hype-parameters more efficiently.