Machine Learning: GridSearchCV & RandomizedSearchCV | by Papa Moryba Kouate | Towards Data Science

Introduction
In this article, I’d like to speak about how we can improve the performance of our machine learning model by tuning the parameters.
Obviously it is important to know the meaning of the parameters that we want to adjust in order to improve our model. For this reason, before to speak about GridSearchCV and RandomizedSearchCV, I will start by explaining some parameters like C and gamma.
Part I: An overview of some parameters in SVC
In the Logistic Regression and the Support Vector Classifier, the parameter that determines the strength of the regularization is called C.
For a high C, we will have a less regularization and that means we are trying to fit the training set as best as possible. Instead, with low values of the parameter C, the algorithm tries to adjust to the “majority” of data points and increase the generalization of the model.
There is another important parameter called gamma. But before to talk about it, I think it is important to understand a little bit the limitation of linear models.
Linear models can be quite limiting in low-dimensional spaces, as lines and hyperplanes have limited flexibility. One way to make a linear model more flexible is by adding more features, for example, by adding interactions or polynomials of the input features.
A linear model for classification is only able to separate points using a line, and that is not always the better choice. So, the solution could be to represent the points in a three-dimensional space and not in a two-dimensional space. In fact, in three-dimensional space, we can create a plane that divides and classifies the points of our dataset in a more precise way.
There are two ways to map your data into a higher-dimensional space: the polynomial kernel, which computes all possible polynomials up to a certain degree of the original features; and the radial basis function(RBF) kernel, also known as the Gaussian kernel which measures the distance between data points. Here, the task of gamma is to control the width of the Gaussian Kernel.
Part II: GridSearchCV
As I showed in my previous article, Cross-Validation permits us to evaluate and improve our model. But there is another interesting technique to improve and evaluate our model, this technique is called Grid Search.
Grid Search is an effective method for adjusting the parameters in supervised learning and improve the generalization performance of a model. With Grid Search, we try all possible combinations of the parameters of interest and find the best ones.
Scikit-learn provides the GridSeaechCV class. Obviously we first need to specify the parameters we want to search and then GridSearchCV will perform all the necessary model fits. For example, we can create the below dictionary that presents all the parameters that we want to search for our model.
parameters = {‘C’: [0.001, 0.01, 0.1, 1, 10, 100], ‘gamma’: [0.001, 0.01, 0.1, 1, 10, 100]}
Then we can instantiate the GridSearchCV class with the model SVC and apply 6 experiments with cross-validation. Of course, we need also to split our data into a training and test set, to avoid overfitting the parameters.
from sklearn.model_selection import GridSearchCV from sklearn.model_selection import train_test_splitfrom sklearn.svm import SVC search = GridSearchCV(SVC(), parameters, cv=5)X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
Now we can fit the search object that we have created with our training data.
search.fit(X_train, y_train)
So the GridSearchCV object searches for the best parameters and automatically fits a new model on the whole training dataset.
Part III: RandomizedSearchCV
RandomizedSearchCV is very useful when we have many parameters to try and the training time is very long. For this example, I use a random-forest classifier, so I suppose you already know how this kind of algorithm works.
The first step is to write the parameters that we want to consider and from these parameters select the best ones.
param = {‘max_depth: [6,9, None],          ‘n_estimators’:[50, 70, 100, 150],           'max_features': randint(1,6),          'criterion' : ['gini', 'entropy'],          'bootstrap':[True, False],          'mln_samples_leaf': randint(1,4)}
Now we can create our RandomizedSearchCV object and fit the data. Finally, we can find the best parameters and the best scores.
from sklearn.model_selection import RandomSearchCVfrom sklearn.ensemble import RandomForestClassifierrnd_search = RandomizedSearchCV(RandomForestClassifier(), param, n_iter =10, cv=9)rnd_search.fit(X,y)rnd_search.best_params_rnd_search.best_score_
Conclusion
So, Grid Search is good when we work with a small number of hyperparameters. However, if the number of parameters to consider is particularly high and the magnitudes of influence are imbalanced, the better choice is to use the Random Search.
Thanks for reading this. There are some other ways you can keep in touch with me and follow my work:
Subscribe to my newsletter.
You can also get in touch via my Telegram group, Data Science for Beginners.