How to tune multiple ML models with GridSearchCV at once? | by Satyam Kumar | Towards Data Science

Model selection is an essential component for a data science model development pipeline. After performing feature engineering the data scientist needs to choose the model with the best set of hyperparameters that performs best for the training dataset. There are various Auto-ML libraries that automate the model selection component.
Grid Search is a cross-validation technique that can be used to tune the hyperparameters of a machine learning model. To choose the robust model from a list of ML models, one needs to perform cross-validation multiple times for each model. In this article, we will discuss a hack or technique to tune multiple models simultaneously using Grid Search and Randomized Searchcross-validation.
GridSearchCV or RandomizedSearchCV are cross-validation techniques to perform hyperparameter tuning to determine the optimal values for a machine learning model. GridSearchCV ideally tries all the possible values of the parameters while RandomizedSearchCV randomly picks the parameters and speeds up the cross-validation workflow.
The above-mentioned code snippet can be used to select the best set of hyperparameters for the random forest classifier model.
Ideally, GridSearchCV or RandomizedSearchCV need to run multiple pipelines for multiple machine learning models, to pick the best model with the best set of hyperparameter values. A data scientist might take a lot of time to develop the code and work through it.
You can tune multiple machine learning models simultaneously with GridSearchCV or RandomizedSearchCV by creating multiple parameters dictionaries and specifying models for each dictionary. The step by step approaches to tune multiple models at once are:
Initialize multiple classifier estimatorsPrepare a pipeline of the 1st classifier.Prepare hyperparameter dictionary of each estimator each having a key as ‘classifier’ and value as estimator object. The hyperparameter keys should start with the word of the classifier separated by ‘__’ (double underscore).List the hyperparameter dictionary.Train a GridSearchCV model with the pipeline and parameter dictionary list.
Initialize multiple classifier estimators
Prepare a pipeline of the 1st classifier.
Prepare hyperparameter dictionary of each estimator each having a key as ‘classifier’ and value as estimator object. The hyperparameter keys should start with the word of the classifier separated by ‘__’ (double underscore).
List the hyperparameter dictionary.
Train a GridSearchCV model with the pipeline and parameter dictionary list.
# Import necessary packages
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.naive_bayes import MultinomialNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import *

# Read the dataset
df = pd.read_csv("DATA/creditcard.csv")
df = df.drop('Time', axis=1)
print(df.shape)
df.head()

(284807, 30)

5 rows × 30 columns
# Dataset sampling for faster computation
pos_idx = list(df[df['Class']==1].index)
neg_idx = list(df[df['Class']==0].sample(5000).index)

df = df.loc[pos_idx+neg_idx]
print(df.shape)
df['Class'].value_counts()

(5492, 30)

0    5000
1     492
Name: Class, dtype: int64
# Train Test Split
y = df['Class']
X = df.drop('Class', axis=1)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
print(X_train.shape)
print(X_test.shape)

(4393, 29)
(1099, 29)

# Initialze the estimators
clf1 = RandomForestClassifier(random_state=42)
clf2 = SVC(probability=True, random_state=42)
clf3 = LogisticRegression(random_state=42)
clf4 = DecisionTreeClassifier(random_state=42)
clf5 = KNeighborsClassifier()
clf6 = MultinomialNB()
clf7 = GradientBoostingClassifier(random_state=42)

# Initiaze the hyperparameters for each dictionary
param1 = {}
param1['classifier__n_estimators'] = [10, 50, 100, 250]
param1['classifier__max_depth'] = [5, 10, 20]
param1['classifier__class_weight'] = [None, {0:1,1:5}, {0:1,1:10}, {0:1,1:25}]
param1['classifier'] = [clf1]

param2 = {}
param2['classifier__C'] = [10**-2, 10**-1, 10**0, 10**1, 10**2]
param2['classifier__class_weight'] = [None, {0:1,1:5}, {0:1,1:10}, {0:1,1:25}]
param2['classifier'] = [clf2]

param3 = {}
param3['classifier__C'] = [10**-2, 10**-1, 10**0, 10**1, 10**2]
param3['classifier__penalty'] = ['l1', 'l2']
param3['classifier__class_weight'] = [None, {0:1,1:5}, {0:1,1:10}, {0:1,1:25}]
param3['classifier'] = [clf3]

param4 = {}
param4['classifier__max_depth'] = [5,10,25,None]
param4['classifier__min_samples_split'] = [2,5,10]
param4['classifier__class_weight'] = [None, {0:1,1:5}, {0:1,1:10}, {0:1,1:25}]
param4['classifier'] = [clf4]

param5 = {}
param5['classifier__n_neighbors'] = [2,5,10,25,50]
param5['classifier'] = [clf5]

param6 = {}
param6['classifier__alpha'] = [10**0, 10**1, 10**2]
param6['classifier'] = [clf6]

param7 = {}
param7['classifier__n_estimators'] = [10, 50, 100, 250]
param7['classifier__max_depth'] = [5, 10, 20]
param7['classifier'] = [clf7]

pipeline = Pipeline([('classifier', clf1)])
params = [param1, param2, param3, param4, param5, param6, param7]

%%time
# Train the grid search model
gs = GridSearchCV(pipeline, params, cv=3, n_jobs=-1, scoring='roc_auc').fit(X_train, y_train)

# Best performing model and its corresponding hyperparameters
gs.best_params_

{'classifier': SVC(C=10, probability=True, random_state=42),
 'classifier__C': 10,
 'classifier__class_weight': None}
# ROC-AUC score for the best model
gs.best_score_

0.9838906204691353
# Test data performance
print("Test Precision:",precision_score(gs.predict(X_test), y_test))
print("Test Recall:",recall_score(gs.predict(X_test), y_test))
print("Test ROC AUC Score:",roc_auc_score(gs.predict(X_test), y_test))

Test Precision: 0.7142857142857143
Test Recall: 0.9859154929577465
Test ROC AUC Score: 0.9793390694360716

%%time
# Train the random search model
rs = RandomizedSearchCV(pipeline, params, cv=3, n_jobs=-1, scoring='roc_auc').fit(X_train, y_train)

# Best performing model and its corresponding hyperparameters
gs.best_params_

{'classifier': SVC(C=10, probability=True, random_state=42),
 'classifier__C': 10,
 'classifier__class_weight': None}
# ROC-AUC score for the best model
gs.best_score_

0.9838906204691353
# Test data performance
print("Precision:",precision_score(rs.predict(X_test), y_test))
print("Recall:",recall_score(rs.predict(X_test), y_test))
print("ROC AUC Score:",roc_auc_score(rs.predict(X_test), y_test))

Precision: 0.8571428571428571
Recall: 0.9882352941176471
ROC AUC Score: 0.9872142940016244

 

I have used a sample binary classification dataset having ~5000 instances and 29 independent features. To train a robust machine learning model, I have initialized 7 machine learning estimators including:
Random Forest
Support Vector Classifier
Logistic Regression
Decision Tree
k-Nearest Neighbours
Naive Bayes
Gradient Boosting
and further prepared their corresponding hyperparameters dictionary with the variable name param_i.
Scikit-learn package comes with the GridSearchCV implementation. I have passed the 1st estimator with the pipeline and the parameter dictionary list to the GridSearchCV model.
The GridSearchCV takes 120 secs to train 176 models for 7 estimators. The Support Vector Classifier with C=10, class_weight=None performs the best with a cross-validation ROC AUC score of 0.984 and test ROC AUC score of 0.979.
The RandomizedSearchCV model takes 9 secs to train the same set of models. The same Support Vector Classifier resulted to perform the best.
In this article, we have discussed how GridSearchCV and RandomizedSearchCV can be used to tune multiple machine learning models simultaneously. This is a very handy technique and speeds up the model selection workflow.
[1] Scikit-learn documentation: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
Loved the article? Become a Medium member to continue learning without limits. I’ll receive a small portion of your membership fee if you use the following link, with no extra cost to you.
satyam-kumar.medium.com
Thank You for Reading