An Introduction to Building Pipelines and Using Grid Searches in Scikit-learn
By: Apratim Biswas
Tags: Gridsearchcv, Grid Search, Machine Learning, Pipeline, Scikit Learn

Building even the most basic machine learning model involves several steps. Features have to be selected, data needs to be standardized, the type of estimator to be used has to be determined and then fitted to training data. Once we have a working model, the next step is finding and optimizing parameters.

Models involve two types of parameters:

i) model parameters: configuration variables internal to the model and which can be estimated from data; and,

ii) model hyperparameters: configuration variables external to the model and cannot be estimated from data.[1]

Throughout the model building process there are many steps where it is necessary to provide hyperparameters. Such parameters must be supplied externally and adjusting them is a large part of developing a model. Often referred to as hyperparameter tuning, this involves the data scientist optimizing such parameters for performance.

Building a model is an iterative process. And as one would imagine, the process can easily become tedious, unwieldy and error-prone. Thankfully, Scikit-learn has a great set of tools meant to address exactly this: pipeline and gridsearch. The goal of this article is to demonstrate the usage of these tools. Before we go any further, it’s important to mention that the rest of the article is not a exercise in regression analysis. The analytical work here is simply a vehicle to demonstrate the Scikit-learn tools. With that out of the way, let’s get started.

As always we start by importing the necessary libraries. In our model we’ll build a pipeline, and for that we need the make_pipeline method from the sklearn.pipeline module in Scikit-learn. GridSearchCV , another method we’ll use, comes from the model_selection module ofthe same Scikit-learn library. We will use the Penguins dataset that ships with Seaborn.

#import libraries

import numpy as np

import pandas as pd

import seaborn as sns;

import matplotlib.pyplot as plt

from sklearn.preprocessing import StandardScaler

from sklearn.pipeline import make_pipeline

from sklearn.feature_selection import SelectKBest, f_regression

from sklearn.model_selection import train_test_split,GridSearchCV

from sklearn.linear_model import Ridge, Lasso #loading dataset

penguins=sns.load_dataset('penguins') #looking at a snapshot of newly loaded data

penguins.info() <class 'pandas.core.frame.DataFrame'>

RangeIndex: 344 entries, 0 to 343

Data columns (total 7 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 species 344 non-null object

1 island 344 non-null object

2 bill_length_mm 342 non-null float64

3 bill_depth_mm 342 non-null float64

4 flipper_length_mm 342 non-null float64

5 body_mass_g 342 non-null float64

6 sex 333 non-null object

dtypes: float64(4), object(3)

memory usage: 18.9+ KB

To illustrate the process of building a pipeline, I’ll use regression to estimate body mass, which is a continuous numeric variable. Let’s choose 3 columns with numerical data ( bill length , bill depth , flipper length ) and 2 with categorical data ( sex , species ) as features. The sex column has some rows with null values and we'll drop them.

#dropping rows with NA or NaN values

penguins.dropna(inplace=True)

Let’s create a feature matrix (X) and a vector with target variable (y). Since sex is a categorical variable, we'll also need to dummify them before splitting the matrices into training and testing sets.

#creating a feature matrix X

X=penguins.drop(['island','body_mass_g'],axis=1) #Quick look at columns with categorical variable

X[['species','sex']]

#Let's pass 'species' column to OneHotEncoder

X=pd.get_dummies(data=X,columns=['species','sex'],drop_first=True) y=penguins['body_mass_g'] #Split data into training and test sets

X_train,X_test,y_train,y_test=train_test_split(X,y) #taking a quick look at the newly created training set

X_train.head(2)

We are finally ready to set-up a pipeline and build a model. The pipeline module in Scikit-learn has a make-pipeline method. The first step is to instantiate the method. We do this by passing it the steps we want our input data to go through, in order. An instantiated pipeline works just like any other Scikit-learn estimator. Code below is an example of building a new pipeline. We name the newly created pipeline with a not-so-creative name: pipe.

#Setting up a pipeline

pipe=make_pipeline(StandardScaler(),SelectKBest(f_regression),Ridge())

The methods specified as arguments to make_pipeline , from left to right are:

i) StandardScaler() -> We have not standardized the data before train_test_split . So incoming data needs to standardized before any other transformation is performed on it.

ii) SelectKBest() -> This method comes from the feature_selection module of Scikit-learn. It selects the best features based on a specified scoring function (in this case, f_regression) . The number of features is specified by the value of parameter k . Even within the selected features, we want to vary the final set of features that are fed to the model, and find what performs best. We can do that with the GridSearchCV method, which I’ll come back to shortly.

iii) Ridge() -> This is an estimator that performs the actual regression. The name of the method refers to Tikhonov regularization, more commonly known as ridge regression, that is performed to reduce the effect of multicollinearity. Like the parameter k discussed previously, we want to test several different values for the various parameters of ridge regression. We do that as part of a grid search, which we discuss next.

Our pipeline is now ready to be fitted. As I mentioned previously, an instantiated pipeline acts just like any other estimator. It can accept arguments for every method that ends up being a part of the pipeline. A quick way to get a list of parameters that a pipeline can accept is show below:

#Looking up parameters that can be passed to the pipeline

pipe.get_params().keys() Output:

dict_keys(['memory', 'steps', 'verbose', 'standardscaler', 'selectkbest', 'ridge', 'standardscaler__copy', 'standardscaler__with_mean', 'standardscaler__with_std', 'selectkbest__k', 'selectkbest__score_func', 'ridge__alpha', 'ridge__copy_X', 'ridge__fit_intercept', 'ridge__max_iter', 'ridge__normalize', 'ridge__random_state', 'ridge__solver', 'ridge__tol'])

Grid search gives us the ability to search over specified values for each of the parameters listed above. We do this by passing GridSearchCV a dictionary with parameter names as keys, and lists of values to try as arguments for those parameters. In this example I call this dictionary params and pass it to GridSearchCV . Once fitted, the GridSearchCV instance gs , acts just like any other estimator.

As a side note, I want to highlight an optional, but very useful parameter: n_jobs. It tells Scikit-learn how many jobs to run in parallel. Setting it to -1 is equivalent to instructing Scikit-learn to use all available processors. Nowadays, most CPUs have more than one core. If you have a quad-core processor, using all 4 cores instead of 1 can make the process significantly faster.

#putting together a parameter grid to search over using grid search

params={

'selectkbest__k':[1,2,3,4,5,6],

'ridge__fit_intercept':[True,False],

'ridge__alpha':[5,10],

'ridge__solver':[ 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag',

'saga']

} #setting up the grid search

gs=GridSearchCV(pipe,params,n_jobs=-1,cv=5) #fitting gs to training data

gs.fit(X_train, y_train)

Other than searching over all the permutations of the selected parameters, GridSearchCV performs cross-validation on training data. The default is 5 folds, but we can specify any other number using the parameter cv . The attribute cv_results_ includes detailed results for each cross-validation run and provides a wealth of data that can be used to determine the fit and robustness of the newly developed model. Details of the permutation of parameters that performs the best is provided by the best_params_ attribute.

#building a dataframe from cross-validation data

df_cv_scores=pd.DataFrame(gs.cv_results_).sort_values(by='rank_test_score') #selecting specific columns to create a view

df_cv_scores[['params','split0_test_score', 'split1_test_score', 'split2_test_score',\

'split3_test_score', 'split4_test_score', 'mean_test_score',\

'std_test_score', 'rank_test_score']].head()

#checking the selected permutation of parameters

gs.best_params_ Output:

{'ridge__alpha': 5,

'ridge__fit_intercept': True,

'ridge__solver': 'sag',

'selectkbest__k': 6}

Finally, we can predict target values for test set by passing it’s feature matrix to gs . The predicted values can be compared with actual target values to visualize and communicate performance of the model.

#checking how well the model does on the holdout-set

gs.score(X_test,y_test) Output:

0.8707008994531131 #plotting predicted body weights vs actual body weights of penguins

y_preds=gs.predict(X_test)

plt.scatter(y_test,y_preds);

Predicted vs actual body mass of penguins (g)

This is a very simple model. But even at this level of simplicity, it’s very evident how effective pipelines and grid-searches can be. Perhaps, what’s not so obvious is the value of critical thinking that goes into building a pipeline and putting it in a grid-search. An example of this is feature selection. Say we want to choose k features in a classification problem. Outside of a grid-search, we can choose k features randomly, without giving much thought to the process. However, performing feature selection inside a grid-search forces one to think of available approaches/algorithms, problems where they are applicable, and their strengths and weakness. So arguably, using pipelines and grid-searches can make one a better data scientist. A caveat with these techniques is that the number of models to be built increases multiplicatively with every additional parameter value. So instead of grid-searching over a large array of possible parameter values, it’s more practical to search over a small number of values for each parameter, each time, and then shift the range and re-run the grid-search. Once in the desired range, we can just zoom-in to find the best values. Used correctly, they are true “lifesavers” when dealing with unknown complex systems. If you haven’t been using them already, I highly recommend them.