ML Pipeline with Grid Search in Scikit-Learn | by Benjamin Wang | Towards Data Science

Let’s start by outlining some main benefits of using ML pipeline, already.
Unified API for both data transformations and modelling
No extra preprocessing steps needed during model serving time
Joint grid search possible including data preprocessing steps/parameters and model selection/parameters
Compatible with big data framework such as Spark
The last point might not be immediately obvious for some. Imagine if one day your model needs to train on large amount of data. You might have no better choice but to use some distributed framework like Spark. Using ML pipeline will make migrating to Spark a breeze to work with. One reason is that Spark MLlib has a similar ML pipeline feature. Another more fundamental reason is functional programming.
ML pipeline draws many similarities with functional programming (which Spark also follows), in that it treats transformations and estimators(models) as operators on data. Data maintains immutability (a core idea of FP), as they just pass through different components of the pipeline, get transformed into new format during the process. At the end of the pipeline, we get some prediction results out.
In the example pipeline, we have a preprocessor step, which is of type ColumnTransformer, containing two sub-pipelines:
num_transform is a sub-pipeline intended for numeric columns, which fills null values and convert the column to a standard distribution
cat_transform is a another sub-pipeline intended for categorical columns, which does one-hot encoding
Following the preprocessor transformer, we have an estimator , a default logistic regressor in this case.
With this pipeline, one can combine data preprocessing together with modelling, and even include more complex feature engineering by creating custom transformers.
To create your own transformer, you need to create a class which inherits from both sklearn.base.BaseEstimator and sklearn.base.TransformerMixin and implement fit and transform methods.
Let’s go through a running example (the complete code is available on Github) with the Titanic dataset containing features of passengers and labels indicating if one has survived.
We want to build a classifier to predict if a given passenger will survive.
from sklearn.datasets import fetch_openmlimport numpy as npnp.random.seed(0)# Load data from https://www.openml.org/d/40945X, y = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)
We will consider the following 4 features, of which age and fare are numeric and sex and pclass are categorical (note: one can also treat pclass as numeric, but that’s not relevant for discussion here).
Now, let’s build the pipeline to do some data preprocessing as indicated previously.
numeric_features = ["age", "fare"]numeric_transformer = Pipeline(steps=[    ("imputer", SimpleImputer(strategy="mean")),    ("scaler", StandardScaler())])categorical_features = ["sex", "pclass"]categorical_transformer = OneHotEncoder(handle_unknown="ignore")preprocessor = ColumnTransformer(transformers=[    ("num_transform", numeric_transformer, numeric_features),    ("cat_transform", categorical_transformer, categorical_features)])pipeline = Pipeline(steps=[("preprocesser", preprocessor), ("classifier", LogisticRegression())])pipeline
Alternatively, there is a slightly more compact API make_pipeline and make_column_transformer where we don’t need to pass names of each step explicitly. However, the following section will be based on the previous verbose version, because we like to name things as we like :).
from sklearn.compose import make_column_transformerfrom sklearn.pipeline import make_pipelinenumeric_transformer = make_pipeline(    SimpleImputer(strategy='mean'),    StandardScaler())categorical_transformer = make_pipeline(OneHotEncoder(handle_unknown='ignore'))preprocessor = make_column_transformer(    (numeric_transformer, numeric_features),    (categorical_transformer, categorical_features))pipeline = make_pipeline(preprocessor, LogisticRegression())pipeline
One handy tip, one can visualize the pipeline in Jupyter notebook by including the code below. It creates a nice diagram as shown before.
from sklearn import set_configset_config(display='diagram')pipeline
Training a model is no different than a pure estimator. You just call fit. Below we first do a a standard train test data split to test model performance. Then, you just call pipeline.fit() on your training data, that’s it.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)pipeline.fit(X_train, y_train)score = pipeline.score(X_test, y_test)print(score)0.7977099236641222
What really happens under the hood is that the training data pass through the steps defined in the pipeline (including the classifier estimator) as if it is one model.
At this point, one might wonder can I also save my pipelined model as I used to do with a standalone model? The answer is a definite yes.
And there is a bonus point of putting the data processing and modelling together in one saved model. Now you don’t need to think about what preprocessing steps to take during prediction time, they are already an integral part of the model. This prevents strange prediction results from happening in production.
import joblibjoblib.dump(pipeline, 'your_pipeline.pkl')joblib.load('your_pipeline.pkl').score(X_test, y_test)0.7977099236641222
We use joblib to save and load the pipeline, and it produces the same score, which verifies the correctness, and indeed we see no preprocessing is needed, as promised.
ML pipeline enables us not only to search parameters for the estimator, but also parameters for all the preprocessing steps. In other words, we can now tune ML models, and preprocessing options all together.
For argument sake, say, we are not sure if filling null values by mean ormedian strategy is a better option, and we are also not sure about how much regularization is needed for the LogisticRegressor . Therefore, we want to find out which is the best missing value filling strategy and the best regularization weight at the same time. To achieve that, we can construct the paramter grid as follows:
param_grid = [    {        "preprocesser__num_transform__imputer__strategy": ["mean", "median"],        "classifier__C": [0.1, 1.0, 10.0, 100.0],        "classifier": [LogisticRegression()]    }]
Note: sklearn uses double underscore as separator. classifier key is the same as the pipeline name for estimator in the pipeline definition, classifier__C basically tells the grid searcher that we would like to try these provided values for C which is a parameter for the classifier LogisticRegressor to define regularization weight.
As you might notice, you can also concat the double underscore syntax. preprocesser__num_transform__imputer_strategy defines different values to try out for the imputer step’s strategy parameter.
Finally, we can start the grid search, since we have 2 values for strategy and 4 values for C , in total there are 2*4=8 candidates to in the search space.
grid_search = GridSearchCV(model, param_grid, cv=10, verbose=1,n_jobs=-1)grid_search.fit(X_train, y_train)
The output is shown below, since we have a 10 fold cross validation for each combination, in total we need to fit 80 models! One can see how easily grid search can explode.
Fitting 10 folds for each of 8 candidates, totalling 80 fits[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.[Parallel(n_jobs=-1)]: Done  80 out of  80 | elapsed:    4.5s finished
What if we want to try out different classifiers? We can construct more complex parameter grid to achieve this.
param_grid = [    {        "preprocesser__num_transform__imputer__strategy": ["mean", "median"],        "classifier__C": [0.1, 1.0, 10.0, 100.0],        "classifier": [LogisticRegression()]    },    {        "preprocesser__num_transform__imputer__strategy": ["mean", "median"],        "classifier__n_estimators": [10, 100, 1000],        "classifier": [RandomForestClassifier()]    }]grid_search = GridSearchCV(model, param_grid, cv=10, verbose=1,n_jobs=-1)grid_search.fit(X_train, y_train)
In the code above, we use two dictionaries in the parameter grid to add an extra set of search options, which uses a random forest classifier instead of a logistic regressor.
One might ask, what is the number of combinations now? Since each dictionary defines a different set, we should add up across dictionaries, multiply within dictionaries. That gives us 2*4*1 + 2*3*1 = 14 candidates.
This is verified in the output:
Fitting 10 folds for each of 14 candidates, totalling 140 fits[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.[Parallel(n_jobs=-1)]: Done  76 tasks      | elapsed:    1.5s[Parallel(n_jobs=-1)]: Done 140 out of 140 | elapsed:   26.6s finished
We have discussed the main benefits of using ML pipeline with a running example.
We have also seen how convenient the pipeline concept can be, and powerful at the same time. The latter point becomes evident after we see how easy it is to do a joint grid search.
Last but not least, it should not be a surprise to see that Spark MLlib has almost identical features as ML pipeline in scikit-learn , interested readers can checkout: https://spark.apache.org/docs/latest/ml-pipeline.html, or I might write another blog on that.
Code available on GitHub: https://github.com/wlongxiang/mlpipeline/blob/main/ml_pipeline_with_grid_search.ipynbSome materials to read for making your own transformers: https://sklearn-template.readthedocs.io/en/latest/user_guide.html
Code available on GitHub: https://github.com/wlongxiang/mlpipeline/blob/main/ml_pipeline_with_grid_search.ipynb
Some materials to read for making your own transformers: https://sklearn-template.readthedocs.io/en/latest/user_guide.html