How to Build a Reusable Custom NLP Pipeline with Scikit-Learn
By: Jenny Lee
Tags: Towards Data Science, Feature Engineering, NLP, Data Science, Machine Learning

How to Build a Reusable Custom NLP Pipeline with Scikit-Learn

One of the most essential characteristics of a machine learning code pipeline is reusability. A reusable, sharable and extensible pipeline will ensure process and code integrity by enforcing consistent use of intuitive structural elements to program flows and can therefore enhance the data scientist’s development process, which is iterative by nature. In this article, I will demonstrate how to build a custom machine learning code pipeline from scratch using scikit-learn with an emphasis on the following two components:

A featurization pipeline that enables flexible definition and selection of features A training pipeline that incorporates the output of the featurization pipeline for subsequent steps: vectorization and model training

Both components exploit class inheritance in an analogous manner; that is, they are characterized by an underlying class structure from which more specific classes can be derived to address different kinds of supervised learning problems or project requirements. As a bonus of this article, in section 2, I will touch on a powerful concept related to inheritance, namely, abstract methods, and illustrate how it makes the proposed pipeline design so intuitive and easy to work with.

1. Featurization pipeline

A typical NLP task requires a great deal of feature engineering, a process that involves preparing the proper input data for training a model. This step may require transforming input text data into bag-of-words representations or multi-dimensional vectors, or it may involve writing more original user-defined functions to extract values for features like sentence length standard deviation, number of words before the main verb (a rough measure of syntactic complexity), readability score, etc. In any given project, any number of such features can be chosen to train a model or first determine feature importances, reduce the dimensionality, or conduct an ablation study.

To allow for ease, control and flexibility in selecting or testing features, you can imagine having a kind of “feature storehouse” from which you can draw any features you (or your teammate) have ever written, just when you need them. That storehouse can be represented as a class called BaseFeaturizer below, and each of its methods contains the logic for computing the value of a feature.

Once we have defined these basic methods in BaseFeaturizer , we can derive new classes to modify any of those methods or add new ones. But perhaps the most important property of these derived classes is that they can vary according to the number and types of features they select/introduce. And that information should be defined in a consistent way across all derived classes, i.e., in the main method of each derived class, which I will name featurize .

For example, let’s say we want to train a text classification model that predicts the syntactic complexity of documents, classifying them into two discrete classes: “simple” and “complex”. Two potentially useful syntactic features for this task are:

Number of words before the main verb: the longer the text before the main verb, the more syntactically complex the text (presumably due to the presence of clausal and/or subject modifiers such as a subordinate clause) Readability: the higher the readability score, the more complex. (For example, read about Flesch–Kincaid readability tests here.)

(I have somewhat arbitrarily chosen these features for this illustration, but there are various feature selection techniques one may utilize to first determine which features are the most useful and include only those in the final training.)

Below, SyntacticComplexityFeaturizer is defined as a subclass of BaseFeaturier , and the values of the two features are computed inside featurize by calling the functions defined earlier in BaseFeaturizer . These values are stored in feature_dict for each document, and featurize returns the feature-value mappings for the entire corpus of documents.

Here’s an example of feature_dicts , the output of calling featurize on an instance of SyntacticComplexityFeaturizer .

2.Training Pipeline

The output of this featurization pipeline naturally serves as input to the next component, the training pipeline. The two pipelines are isomorphic in form, as the latter, too, contains a base design (class) from which more specific training pipelines can be derived to meet different training requirements. I have named this class BaseTrainingPipeline . An important property of this class is that it inherits from ABC (short for “Abstract Base Class”) for reasons that will become clear soon.

BaseTrainingPipeline can be initialized with any attributes that you may want to access later, for example, self.y_test and self.y_preds (these values are good to store for evaluation purposes), as well as self.models (all models trained on the same data). I have defined the following attributes in the __init__ function:

Next, the class’s main method, pipeline_main , contains the minimal code required to transform input text first into numeric features and then into vectors— steps that are repeated in most supervised NLP tasks. Note that the two pipelines (featurization and training) are connected through this method as the method takes FeaturizerClass as an argument, for example, SyntacticComplexityClass.

Once instantiated ( featurizer = FeaturizerClass() ), the object will call the featurize method on the data (corpus), returning a list of feature-value mappings (lines 22–23). This list can then be transformed into numpy arrays or matrices (vectors) using DictVectorizer (lines 31–33). The vectors X_train and y_train can then be used to train any scikit-learn estimator (line 39–42).

To elaborate on an earlier point, train_model is defined as an abstract method of BaseTrainingPipeline (which makes it an abstract class). Abstract methods can be decorated with the decorator @abstractmethod , the relevant portion reproduced below:

You can read more about abstract methods here, but in a nutshell:

Abstract classes are classes that contain one or more abstract methods. An abstract method is a method that is declared, but contains no implementation. Abstract classes may not be instantiated, and require subclasses to provide implementations for the abstract methods. Subclasses of an abstract class in Python are not required to implement abstract methods of the parent class.

Defining train_model as an abstract method is necessary because it provides a common functionality for all implementations of that method which are supplied by subclasses of BaseTrainingPipeline . That is, because we define train_model as a method of the base class, we can call it from another method defined in the same class (in our case, from pipeline_main ), yet because its implementation can only be provided by a subclass, we are able to define train_model in a number of different ways. One dimension along which these implementations can vary is estimator type.

To make all this more concrete, consider two example training pipelines I have derived from BaseTrainingPipeline : ClassificationTrainingPipeline and RegressionTrainingPipeline . This separation makes intuitive sense as one deals with classifiers whose target variable is categorical/discrete while the other deals with regressors whose target variable is numerical/continuous. This allows us to support different types of supervised learning problems using the same general (training) pipeline without rewriting any part of the core elements and repeatable steps defined in pipeline_main above. Note how the implementation of train_model differs between the two cases: In ClassificationTrainingPipeline , train_model specifies new parameter settings for a model that meets a certain condition, i.e., if algorithm == "logistic_regression" (lines 12–19 below), overriding the original settings of the initialized model. RegressionTrainingPipeline does not contain such overriding settings, though it certainly may for any of its initialized models.

Putting it all together, we can now instantiate one of these training pipelines, which takes a certain featurization class as an argument: