Automated model selection and composite feature spaces in scikit-learn
By: Oliver Batey
Tags: Feature Engineering, Editors Pick, Pipeline, Model Selection, Scikit Learn

Automated model selection and composite feature spaces in scikit-learn

Sometimes there are hundreds, if not thousands, of possible configurations of a machine learning model, making it impossible to find the best one by hand, and making automation essential. This is especially true when working with composite feature spaces, where we want to apply different transformations to different features of the dataset. A great example is combining text documents with numerical data, however, I couldn't find much info on how to automatically model this type of feature space in scikit-learn.

Instead of manually transforming your text into a bag-of-words, and then manually appending some numerical columns onto it, you can work much more efficiently using scikit-learn pipelines. This post shows you how.

Using pipelines allows you to treat a sequence of transformation steps and an estimator (classifier or regressor) as a single model, called a composite estimator. Not only does this keep your code neat and prevent information leakage between your training and test sets, but it also allows you to treat your transformation steps as hyperparameters of your model, then optimise your model over the hyperparameter space with a grid search. This means you can experiment with different combinations of numerical features alongside your text data, as well as different text processing methods, and more, automatically.

In what follows you’ll see how to build a system that: takes a collection of labelled text documents as input; automatically generates some numerical features; transforms the different data types; passes the data to a classifier; then searches different combinations of features and transformations, to find the best performing model.

Modelling

I’m using the Spam SMS dataset, downloadable from the UCI machine learning repository here, which contains two columns: a column of SMS text documents, text, and a corresponding label column, containing the strings ‘spam’ and ‘ham’, which we have to predict. As always, the entire project code is available on GitHub.

The first step is to define the transformations that we want to apply to the dataset. To include a data transformation in a scikit-learn pipeline we must write it as a class, not an ordinary Python function; this may sound daunting initially but it's straightforward. One method is to simply define an ordinary Python function and pass it to the FunctionTransformer class, which turns it into a scikit-learn transformer object. Here, however, I will show you the more manual approach, so you can see what’s actually happening since I think it helps to understand how scikit-learn works.

You create a class that inherits from the BaseEstimator and TransformerMixin classes provided by scikit-learn, which provide the attributes and methods required to create an object that is compatible with scikit-learn pipelines. Then, you include what would be your function parameters as class attributes in the __init__() method, and override the transform() method with the body of the function you would use to transform your dataset. I’ve provided three examples below.

The custom transformer objects used in the pipeline. In every example, the fit() method does nothing, and all of the work is embodied in the transform() method.

The first two transformers are used to create new numerical features, here I'm choosing to use the number of words in the document, and the mean length of the words in the document, as features. Since our dataset contains only two columns, text and label, our text is stored as a pandas Series after separating the label column, which we should do at the very beginning of the project. As a result, CountWords.transform() is designed to accept a Series and return a DataFrame, since I will use this as the first transformer in our pipeline.

The final transformer FeatureSelector will allow us to treat various features as hyperparameters of our model. Its transform() method takes a list of column names and it will return a DataFrame containing only those columns; by passing it different lists of column names, we can search over different feature spaces to find the best one. These three transformers provide all the additional functionality we need to construct the pipeline.

Constructing the pipeline

The final pipeline is constructed out of three components: an initial Pipeline object, a ColumnTransformer object, and an estimator. The second component, ColumnTransformer is a convenient class introduced in version 0.20 which allows you to apply separate transformations to specified columns of your dataset. We will use it here to apply the CountVectorizer to the text column and apply another pipeline, num_pipeline, which contains FeatureSelector and scikit-learn’s SimpleImputer class, to the numerical columns. The structure of the entire pipeline is shown in this diagram:

Diagram of the pipeline. This entire object (called a composite estimator) can be used as a model; all the transformer and estimator objects, and their parameters, become hyperparameters of our model.

A Series of documents enters the pipeline, where CountWords and MeanWordLength create two numerical columns named n_words and mean_word_length. The text column is passed to CountVectorizer, while n_words and mean_word_length first pass through FeatureSelector and then SimpleImputer. The transformed dataset is passed to an estimator object.

The final pipeline is made up of an initial Pipeline object, a ColumnTransformer object, and an Estimator object. Note that the ColumnTransformer can apply entire pipelines to certain columns.

In the code example above, we are using the default parameters of CountVectorizer and SimpleImputer, keeping both numerical columns, and using a support vector classifier as an estimator. This final pipeline is our composite estimator, and every object inside of it, as well as the parameters of those objects, is a hyperparameter which we are free to vary. This means we can search different feature spaces, different vectorization settings, and different estimator objects.

Selecting the best model with a grid search

With the composite estimator set-up, it is very easy to find the best performing model; all you need to do is to create a dictionary, specifying which hyperparameters you want to vary and the values you’d like to test. You then pass this to scikit-learn’s GridSearchCV class which evaluates the model using cross-validation for each combination of hyperparameter values, then returns the best.

The parameter grid is defined as a dictionary, the keys are the hyperparameters and the values are lists of values that you want to search over. You then pass this to GridSearchCV, along with your composite estimator, and fit it to the training data.

Our composite estimator has a total of 65 tunable parameters, however, we are varying only two here: the numerical columns that are used and the max_features parameter of CountVectorizer, which sets the maximum number of words we use from the vocabulary. In the notebook, you can see how to get a list of all the available hyperparameters. Below is a visualisation of the mean balanced accuracy plotted over the hyperparameter space.

The best performance is achieved when we use only one numerical column, n_words, and use all the words in the vocabulary, that is when max_features = None. The balanced accuracy of this model is 0.94 during cross-validation and 0.93 when evaluated on the test set. Note that the exact numbers may change if you run the notebook yourself.

Balanced accuracy plotted across the hyperparameter grid, showing how the model performance varies across the hyperparameter space.

Conclusions

We’ve covered a lot, specifically, how we can automate our entire modelling process by setting up a composite estimator, which is a series of transformations and an estimator contained within a single pipeline. Not only is this good practice, but it is the only feasible way to search large hyperparameter spaces, which often arise when working with composite feature spaces. We saw an example of combining text data with numerical data, but it is easy to follow the same procedure with any data type, enabling you to work faster, more efficiently, and more effectively.