Introducing Skippa. SciKIt-learn Pre-processing Pipelines... | by Robert van Straalen | Towards Data Science

Any data scientist is probably familiar with pandas and scikit-learn. The usual workflow entails data cleaning in pandas, further pre-processing using either pandas again or scikit-learn transformers like StandardScaler, OneHotEncoder etc., after which a machine learning algorithm is fitted (something from scikit-learn again).
Now there are some problems with this approach:
1. The development workflow is rather complex and requires a lot of code 😰
2. It’s hard to reproduce the workflow for prediction in a deployment 😩
3. Existing solutions to mitigate these issues are simply not good enough 😞
Skippa is a package designed to:
✨ drastically simplify development
📦 package / serialize all data cleaning, pre-processing together with your model algorithm into a single pipeline file
😌 reuse the interface/components from pandas & scikit-learn that you’re already familiar with
Skippa helps you to easily define data cleaning & pre-processing operations on a pandas DataFrame and combine it with a scikit-learn model/algorithm into a single executable pipeline. It works roughly like this:
from skippa import Skippa, columnsfrom sklearn.linear_model import LogisticRegressionX, y = get_training_data(...)pipeline = (    Skippa()    .impute(columns(dtype_include='object'), strategy='most_frequent')    .impute(columns(dtype_include='number'), strategy='median')    .scale(columns(dtype_include='number'), type='standard')    .onehot(columns(['category1', 'category2']))    .model(LogisticRegression()))pipeline.fit(X, y)predictions = pipeline.predict_proba(X)
☝️ Skippa does not presume to fix all problems, does not cover all functionality you might ever need and is not a highly scalable solution, but it should be able to offer a massive simplification for > 80% of regular pandas/sklearn based machine learning projects.
📂 Github
📦 pypi
📝 readthedocs
⏩ skip the essay and go directly to the fun part
For tasks that are conceptually quite simple — imputing missing values, one-hot encoding categorical values, selecting a subset of columns, scaling etc. — you need quite a lot of code. If generally you start out with a pandasdataframe consisting of columns of various data types, you require different transformations of different columns and a lot of your code involves selecting column subsets and merging transformed data.
What you want is basically a set of steps where you say: I want to apply this transformation to these columns. If only there was a package that enabled you to code in such idioms...
If you want to deploy your model, you need to be able to reproduce your pre-processing code for use in a prediction context. Also, besides your pickled model file, you need pickled files for your fitted scaler, fitted one-hot encoder(s) and possible other transformations, since they are needed to generate predictions on new data! This quickly becomes hard to maintain. If only you could save all this logic into a single file...
Now, a lot of work has already been done on this subject. In scikit-learn you have
the Pipeline which you can use to chain certain transformers to your algorithm
and the ColumnTransformer which you can use to define pipelines of transformers on different sets of columns.
I find this cumbersome to work with though. Look at this example:
import numpy as npimport pandas as pdfrom sklearn.compose import ColumnTransformerfrom sklearn.impute import SimpleImputerfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import OneHotEncoderfrom sklearn.tree import DecisionTreeClassifier# this is the input dataframedf = pd.DataFrame({    'favorite_color':['blue','green','red','green','blue'],    'age': [10,15,10,np.nan,10],    'target':[1,0,1,0,1]})# define individual transformers in a pipelinecategorical_preprocessing = Pipeline([('ohe', OneHotEncoder())])numerical_preprocessing = Pipeline([('imputation', SimpleImputer())])# define which transformer applies to which columnspreprocess = ColumnTransformer([    ('categorical_preprocessing', categorical_preprocessing, ['favorite_color']),    ('numerical_preprocessing', numerical_preprocessing, ['age'])])# create the final pipeline with preprocessing steps and # the final classifier steppipeline = Pipeline([    ('preprocess', preprocess),    ('clf', DecisionTreeClassifier())])# now fit the pipeline using the whole dataframedf_features = df[['favorite_color','age']]df_target = df['target']# call fit on the dataframespipeline.fit(df_features, df_target)
Now this does work, but
it’s rather complex and cumbersome to work with
you run into limitations if you need serial transformations on various sets of columns, because the output of each transformer is a numpy array, rather than a pandas dataframe
There is a package sklearn-pandas which is a noble effort to remedy these issues. While it does solve some problems, I’m not satisfied with it.
The most important design principle of skippa is that the output of a transformation is always a pandas DataFrame. Why? Because conceptually you start out with a DataFrame, consisting of different columns of different data types, requiring different transformations. Using sklearn, the output of a transformation is a numpy array. This generally means you lose column names and data types (everything becomes float). Conceptually, when chaining together a pipeline, you want to keep referring to columns using their names and/or data types. This is only possible when the DataFrame is consistently used throughout the pipeline as the central data representation.
Skippa achieves this result by implementing customized versions of existing transformations. Now, there is of course a drawback to this approach: those transformations cannot be used directly and a specific wrapper needs to be implemented. This always means a development cost and a maintainability cost. I hope this cost is outweighed by the benefits offered by Skippa.
Other design considerations:
Skippa aims to have a syntax that is the shortest and simplest way possible to define your transformations.
Skippa doesn’t support all functionality you might ever need. The intent is to provide added value for the large majority of use cases, not for all use cases. But there’s still a fallback way to define arbitrary transformations!
Yes, you need to learn a slightly new ‘language’ (i.e. the Skippa methods), but I feel this is quite straightforward and very easy to learn for anyone familiar with pandas and scikit-learn.
So, after all this complaining and being a smart ass claiming to know better, let’s show what Skippa can do!
Installation is just via pypi:
$ pip install skippa
Import the Skippa object and the columns helper function:
from skippa import Skippa, columns
The idea is that you start out with a pandas DataFrame. You then initiate a Skippa pipeline and then use chaining commands to define transformations that need to be applied to certain columns. The last step is generally a model estimator (e.g. a classifier, regressor, clusterer) from scikit-learn.
X, y = get_training_data(...)pipeline = (    Skippa()    .<transformation-1>(<column-definition>, <optional-args>)    .<transformation-2>(<column-definition>, <optional-args>)    ...    .model(<model-estimator>(<optional-args>)))pipeline.fit(X, y)pipeline.save('./models/mypipeline')
Ok, now a specific example. We have a pandas DataFrame X with 10 columns:
4 float columns, 2 integer columns, 3 categorical columns and 1 date column
We have missing values in float columns and in a categorical column
and we have a set of labels y (either 0 or 1)
What we want to do is the following
remove a bunch of columns we don’t need
impute missing values (median value for numeric columns, most frequent value for categorical columns)
one-hot encode categorical columns
scale the numerical columns
fit a logistic regression model
We can do this in a couple of lines:
pipe = (    Skippa()    .select(columns(exclude=['a', 'f', 'i', 'j']))    .impute(columns(dtype_include='number'), strategy='median')    .impute(columns(dtype_include='object'), strategy='most_frequent')    .scale(columns(dtype_include='number'), type='standard')    .onehot(columns(['g', 'h']))    .model(LogisticRegression()))pipe.fit(X, y)pipeline.save('./models/mypipeline')
The file we saved is a SkippaPipeline which is a child class of scikit-learn’s Pipeline. It contains the entire pipeline of the pre-processing and the logistic regression. It’s easy to deploy and reuse:
df_scoring = get_scoring_data(...)my_pipeline = Skippa.load_pipeline('./models/mypipeline')predictions = my_pipeline.predict_proba(df_scoring)
Since the Skippa pipeline is (extends) a regular scikit-learn Pipeline object, you can use all the standard methods like .fit, .transform, .predict, .predict_proba etc.
It’s entirely feasible that you’re just interested in processing data, without the wish to finish your pipeline with a modeling algorithm. Or maybe you simply want to inspect the final dataset that goes into your algorithm. That’s possible using the .build() method:
pipe = (    Skippa()    .select(columns(exclude=['a', 'f', 'i', 'j']))    .impute(columns(dtype_include='number'), strategy='median')    .impute(columns(dtype_include='object'), strategy='most_frequent')    .scale(columns(dtype_include='number'), type='standard')    .onehot(columns(['g', 'h']))    .build())df_processed = pipe.fit_transform(X)df_processed.info()
Defining a subset of columns for a transformation is done using the columns() function. There are several ways to use this. E.g. including or excluding a list of column names:
columns(['a', 'b']) simply define a list of column names
columns(include=['a', 'b']) this is equivalent
columns(exclude=['a', 'b']) analogously, you can de-select columns
You can also use dtype_include, dtype_exclude and pattern from sklearn’s make_column_selector() function.
columns(dtype_include='float') selects all float columns
columns(dtype_exclude='number') deselect all numeric columns
columns(pattern='myprefix_*') selects all columns starting with myprefix_
You can also add / subtract multiple column selectors!
columns(dtype_include='object') — columns(['c', ‘d']) selects all object columns except c and d.
These are the transformations Skippa supports. They can be either a wrapper around existing scikit-learn transformers, existing pandas.DataFrame methods, or custom implementations if needed.
.impute() Wrapper around sklearn’s SimpleImputer. Example: .impute(columns(dtype_include='int'), method='medium')
.scale() Wrapper around sklearn’s StandardScaler (if type='standard') or MinMaxScaler (if type='minmax').Example: .scale(columns(dtype_include='numeric'), 'minmax')
.onehot() Wrapper around sklearn’s OneHotEncoder.Example: .onehot(columns(dtype_include='category'))
.encode_date() Custom transformer for creating features out of a date values. It applies pandas’ pd.to_datetime() and uses the properties from the resulting .dt object.Example: .encode_date(columns(['mydate']), {'year': False, 'month': True, 'day': True})
.cast() Wrapper around pandas.DataFrame.astype(). It has aliases .astype() and .as_type().Example: .cast(columns(dtype_include='int'), 'float')
.rename() Wrapper around pandas.DataFrame.rename().N.B. Does not have a columns specifier!Example: .rename({'a': 'name', 'b': 'age'})
.assign() Wrapper around pandas.DataFrame.assign(). N.B. Does not have a columns specifier!Example: .assign(y_sq=lambda df: df['y']*df['y'])
.apply() Wrapper round pandas.DataFrame.apply().Example: .apply(np.log)
.select() Simply selecting a subset of columns.Example: .select(columns(dtype_exclude='object'))
.model() Not a transformer but an estimator. Can only be used as the final step in your pipeline. Provide an (instantiated) sklearn estimator as argument.Example: .model(RandomForestClassifier(n_estimators=200, max_depth=5))
.concat() Concatenate another parallel pipeline to the current one. This will concatenate those pipelines’ resulting set of features.Example:pipe1 = Skippa().apply(np.log)pipe2 = Skippa().apply(np.sqrt)final = pipe2.concat(pipe1).model(LinearRegression())
☝️ The idea is to keep adding transformers for extra functionality, but for any functionality that is not (or not yet) supported there is always an option to use the .assign or .apply methods to apply any transformation.
I’m eager to receive feedback on your experiences, opinions and feature requests, so please leave a reaction or create an issue in Github!