Introducing Fastpipeline for ML
By: Shashank Yadav
Tags: Deep Learning, Pipeline, Machine Learning, AI, Classification

TLDR; Quick and easy pipelines that autodetect duplicate runs and reuse intermediate results. Here are the relevant links:

Documentation: https://shashank-yadav.github.io/fastpipeline/

Source Code: https://github.com/shashank-yadav/fastpipeline

If you’re a data scientist you might have used sklearn pipelines at some point. Using pipelines makes your workflow easy to read and comprehend. It helps divide a big task into logical chunks while making your work reproducible. However, I found several issues when using sklearn pipelines in my workflow:

Restrictive API: sklearn pipeline consists of transformers, you can use the existing ones or create your own. However, it forces you to work with only array-like data and follow fit-transform paradigm. For example, it is not possible to pass data other than X and y from one step to another. In short, the API is too strict and tightly coupled with sklearn constructs Wasteful Reruns: Say your pipeline has 5 steps and some failure happens at the 3rd step. Now if you rerun your pipeline after making the required change, it’ll start from the beginning (even when using the memory argument for serialization). This makes experimentation annoying and time-consuming. Logging intermediate data: By very definition, a pipeline will have some data generated at each of the steps and you’d want to have a look at that data for debugging purposes. However, when using sklearn, you’d need to manually store and version most of the data yourself.

I tried several other solutions like Luigi and Airflow, they seemed to be meant mainly for production and not for fast experimentation. Other tools like MLFlow seemed like a big buy-in.

Requirements:

Because of the above-mentioned reasons, I decided to build a simple framework that allowed me to experiment faster while using pipelines for their advantages. There were 5 major requirements:

Should be part of the experimentation process rather than a hindrance Should allow examination of everything that is generated during the run Should reuse results whenever possible automatically Should generate usable logs for every run Should not be too restrictive and allow using different libraries

Out of this process came fastpipeline. It satisfies all the above requirements up to a reasonable extent.

Example:

Let’s train an SVM classifier over MNIST using fastpipeline:

Install the package using: pip install fastpipeline

Create a file mnist_pipeline.py

Make necessary imports and create a class DataLoader that extends the BaseNode class from the fastpipeline package. This is something we'll refer to as a Node

# Import datasets, classifiers and performance metrics

from sklearn import datasets, svm, metrics

from sklearn.model_selection import train_test_split

import numpy as np # Import pipeline and node constructs

from fastpipeline.base_node import BaseNode

from fastpipeline.pipeline import Pipeline # Node for loading data

class DataLoader(BaseNode):

def __init__(self):

super().__init__()



def run(self, input = {}):

# The digits dataset

digits = datasets.load_digits() # To apply a classifier on this data, we need to flatten the image, to

# turn the data in a (samples, feature) matrix:

n_samples = len(digits.images)

data = digits.images.reshape((n_samples, -1))

return {

'data': data,

'target': digits.target

}

Create another Node whose input is the output of DataLoader and that trains an SVM classifier

# Node for training the classifier

class SVMClassifier(BaseNode):

def __init__(self, config):

super().__init__(config)

gamma = config['gamma']

# Create a classifier: a support vector classifier

self.classifier = svm.SVC(gamma=gamma)



def run(self, input):

data = input['data']

target = input['target'] # Split data into train and test subsets

X_train, X_test, y_train, y_test = train_test_split(

data, target, test_size=0.5, shuffle=False) # We learn the digits on the first half of the digits

self.classifier.fit(X_train, y_train) # Now predict the value of the digit on the second half:

y_pred = self.classifier.predict(X_test) return {

'acc': np.mean(y_test == y_pred),

'y_test': y_test,

'y_pred': y_pred

}

Now let’s instantiate the nodes and create our pipeline

if __name__ == "__main__":

# Initialize the nodes

dl_node = DataLoader()

svm_node = SVMClassifier({'gamma': 0.01})



# Create the pipeline

pipeline = Pipeline('mnist', [dl_node, svm_node]) # Run pipeline and see results

result = pipeline.run(input={})

print('Accuracy: %s'%result['acc'])

Run the pipeline using $ python mnist.py . You should see something like:

As expected it says that this is the first run and hence for both nodes outputs are being computed by calling their run method. The log here shows where the data is being stored

Try running it again with the same command: $ python mnist.py . This time you should see something different:

Since all the intermediate outputs are already computed, the pipeline just reloads the data at each step instead of re-computing

Let’s make a change to the value of config inside __main__ :

# svm_node = SVMClassifier({'gamma': 0.01})

svm_node = SVMClassifier({'gamma': 0.05})

Run the pipeline again. You’ll see something like:

This time it used the result from the first node as-is and recomputed for the second node since we made a change to the config.

If you make any changes to the class SVMClassifier the same thing will happen again

The logs of a pipeline run are saved as JSON files and are of the form yyyy-mm-dd--hh-mm-ss . For us, the one generated in the final run is: 2020-12-04--01-46-46.json

2020-12-04--01-46-46.json

As you can see, it tells you the paths of all data which have been saved, also tells you if a result was reused from a previous run. We’ll be working towards making these logs more informative and useful in the future.

For more detailed information about the inner working you can have a look here: https://shashank-yadav.github.io/fastpipeline/about/

Conclusion:

The library is pretty easy to get started with and won’t require much change in your workflow, for benefits you’ll get reproducibility, fast experimentation, and better logging. As mentioned at the beginning of the post, here are some important links:

Documentation: https://shashank-yadav.github.io/fastpipeline/

Source Code: https://github.com/shashank-yadav/fastpipeline

You’ll find some examples in the Github repo. More on the way. Happy to hear your feedback!