Package: fastpipeline
FastPipelinePersistent, easy to use, fast to codeDocumentation:https://shashank-yadav.github.io/fastpipeline/Source Code:https://github.com/shashank-yadav/fastpipelineFastPipeline is a framework for creating general purpose pipeline in your ML projects. It helps in keeping track of your experiments by automatically storing all the intermediate data and source code.The key features are:Persistence: Automatically stores all the intermediate data and variables during the run.Autoreload: Detects if something has been computed before and reloads it instead of a do-over.Accessible Intermediate Data: The intermediate data is stored as pickle and json files, can be easily accessed and analyzed.General Purpose: Unlike sklearn pipelines you don't need to format your data into the required X, y format.Intuitive: Great editor support.Completioneverywhere. Less time debugging.Easy: Designed to be easy to use and learn. Less time reading docs.Installation$pipinstallfastpipeline---> 100%ExampleTrain a classifier over the (in)famous MNIST datasetCreate a filemnist_pipeline.pyMake necessary imports and create a classDataLoaderthat extends theBaseNodeclass from the fastpipeline package. This is something we'll refer to as aNode# Import datasets, classifiers and performance metricsfromsklearnimportdatasets,svm,metricsfromsklearn.model_selectionimporttrain_test_splitimportnumpyasnp# Import pipeline and node constructsfromfastpipeline.base_nodeimportBaseNodefromfastpipeline.pipelineimportPipeline# Node for loading dataclassDataLoader(BaseNode):def__init__(self):super().__init__()defrun(self,input={}):# The digits datasetdigits=datasets.load_digits()# To apply a classifier on this data, we need to flatten the image, to# turn the data in a (samples, feature) matrix:n_samples=len(digits.images)data=digits.images.reshape((n_samples,-1))return{'data':data,'target':digits.target}Create anotherNodewhose input is output ofDataLoaderand that trains an SVM classifier# Node for training the classifierclassSVMClassifier(BaseNode):def__init__(self,config):super().__init__(config)gamma=config['gamma']# Create a classifier: a support vector classifierself.classifier=svm.SVC(gamma=gamma)defrun(self,input):data=input['data']target=input['target']# Split data into train and test subsetsX_train,X_test,y_train,y_test=train_test_split(data,target,test_size=0.5,shuffle=False)# We learn the digits on the first half of the digitsself.classifier.fit(X_train,y_train)# Now predict the value of the digit on the second half:y_pred=self.classifier.predict(X_test)return{'acc':np.mean(y_test==y_pred),'y_test':y_test,'y_pred':y_pred}Now let's instantiate the nodes and create our pipelineif__name__=="__main__":# Initialize the nodesdl_node=DataLoader()svm_node=SVMClassifier({'gamma':0.01})# Create the pipelinepipeline=Pipeline('mnist',[dl_node,svm_node])# Run pipeline and see resultsresult=pipeline.run(input={})print('Accuracy:%s'%result['acc'])Run the pipeline using$ python mnist.py. You should see somthing like:As expected it says that this is the first run and hence for both nodes outputs are being computed by calling theirrunmethod. The log here shows where the data is being storedTry running it again with the same command:$ python mnist.py. This time you should see something different:Since all the intermediate outputs are already computed, the pipeline just reloads the data at each step instead of re-computingLet's make a change to the value of config inside__main__:# svm_node = SVMClassifier({'gamma': 0.01})svm_node=SVMClassifier({'gamma':0.05})Run the pipeline again. You'll see something like:This time it used the result from first node as-is and recomputed for second node, since we made a change to the config.If you make any changes to the classSVMClassifiersame thing will happen again.