Package: vflow
A library for making stability analysis simple. Easily evaluate the effect of judgment calls to your data-science pipeline (e.g. choice of imputation strategy)!Why usevflow?Usingvflows simple wrappers facilitates many best practices for data science,
as laid out in the predictability, computability, and stability (PCS) framework
forveridical data science. The goal
ofvflowis to easily enable data science pipelines that follow PCS by
providing intuitive low-code syntax, efficient and flexible computational
backends viaRay,
and well-documented, reproducible experimentation viaMLflow.ComputationReproducibilityPredictionStabilityAutomatic parallelization and caching throughout the pipelineAutomatic experiment tracking and savingFilter the pipeline by training and validation performanceReplace a single function (e.g. preprocessing) with a set of functions and easily assess the stability of downstream resultsHere we show a simple example of an entire data-science pipeline with several
perturbations (e.g. different data subsamples, models, and metrics) written
simply usingvflow.importsklearnfromsklearn.datasetsimportmake_classificationfromsklearn.linear_modelimportLogisticRegressionfromsklearn.metricsimportaccuracy_score,balanced_accuracy_scorefromsklearn.model_selectionimporttrain_test_splitfromsklearn.treeimportDecisionTreeClassifierfromvflowimportVset,init_args# initialize dataX,y=make_classification()X_train,X_test,y_train,y_test=init_args(train_test_split(X,y),names=["X_train","X_test","y_train","y_test"],# optionally name the args)# subsample datasubsampling_funcs=[sklearn.utils.resamplefor_inrange(3)]subsampling_set=Vset(name="subsampling",vfuncs=subsampling_funcs,output_matching=True)X_trains,y_trains=subsampling_set(X_train,y_train)# fit modelsmodels=[LogisticRegression(),DecisionTreeClassifier()]modeling_set=Vset(name="modeling",vfuncs=models,vfunc_keys=["LR","DT"])modeling_set.fit(X_trains,y_trains)preds_test=modeling_set.predict(X_test)# get metricsbinary_metrics_set=Vset(name="binary_metrics",vfuncs=[accuracy_score,balanced_accuracy_score],vfunc_keys=["Acc","Bal_Acc"],)binary_metrics=binary_metrics_set.evaluate(preds_test,y_test)Once we've written this pipeline, we can easily measure the stability of metrics (e.g. "Accuracy") to our choice of subsampling or model.DocumentationSee thedocsfor reference on the APINotebook examplesNote that some of these require more dependencies than just those required forvflow. To install all, runpip install vflow[nb].Synthetic classificationEnhancer genomicsfMRI voxel predictionFashion mnist classificationFeature importance stabilityClinical decision rule vettingInstallationStable versionpipinstallvflowDevelopment version (unstable)pipinstallvflow@git+https://github.com/Yu-Group/veridical-flowReferencesinterface: easily build onscikit-learnanddvc(data version control)computation: integration withrayand caching withjoblibtracking:mlflowpull requests very welcome! (seecontributing.md)@software{duncan2020vflow,author={Duncan,JamesandKapoor,RushandAgarwal,AbhineetandSingh,ChandanandYu,Bin},doi={10.21105/joss.03895},month={1},title={{VeridicalFlow:aPythonpackageforbuildingtrustworthydatasciencepipelineswithPCS}},url={https://doi.org/10.21105/joss.03895},year={2022}}