Package: jenga
JengaOverviewJengais an open source experimentation library that allows data science practititioners and researchers to study the effect of common data corruptions (e.g., missing values, broken character encodings) on the prediction quality of their ML models.We design Jenga around three core abstractions:Taskscontain a raw dataset, an ML model and a prediction taskData corruptionstake raw input data and randomly apply certain data errors to them (e.g., missing values)Evaluatorstake a task and data corruptions, and execute the evaluation by repeatedly corrupting the test data of the task, and recording the predictive performance of the model on the corrupted test data.Jenga's goal is assist data scientists with detecting such errors early, so that they can protected their models against them. We provide ajupyter notebook outlining the most basic usage of Jenga.Note that you can implement custom tasks and data corruptions by extending the corresponding providedbase classes.We additionally provide three advanced usage examples of Jenga:Studying the impact of missing valuesStress testing a feature schemaEvaluating the helpfulness of data augmentation for an image recognition taskInstallationThe following options are possible:pipinstalljenga# jenga is ready for the most corruptions (not images)pipinstalljenga[all]# install all dependencies, optimal for developmentpipinstalljenga[image]# also installs tensorflow ad image corruption/augmentation librariespipinstalljenga[validation]# also install tensorflow and tensorflow-data-validation necessary for SchemaStresstestResearchJengais based on experiences and code from our ongoing research efforts:Sebastian Schelter, Tammo Rukat, Felix Biessmann (2020).Learning to Validate the Predictions of Black Box Classifiers on Unseen Data.ACM SIGMOD.Tammo Rukat, Dustin Lange, Sebastian Schelter, Felix Biessmann (2020):Towards Automated ML Model Monitoring: Measure, Improve and Quantify Data Quality.ML Ops workshop at the Conference on Machine Learning and Systems (MLSys).Felix Biessmann, Tammo Rukat, Philipp Schmidt, Prathik Naidu, Sebastian Schelter, Andrey Taptunov, Dustin Lange, David Salinas (2019).DataWig - Missing Value Imputation for Tables.JMLR (open source track)Dependency Management & ReproducibilityAlways keep your abstract (unpinned) dependencies updated inenvironment.yamland eventually
insetup.cfgif you want to ship and install your package viapiplater on.Create concrete dependencies asenvironment.lock.yamlfor the exact reproduction of your
environment with:conda env export -n jenga -f environment.lock.yamlFor multi-OS development, consider using--no-buildsduring the export.Update your current environment with respect to a newenvironment.lock.yamlusing:conda env update -f environment.lock.yaml --pruneInstallation for DevelopmentIn order to set up the necessary environment:create an environmentjengawith the help ofconda,conda env create -f environment.yamlactivate the new environment withconda activate jengainstalljengawith:python setup.py install # or `develop`Optional and needed only once aftergit clone:install severalpre-commitgit hooks with:pre-commit installand checkout the configuration under.pre-commit-config.yaml.
The-n, --no-verifyflag ofgit commitcan be used to deactivate pre-commit hooks temporarily.Then take a look into thenotebooksfolder.NoteThis project has been set up using PyScaffold 3.2.2 and thedsproject extension0.4.
For details and usage information on PyScaffold seehttps://pyscaffold.org/.