Reproducibility and Risk Audit in Data Science Projects.
By: Linda Cheluget
Tags: Data Science, Deep Learning

1576 researchers took an online questionnaire reproducibility research by nature’s survey and more than 70% of researchers claimed that they’ve tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their experiments.

Reproducibility is the ability of another person to use the same tools and data to achieve similar results. This goal has not been achieved in data science with real-world projects and researches. Forking a repository might be achievable but when it comes to dealing with real-world data it becomes trivial. Having work that’s not replicable makes it insignificant. Reproducibility is a continuing issue within the data science community that has taken quite some time to be addressed. It’s still a challenge for data scientists to share and author models with teams and get similar results, refactoring, replacing, migrating, and transforming codes is quite hard. The crisis of reproducibility is also causing technical debt on many startups.

Reproducibility crisis faced by Deep Learning.

I re-worked a project I had done some time back, Keras model for image captions generation using inception V3, and got different results. The lifecycle of deep learning is quite chaotic this would mean reproducibility is difficult. Management of memory, tracking experiments, and envisioning output are usual complications developers face.

Reproducibility crisis has occurred due to different reasons such as;

• Changes in layers of the deep learning framework such as this could change results if ran many times on different environments, GPU drivers, and datasets such as the training and the validation datasets highly impact the outcome.

• The unavailability of good infrastructure to carry out this work such as enough storage and GPUs to replicate the model, opting to use platforms such as google colab and Kaggle could help curb this.

• The more you run the test set of the data, the more the chances of overfitting it and therefore, results will be unreliable. Early stopping at the best-epoch produces good results not concerning the true performance of the model if epochs which are hyperparameters are set according to the validation error.

• Lack of source control mechanisms of trained data will cause messy results. Training large amounts of data such as videos and images that are eventually fed into the model without any source control tool would mean the difficult replicability of the model.

Reproducibility Issues that could be faced by more complex models;

• With complex models, teams or individuals might not have enough resources or time for the computational of the dataset. Due to this, a subset of the test data might be used instead, the test data is therefore different from what was used on the baseline. This would cause the model to be unreliable.

• Some baselines are not well fine-tuned to handle real-world challenges, additionally, these baselines could have been badly evaluated and since the very critical hyperparameter tuning is not in the codebase, the whole process will be undependable causing a weak baseline.

• Datasets display different characteristics, hence, variations in the behavior of the model.

• Fitting training and testing set is one of the common mistakes as this would mean the features have been normalized dependently and lead to data leakage. Test sets should be exploited for testing and not pre-processing. The best way to normalize is by fitting the scalar on the training set and then evaluate both sets based on the scaler parameter.

• Most complex models don’t use random test set sampling and end up getting good results but it would have been worse, it’s important to have test sets that represent the training set, and to avoid such models that contribute to the replicability crisis.

Remediation strategies.

Reproducibility highly depends on the sequence of steps that were used to produce a certain model;

Code versioning Capturing the model’s development process or lifecycle is crucial, this means tracing all code variations. Without appropriate code versioning, others will probably not understand the outcome by reviewing the project and also at risk of losing the work history. Deepops is one of the software that could be used to manage code versioning.

Data versioning This enables data scientists to use datasets over and over again without experiencing data corruption from new data. Re-running data from traditional databases contributes to the risk of reproducibility. Software such Deepops manage data to avoid such issues.

A packrat is an open-source tool that stores package dependencies for project directories. This normally used by R users. The directory is the private library, that anytime changes are made such as installing or removing a package the change reflects on the private library. Snapshots are used for sharing private libraries through the process of restoring since Packrat doesn’t support directly sharing private libraries, but it still has extensive benefits; Packrat records the exact versions the project depends on and ensures no changes affect it hence the project is reproducible. Projects can be shared across platforms and from one computer to another. Projects are isolated from each other, hence installing or updating packages will not affect projects.

Datmo is another open-source tool that can be used to manage reproducibility by keeping work done intact and managing environments. Datmo enables one to manage interactions of the ever-changing frameworks and tools used for creating models.Datmo tracks your work, factors such as hyperparameter and pre-processing versions that normally change results obtained are kept in check hence one can easily reuse the model.

Containers as a Solution to Reproducibility.

This is an isolated environment that’s used to set up dependencies such as frameworks, code, and libraries that are used to carry out a task. Tasks such as hosting web applications that are interactive and monitoring the quality of the data. This framework aims to provide isolations between projects, rather than creating new environments for every project you can separately specify your dependencies for each project into a container, snapshot the container, and users can boot up from the image which is a standardized and isolated environment.

So, what exactly is an image? This is a snapshot of a container that can be used as a starting point for other containers. Each running container is created from another running container that originated from an image. Many containers can be launched from one container.

How containers work.

Containers store the components required to run the desired project. Components such as libraries, files, dependencies, and variables. Access to physical resources such as memory, GPU, and storage are restrained by the host OS. Therefore, the host’s physical resources will not all be used up by the containers.

Containers contain image files, these are applications’ versions that are complete, static, and ready to be executed. The images contain a base image consisting of multiple layers. The base image comprises of dependencies required for code execution in a container. Every image has a layer that’s writable and readable situated on top of static layers that are unchangeable, this means core image layers can be saved and reused on several containers, since, each container is made up of a layer tasked for customization of that specific container.

An open container initiative image contains file system layers, manifests, and configurations. The OCI image comprises a specified image that entails the information required to launch an application and a runtime defines how a file system works, these files have all data needed for runtimes and performances. Several organizations use container schedulers for the management of deployments. The containers are flexible meaning the same image executed on cloud web service during the test process can be used on an on-premise server without tampering with the application code in the container.

Docker Containers.

Docker can be used to reduce reproducibility in this case, it’s an open-source that was launched in 2013 as Docker Engine. Docker technology focuses on the needs of system operators and developers to isolate dependencies from infrastructure. As a docker user you are completely in control of the environment and aware of what has been installed. Docker allows portability of application and its dependencies hence its efficiency. Configurations in docker are tied to the project, therefore, one is completely aware of the runtime environment at all stages of development.

Why Docker containers?

Library requirements are different for every model. The Docker container isolates these projects, meaning the app will not break. For instance, you have TensorFlow 1.10 in one container and TensorFlow 2.0 is another, both running different models. Models serving different requests can be handled by docker, and if one model needs to be scaled up at a particular request that can also be performed without the app crushing.

Models with different environments such as Lang level set to ‘en_us.UTF-8’ and another at ‘c’ can work in isolated environments but different containers.

References:

Tina Bu, 2018, ‘Build a Docker Container with Your Machine Learning Model’

Jon Machtynger, 2019, ‘How to use containers in data science with Docker and Azure: Part1’