5 Tools for Reproducible Data Science | by Rebecca Vickery | Towards Data Science

The definition of reproducibility in science is the “extent to which consistent results are obtained when an experiment is repeated”. Data, in particular where the data is held in a database, can change. Additionally, data science is largely based on random-sampling, probability and experimentation. In this field, it can, therefore, be quite challenging to produce analysis and build models where the results and outcomes can be reproduced either by a colleague or by you at some future point in time.
Although it can be challenging to obtain, there are a number of reasons why reproducibility is essential for good data science.
Reproducibility supports collaboration. Rarely does a data science project take place in isolation. In most situations, data scientists work together with other data scientists and with other teams to see a project through to integration into a business process. In order to collaborate effectively, it is important that other people can repeat, build on and maintain your work.Reproducibility also supports efficiency. To be able to work most efficiently it is essential that you and your colleagues can build on the work that you produce. If results or processes cannot be accurately repeated then it is very difficult to develop on top of existing work and instead, you will find that you have to start a project all over again.Reproducibility builds trust. As previously stated, data science is a discipline built on probability and experimentation. In this field trust in results is extremely important in order to develop buy-in for projects and to work effectively with other teams.
Reproducibility supports collaboration. Rarely does a data science project take place in isolation. In most situations, data scientists work together with other data scientists and with other teams to see a project through to integration into a business process. In order to collaborate effectively, it is important that other people can repeat, build on and maintain your work.
Reproducibility also supports efficiency. To be able to work most efficiently it is essential that you and your colleagues can build on the work that you produce. If results or processes cannot be accurately repeated then it is very difficult to develop on top of existing work and instead, you will find that you have to start a project all over again.
Reproducibility builds trust. As previously stated, data science is a discipline built on probability and experimentation. In this field trust in results is extremely important in order to develop buy-in for projects and to work effectively with other teams.
In the following article, I will share 5 tools that facilitate reproducibility in data science. Each of these tools provides the functionality to solve specific challenges around creating reproducible data science projects, these include:
Recording the tools, libraries and versions used in a project.
Creating self-contained, consistent environments for data science projects.
Developing a consistent, easy to read file structure.
Writing tests for your code.
Using version control.
Watermark is an IPython magic extension that prints information about software versions, hardware and dates and times in any IPython shell or Jupyter Notebook session.
Watermark provides a very quick and simple way to keep track of tools, libraries, versions, authors and dates involved in a project. It is particularly useful for ad hoc or one-off pieces of work, rather than larger projects, that take place in notebooks only. It means that if a colleague wants to repeat a piece of work they have a good idea of the tools and libraries they need to install, who created the work and when it was put together.
To install watermark run the following in your terminal.
pip install watermark
To use the extension in a Jupyter notebook or an IPython shell run the following command.
%load_ext watermark
You can select the information that is printed by specifying a number of flags. The full list of available flags is listed in the documentation. But as an illustration the code below prints the current date, the version of python and IPython I am using, the versions for libraries I have installed and imported, and information about the hardware I am using. The output is shown below the code.
%watermark -d -m -v -p numpy,matplotlib,sklearn,seaborn,pandas 
In data science the environment that a project is created in, that is the tools, libraries and versions installed, has a large impact on reproducing the same results and in many cases being able to run the code at all on another computer.
Virtual environments are tools that provide contained environments for individual projects. These environments contain a file often called a requirements file that lists the dependencies (tools, libraries and versions) for a project. When you create a virtual environment you can choose to install all the dependencies from this file and thus reproduce the same environment that the project was originally created in.
There are a number of tools for creating virtual environments but if you are using python then pipenv is one of the best.
To install pipenv run the following.
$ brew install pipenv
To create a new environment using a specific version of python. Make a new directory and then run the following command from your new directory.
mkdir pip-testcd pip-testpipenv --python 3.7
To activate the environment run pipenv-shell you will now be in a new environment called ‘pip-test’.
If we inspect the contents of the directory we will see that pipenv has created a new file called Pipfile. This is the pipenv equivalent of a requirements file.
ls
Let's inspect the contents of this file. Replace “Sublime Text” with the name of your preferred text editor if necessary.
open -a "Sublime Text" Pipfile
The new file contains the following.
We don’t currently have any packages installed in the environment but if we did we would see them in this file. Let’s install pandas to test this.
pipenv install pandas
If we open the file again we can see that pandas has now been added to the Pipfile.
Now let's suppose a colleague wants to work on your project on a different computer. If they were to clone your repo from Github the Pipfile would be contained within this. They can then simply recreate the exact environment that you have been using simply by running the following.
pipenv install --devpipenv shell
This will create a pipenv environment in the directory you are working in and install all dependencies from the pip file.
If you want to use this environment within a Jupyter notebook session then you need to run this code. You can change the name and display name to something that makes sense for your environment. When you start a Jupyter notebook session this will now appear as an icon to select from the launcher screen (shown below the code). Any new notebooks you create with this kernel will contain all dependencies in your pipenv environment.
python -m ipykernel install --user --name myenv --display-name "Python (pipenv test)"
Having a defined, consistent project structure for data science can make it much easier to collaborate, share and build on a project. Cookie-cutter-data-science is a tool that, in one line of code, creates a standard skeleton project structure.
To install run the following.
pip install cookiecutter
To create a new project you simply run.
cookiecutter https://github.com/drivendata/cookiecutter-data-science
The tool will prompt you for a number of details, many of which are optional.
This has now created the following project structure which will be sufficient for most data science projects.
Pytest is a tool for unit testing your python code. Unit testing is a repeatable action that you can add to your code that checks that the individual units of code work as expected. Unit testing is good practice for adding trust to the code that you write.
Pytest can be installed and imported for use as shown below.
pipenv install pytestimport pytest
A simple test might be to check that the data you are importing has not changed.
import pandas as pdfrom sklearn.datasets import load_bostonboston = load_boston()boston_df = pd.DataFrame(boston.data, columns=boston.feature_names)len(boston_df)assert len(boston_df) == 506
You can add a helpful error message to the statement to make it easier to debug the code.
assert len(boston_df) == 506, "Len(df) should be 506"
If we then change the data the error message will be displayed.
import pandas as pdfrom sklearn.datasets import load_wineboston = load_wine()boston_df = pd.DataFrame(boston.data, columns=boston.feature_names)len(boston_df)assert len(boston_df) == 506, "len(df) should be 506"
Version control is an important tool for developing repeatable projects, sharing work and to facilitate collaboration. Maintaining both a remote and local version of your project allows others to access and develop on their own local version, rollback and monitor changes to code and to safely merge changes and new features.
Github is one of the most well known and widely used platforms for version control. Github uses an application known as Git to apply version control to your code. Files for a project are stored in a central remote location known as a repository. Every time you make a change locally on your machine and push to Github your remote version is updated and a store of that commit is recorded. If you want to rollback to a previous version of your project before you made a commit this record allows you to do this.
Additionally, because the project files are stored remotely anyone else with access can download the repo and make changes to the project. The concept of branching, which in essence means you make a temporary copy of the project which is completely separate, means you can make changes there first without fear of breaking anything.
For usage guidelines, I have written a guide to Github for data scientists here.
Unlike traditional software developments, there are currently minimal standard practices in data science projects. However, the tools described above go a long way towards creating standard practices that in turn will lead to more reproducible data science projects.
Thanks for reading!