DOI: 10.5281/zenodo.2592524
title: Dataset of A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks

description: The self-documenting aspects and the ability to reproduce results have been touted as significant benefits of Jupyter Notebooks. At the same time, there has been growing criticism that the way notebooks are being used leads to unexpected behavior, encourage poor coding practices and that their results can be hard to reproduce. To understand good and bad practices used in the development of real notebooks, we analyzed 1.4 million notebooks from GitHub.

Paper: https://2019.msrconf.org/event/msr-2019-papers-a-large-scale-study-about-quality-and-reproducibility-of-jupyter-notebooks

This repository contains two files:


	dump.tar.bz2
	jupyter_reproducibility.tar.bz2


The <em>dump.tar.bz2</em> file contains a PostgreSQL dump of the database, with all the data we extracted from the notebooks.

The <em>jupyter_reproducibility.tar.bz2</em> file contains all the scripts we used to query and download Jupyter Notebooks, extract data from them, and analyze the data. It is organized as follows:


	analyses: this folder has all the notebooks we use to analyze the data in the PostgreSQL database.
	archaeology: this folder has all the scripts we use to query, download, and extract data from GitHub notebooks.
	paper: empty. The notebook analyses/N12.To.Paper.ipynb moves data to it


In the remaining of this text, we give instructions for reproducing the analyses, by using the data provided in the dump and reproducing the collection, by collecting data from GitHub again.

<strong>Reproducing the Analysis</strong>

This section shows how to load the data in the database and run the analyses notebooks. In the analysis, we used the following environment:


Ubuntu 18.04.1 LTS<br>
PostgreSQL 10.6<br>
Conda 4.5.11<br>
Python 3.7.2<br>
PdfCrop 2012/11/02 v1.38


First, download <em>dump.tar.bz2</em> and extract it:

<pre><code class="language-bash">tar -xjf dump.tar.bz2</code></pre>

It extracts the file <em>db2019-03-13.dump</em>. Create a database in PostgreSQL (we call it "jupyter"), and use psql to restore the dump:

<pre><code class="language-bash">psql jupyter &lt; db2019-03-13.dump</code></pre>

It populates the database with the dump. Now, configure the connection string for sqlalchemy by setting the environment variable <em>JUP_DB_CONNECTTION</em>:

<pre><code class="language-bash">export JUP_DB_CONNECTION="postgresql://user:password@hostname/jupyter";</code></pre>

Download and extract <em>jupyter_reproducibility.tar.bz2</em>:

<pre><code class="language-bash">tar -xjf jupyter_reproducibility.tar.bz2</code></pre>

Create a conda environment with Python 3.7:

<pre><code class="language-bash">conda create -n analyses python=3.7
conda activate analyses</code></pre>

Go to the analyses folder and install all the dependencies of the <em>requirements.txt</em>

<pre><code class="language-bash">cd jupyter_reproducibility/analyses
pip install -r requirements.txt</code></pre>

For reproducing the analyses, run jupyter on this folder:

<pre><code class="language-bash">jupyter notebook</code></pre>

Execute the notebooks on this order:


	Index.ipynb
	N0.Repository.ipynb
	N1.Skip.Notebook.ipynb
	N2.Notebook.ipynb
	N3.Cell.ipynb
	N4.Features.ipynb
	N5.Modules.ipynb
	N6.AST.ipynb
	N7.Name.ipynb
	N8.Execution.ipynb
	N9.Cell.Execution.Order.ipynb
	N10.Markdown.ipynb
	N11.Repository.With.Notebook.Restriction.ipynb
	N12.To.Paper.ipynb


<strong>Reproducing or Expanding the Collection</strong>

The collection demands more steps to reproduce and takes much longer to run (months). It also involves running arbitrary code on your machine. Proceed with caution.

<em><strong>Requirements</strong></em>

This time, we have extra requirements:


All the analysis requirements<br>
lbzip2 2.5<br>
gcc 7.3.0<br>
Github account<br>
Gmail account


<em><strong>Environment</strong></em>

First, set the following environment variables:

<pre><code class="language-bash">export JUP_MACHINE="db"; # machine identifier
export JUP_BASE_DIR="/mnt/jupyter/github"; # place to store the repositories
export JUP_LOGS_DIR="/home/jupyter/logs"; # log files
export JUP_COMPRESSION="lbzip2"; # compression program
export JUP_VERBOSE="5"; # verbose level
export JUP_DB_CONNECTION="postgresql://user:password@hostname/jupyter"; # sqlchemy connection
export JUP_GITHUB_USERNAME="github_username"; # your github username
export JUP_GITHUB_PASSWORD="github_password"; # your github password
export JUP_MAX_SIZE="8000.0"; # maximum size of the repositories directory (in GB)
export JUP_FIRST_DATE="2013-01-01"; # initial date to query github
export JUP_EMAIL_LOGIN="gmail@gmail.com"; # your gmail address
export JUP_EMAIL_TO="target@email.com"; # email that receives notifications
export JUP_OAUTH_FILE="~/oauth2_creds.json" # oauth2 auhentication file
export JUP_NOTEBOOK_INTERVAL=""; # notebook id interval for this machine. Leave it in blank
export JUP_REPOSITORY_INTERVAL=""; # repository id interval for this machine. Leave it in blank
export JUP_WITH_EXECUTION="1"; # run execute python notebooks
export JUP_WITH_DEPENDENCY="0"; # run notebooks with and without declared dependnecies
export JUP_EXECUTION_MODE="-1"; # run following the execution order
export JUP_EXECUTION_DIR="/home/jupyter/execution"; # temporary directory for running notebooks
export JUP_ANACONDA_PATH="~/anaconda3"; # conda installation path
export JUP_MOUNT_BASE="/home/jupyter/mount_ghstudy.sh"; # bash script to mount base dir
export JUP_UMOUNT_BASE="/home/jupyter/umount_ghstudy.sh"; # bash script to umount base dir
export JUP_NOTEBOOK_TIMEOUT="300"; # timeout the extraction


# Frequenci of log report
export JUP_ASTROID_FREQUENCY="5";
export JUP_IPYTHON_FREQUENCY="5";
export JUP_NOTEBOOKS_FREQUENCY="5";
export JUP_REQUIREMENT_FREQUENCY="5";
export JUP_CRAWLER_FREQUENCY="1";
export JUP_CLONE_FREQUENCY="1";
export JUP_COMPRESS_FREQUENCY="5";

export JUP_DB_IP="localhost"; # postgres database IP</code></pre>

Then, configure the file <em>~/oauth2_creds.json</em>, according to yagmail documentation: https://media.readthedocs.org/pdf/yagmail/latest/yagmail.pdf

Configure the <em>mount_ghstudy.sh</em> and <em>umount_ghstudy.sh</em> scripts. The first one should mount the folder that stores the directories. The second one should umount it. You can leave the scripts in blank, but it is not advisable, as the reproducibility study runs arbitrary code on your machine and you may lose your data.

<em><strong>Scripts</strong></em>

Download and extract <em>jupyter_reproducibility.tar.bz2</em>:

<pre><code class="language-bash">tar -xjf jupyter_reproducibility.tar.bz2</code></pre>

Install 5 conda environments and 5 anaconda environments, for each python version. In each of them, upgrade pip, install pipenv, and install the archaeology package (Note that it is a local package that has not been published to pypi. Make sure to use the -e option):

<em><strong>Conda 2.7</strong></em>

<pre><code class="language-bash">conda create -n raw27 python=2.7 -y
conda activate raw27
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Anaconda 2.7</strong></em>

<pre><code class="language-bash">conda create -n py27 python=2.7 anaconda -y
conda activate py27
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology
</code></pre>

<strong><em>Conda 3.4</em></strong>

It requires a manual jupyter and pathlib2 installation due to some incompatibilities found on the default installation.

<pre><code class="language-bash">conda create -n raw34 python=3.4 -y
conda activate raw34
conda install jupyter -c conda-forge -y
conda uninstall jupyter -y
pip install --upgrade pip
pip install jupyter
pip install pipenv
pip install -e jupyter_reproducibility/archaeology
pip install pathlib2</code></pre>

<em><strong>Anaconda 3.4</strong></em>

<pre><code class="language-bash">conda create -n py34 python=3.4 anaconda -y
conda activate py34
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Conda 3.5</strong></em>

<pre><code class="language-bash">conda create -n raw35 python=3.5 -y
conda activate raw35
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Anaconda 3.5</strong></em>

It requires the manual installation of other anaconda packages.

<pre><code class="language-bash">conda create -n py35 python=3.5 anaconda -y
conda install -y appdirs atomicwrites keyring secretstorage libuuid navigator-updater prometheus_client pyasn1 pyasn1-modules spyder-kernels tqdm jeepney automat constantly anaconda-navigator
conda activate py35
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Conda 3.6</strong></em>

<pre><code class="language-bash">conda create -n raw36 python=3.6 -y
conda activate raw36
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Anaconda 3.6</strong></em>

<pre><code class="language-bash">conda create -n py36 python=3.6 anaconda -y
conda activate py36
conda install -y anaconda-navigator jupyterlab_server navigator-updater
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Conda 3.7</strong></em>

<pre><code class="language-bash">conda create -n raw37 python=3.7 -y
conda activate raw37
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Anaconda 3.7</strong></em>

When we executed the experiments, the anaconda package for Python 3.7 was not complete. So, we attempted to install all Anaconda 3.x dependencies manually

<pre><code class="language-bash">conda create -n py37 python=3.7 anaconda -y
conda activate py37
conda install -y _ipyw_jlab_nb_ext_conf alabaster anaconda-client anaconda-navigator anaconda-project appdirs asn1crypto astroid astropy atomicwrites attrs automat
conda install -y babel backports backports.shutil_get_terminal_size beautifulsoup4 bitarray bkcharts blaze blosc bokeh boto bottleneck bzip2
conda install -y cairo colorama constantly contextlib2 curl cycler cython
conda install -y defusedxml docutils et_xmlfile fastcache filelock fribidi
conda install -y get_terminal_size gevent glob2 gmpy2 graphite2 greenlet
conda install -y harfbuzz html5lib hyperlink imageio imagesize incremental isort
conda install -y jbig jdcal jeepney jupyter jupyter_console jupyterlab_launcher keyring kiwisolver
conda install -y libtool libxslt lxml matplotlib mccabe mkl-service mpmath navigator-updater
conda install -y nltk nose numpydoc openpyxl pango patchelf path.py pathlib2 patsy pep8 pkginfo ply pyasn1 pyasn1-modules pycodestyle pycosat pycrypto pycurl pyflakes pylint pyodbc pywavelets
conda install -y rope scikit-image scikit-learn seaborn service_identity singledispatch spyder spyder-kernels statsmodels sympy
conda install -y tqdm traitlets twisted unicodecsv xlrd xlsxwriter xlwt zope zope.interface
conda install -y sortedcollections typed-ast
pip install --upgrade pip
pip install pipenv
pip install -e jupyter_reproducibility/archaeology</code></pre>

<em><strong>Stopwords</strong></em>

Use nltk to download stopwords:

<pre><code class="language-bash">conda activate py36
python -c "import nltk; nltk.download('stopwords')"</code></pre>

Everything should be set to run right now.

<em><strong>Executing</strong></em>

In this step, we recommend using the <em>py36</em> environment to orchestrate the execution. We designed the scripts for Python 3.6, and if they are correctly configured, it can invoke the other environments.

<pre><code class="language-bash">conda activate py36</code></pre>

If you want to extend the execution to more environments, configure the environments on the file <em>archaeology/config.py</em>.

For querying and downloading repositories from github, run on the <em>jupyter_reproducibility/archaeology</em> directory:

<pre><code class="language-bash">python s0_repository_crawler.py</code></pre>

For extracting data from the repositories and notebooks, run on this order:

<pre><code class="language-bash">python s1_notebooks_and_cells.py
python s2_requirement_files.py
python s3_compress.py
python s4_markdown_features.py
python s5_extract_files.py
python s6_cell_features.py
python s7_execute_repositories.py
python p0_local_possibility.py
python p1_notebooks_and_cells.py
python p2_sha1_exercises.py</code></pre>

Alternatively, execute the following script that orchestrates all the executions and notifies when they finish:

<pre><code class="language-bash">python main_with_crawler.py</code></pre>

If some script fails to process all repositories/notebooks/cells, use the option "-e" to rerun it and force the re-extraction.

After this process, refer to the <em>*Reproducing the Analysis*</em> Section for analyzing the collected data.

<strong>Changelog</strong>

2019/01/14 - Version 1 - Initial version<br>
2019/01/22 - Version 2 - Update N8.Execution.ipynb to calculate rate of failure for each reason<br>
2019/03/13 - Version 3 - Update package for camera ready. Add columns to db to detect duplicates, change notebooks to consider it and add N1.Skip.Notebook.ipynb and N11.Repository.With.Notebook.Restriction.ipynb.