Measuring How Data Science Notebooks Evolve Over Time

copies as desired. Git was designed, however, to track changes to code, not user insights. Data science is not always—and arguably, rarely—about the code. How do analysts adapt this software to track and reflect on their process of insight discovery? Furthermore, could we quantify how data science evolves over time by analyzing these digital snapshots? For example, do analysts always start with exploration—searching the data for something interesting by writing lots of code and seeing what “sticks”—and end with explanation—presenting their findings and reasoning process to colleagues through text and visualizations? To answer these questions, we downloaded 60,000 Jupyter notebooks from GitHub, an online platform for sharing Git repositories. Then we filtered the corpus for notebooks that seemed to be used for data science purposes. From our collection of 2,575 data science notebooks we randomly sampled 244 notebooks to analyze by hand. Specifically, we wanted to see whether we could distinguish between exploratory notebooks and explanatory ones. We found that exploratory notebooks deal more with experimentation and usually contain more code. Explanatory notebooks deal more with telling the stories behind the code and thus tend to contain more code comments and textual notes. These findings align with previous definitions of exploratory and explanatory notebooks from the literature [4]. However, this manual approach does not scale beyond a couple hundred notebooks. To enable scale-up, we encoded our observations into a more rigorous definition of exploration versus explanation that a regression model or machine-learning model could learn. We annotated each sampled notebook with score ranges between 0.1 and 1.0, representing whether the notebook appeared Data science is often mercurial, exploratory, and highly iterative [1]. Given a panoply of techniques for analyzing complex datasets, one can easily lose track of the code, output tables/visualizations, and notes associated with a data analysis session. More importantly, it can be difficult to remember why the results of a particular session were important and how they were derived. Online platforms such as JupyterLab, RStudio, and Observable tackle this problem in part by enabling users to store these different artifacts (code, outputs, notes) in one place: a computational notebook. In this way, notebook users do not have to wonder where their code or visualizations went; it’s all contained in one notebook. Computational notebooks are especially interesting as living documents for scientific inquiry. Notebooks naturally evolve over time as their authors discover more about the data they analyze. The process of refining a data science notebook can be just as important as the notebook itself, given that the author makes many critical decisions through their edits [2], such as what data attributes to emphasize in their analysis, what relationships between the data are most meaningful, and how these relationships could be leveraged in downstream machine-learning models. In this way, refining a data science notebook is as much about communication as it is about deriving data insights, even when the only person you communicate with is yourself. This process is like that of a painter honing their craft. A painter could paint only for themselves, but if they want to become a better artist, they must reflect on their technique and consider how they can improve. If a solo painter fails to remember the choices they made and their significance, then the painter’s ability to make better choices in the future, and thus produce better paintings, may be limited. How might we track our progress? One possibility is to take snapshots of paintings as they develop. We can write notes to remind ourselves why we photographed a painting at a particular time, including our impressions of the techniques and materials we used. However, if we make a mistake, it’s impossible to revert back to a photograph. Our only choice is to continue with our flawed painting. Another possibility is to paint multiple copies of the same piece in tandem, keeping the paintings identical until we want to try something new. If we dislike the results, we can abandon that copy and proceed with the others, with the added benefit of being able to write notes about what we disliked about this particular copy so we don’t forget. In our research studying data science workflows [3], we observed that some analysts do something similar to our painting scenario but with computational notebooks. Specifically, we see thousands of users saving versions of their Jupyter notebooks using code versioning software such as Git. Users can save specific versions of their notebooks as separate records in Git, enabling them to revert to a previous milestone as needed by rolling the notebook back to the corresponding Git record. But unlike paintings, computational notebooks are digital, making it easy to generate new Niklas Elmqvist, University of Maryland, College Park Deepthi Raghunandan, University of Maryland, College Park Leilani Battle, University of Washington