Jupyter’s Archive: Searchable Output Histories for Computational Notebooks

When using a computational notebook, programmers tend to run, overwrite, and delete cells many times. These actions, which are core to exploratory programming, tend to create a long history of outputs that become fragmented and difficult to track. These outputs are critical to returning to past states when programmers make mistakes in implementation. They are also critical to understanding the evolution of a notebook which can help programmers improve how they code in different situations. To resolve this, this paper introduces the Output Archive, a thumbnailbased output history built into Jupyter Lab that automatically records all outputs produced over the lifetime of a notebook and makes the code that produced them available. This paper also introduces a new class of grouping filters which allows users to navigate large output histories by clustering outputs based on similarities in their underlying code (similar function name, object names, parameters). To test the tool, a usability study was run on 12 computational notebook users who found the Output Archive useful and were able to use its accompanying grouping filters to quickly find important outputs. INTRODUCTION When a programmer wants to explore a new data set, implement algorithms, or test different hypotheses, they use computational notebooks. These notebooks allow programmers to run individual pieces of their programs independently of each other. This is a critical affordance in exploratory programming because it saves programmers time and enables programmers to iterate and improve their code at a much quicker pace. For example, if a programmer has just trained multiple neural networks, instead of rerunning the entire script to change how one of the models was visualized, in a notebook they could just change the plotting code without affecting the training code. The specific feature that enables this affordance is the cell structure of computational notebooks. Each cell is a runnable script that enables a user to break up their program into modular chunks. Often, each time a programmer runs a cell, they create an output (table, graph, text, etc.) in order to check that their code is working as intended or in order to visualize some part of the data set for analysis. As the programmer tweaks and improves their code in a notebook, they quickly produce a “large number” of outputs that becomes “laborious” to sort through [1]. These past outputs produced are vital to tracking “which steps” in the notebook “lead to Jupyter’s archive: search through all past outputs generated in a notebook which results”, or to recover previous states in the notebook [2]. Unfortunately, there has been a lack of useful tools that help computational notebook users keep track of the outputs they generate. Generally, notebook users resort to using “version control tools”, copying “scripts” and “outputs”, or commenting out code in order to take snapshots of the notebook’s state [1]. Recent work in this field, however, has proved more promising with the advent of output history extensions that enable users to explore different output variants [1]. These recent tools, however, only offer histories relevant to specific cells and fail to look at the relationships between outputs spread through the entire notebook. In this paper, we aim to improve how notebook output history is preserved, presented, and searched. We introduce a tool that automatically organizes and makes past outputs searchable. We use a thumbnail-based interface to display all outputs ever produced in a notebook. This tool is built on top of code gathering tools which associates outputs with the exact slice of code that produced it [9]. A slice is a mini-program that only contains the code that led to a specific output. Without slices, each output would be associated with the entire notebook that produced it, making it tedious or even impossible for the programmer to recover the relevant code. Other techniques leveraged in implementation include Abstract Syntax Tree (AST) node traversal, program parsing, and AST diffing. These techniques allow us to not only compare the lines of code that produced an output, but also break apart those individual lines to find similarities. This paper makes two unique contributions. The first is the design and implementation of the Output Archive for Jupyter Lab, an interface for Jupyter Notebook which is an open-source notebook used by millions of people [3]. This tool allows computational notebook users to quickly recover the exact slice of code that produced any past output. A key feature that enables quick output recovery is our new class of grouping filters that enables a user to group outputs by similarities in the code that produced them. For example, with our group by function name filter, a user can find the name of the function that produced a unique plot even after they overwrote the code preceding the plot. The unique insight powering these grouping filters is that users can see valid relationships between outputs produced across different cells, instead of just in the same cell which is what current tools focus on [9]. This more flexible definition of variants, or the different versions and types of outputs, and similarity in outputs allowed us to achieve a large dimensionality reduction that made output search easier Users looking to recover a past output merely have to click on a toolbar command which visualizes the outputs (text, tables, graphs, errors, etc.) generated and search through the outputs using filters. The second contribution of this paper is a controlled usability study that explores the usability and usefulness of the Output Archive and its filters in navigating large output histories. 12 notebook users from a variety of backgrounds ranging from students to software engineers participated in an in-lab study to recover past outputs produced by a notebook coding session. We discovered that not only was the Archive useful, but the filters we developed enabled easier search. On top of this, we discovered a variety of ways programmers would like to group and search their outputs, which revolved around visualizing different types of variants. For example, one variant that participants stated they would like to see is grouping outputs that were produced by the same data set.