Testing theories of task in visual analytics

to define user analysis behaviors as a form of expression that parallels how people express themselves through natural language, that is, through words, sentences, and paragraphs. In this case, individual interactions represent words; interaction subsequences represent sentences; and interaction subsequences can be chained together to form larger structures akin to paragraphs. Building on this insight and core concepts from automata theory, we developed a new framework for expressing individual visualization taxonomies as regular grammars. A regular grammar consists of a set of terminals, a set of nonterminals, and a set of production rules. Terminals form a dictionary of primitive values akin to words, examples being the drag slider, zoom, and hover interactions mentioned above. Nonterminals are variables representing aliases for any structure of interest, including terminals. Production rules are functions that can be assigned to a nonterminal. Finally, production rules can be as simple as mapping one terminal to a corresponding nonterminal. For example, labeling a logged slider drag with its corresponding interaction label ( filter). They can also be more sophisticated, such as mapping a common subsequence of logged interactions to a nonterminal or mapping entire regular expressions to a nonterminal. Our approach can express virtually any existing visualization taxonomy programmatically by defining it first as a regular grammar (see Figure 1 for an example). To demonstrate the versatility of our approach, we used it to define regular grammars for seven visualization taxonomies from the literature. We then wrote code to apply these grammars to three different interaction log Theory research plays a critical role in many areas of computer science, including artificial intelligence, programming languages, and human-computer interaction. However, striving to encapsulate the human experience within a math equation or a concise model can seem reductive at best, especially when the goal is to design software that actively cooperates with people to achieve complex goals, such as in visualization research. That being said, naive reductions can also help researchers test their understanding of the complexities of human behavior. This column will explore the role of theory in visualization research and our vision for the future of visualization theory work. Visualization has traditionally used qualitative theory approaches to produce guidelines for designing new visualization systems. To do this, researchers conduct user studies and/ or literature surveys to collect relevant data. They then analyze this data for recurring themes and patterns, often formalized as taxonomies, which can be thought of as theoretical models of user analysis behavior. For example, Matthew Brehmer and Tamara Munzner identify 11 interaction types commonly used in visualization interfaces [1], such as filtering irrelevant data points via range sliders and buttons, navigating a dataset via dragging or scrolling, and selecting data points of interest by hovering, clicking, or highlighting. Taxonomies can also represent more sophisticated patterns within the visual analysis process, such as sequences of common interactions or even directed flow charts. With the rise of data-driven modeling techniques such as machine learning, deep learning, and reinforcement learning, researchers are also interested in computational models of user analysis behavior [2,3]. For example, one could instrument a visualization system to log users’ interactions and extract features from these logs to train a machine learning model. With this approach, models of user analysis behavior could potentially be derived automatically rather than by hand. Since computational models are already in a programmed form, they can also be integrated directly into visualization systems, enhancing nascent features such as recommending new visualizations to explore as a person analyzes their dataset [4]. However, a significant challenge is generating cleanly labeled input data to train the models. Although model-training software is automated, the labeling of input data is often still a manual process, eroding the benefits of using this technique over manually derived theoretical models. Rather than keeping these two approaches—taxonomies and computational models—separate, we argue for integrating them, thereby enabling the strengths of one approach to mitigate the weaknesses of the other. For example, theoretical taxonomies can provide classification labels for computational models. If we can write programs to apply these labels to interaction logs, we can also train machine learning models to represent these theories through code that we can then integrate into new and existing visualization systems. To this end, we embarked on a project to investigate the landscape of existing visualization taxonomies and determine which theories we can use to label user interaction logs programmatically [5]. We observe an exciting pattern Testing Theories of Task in Visual Analytics