SlowGuess/NAACL_2022_OCR huggingface dataset:
Henrik Voigt<sup>1</sup>, Özge Alacam<sup>2</sup>, Monique Meuschke<sup>3</sup>, Kai Lawonn<sup>1</sup> and Sina Zarrieß<sup>2</sup>
<sup>1</sup>University of Jena <sup>2</sup>University of Bielefeld <sup>3</sup>University of Magdeburg
first.last@uni-jena.de
$^{2}$ first.last@uni-bielefeld.de
last@isg.cs.uni-magdeburg.de
Natural language as a modality of interaction is becoming increasingly popular in the field of visualization. In addition to the popular query interfaces, other language-based interactions such as annotations, recommendations, explanations, or documentation experience growing interest. In this survey, we provide an overview of natural language-based interaction in the research area of visualization. We discuss a renowned taxonomy of visualization tasks and classify 119 related works to illustrate the state-of-the-art of how current natural language interfaces support their performance. We examine applied NLP methods and discuss human-machine dialogue structures with a focus on initiative, duration, and communicative functions in recent visualization-oriented dialogue interfaces. Based on this overview, we point out interesting areas for the future application of NLP methods in the field of visualization.
Natural language as a modality for interacting with visual models enjoys increasing popularity in human-computer interface research in the fields of Human-Computer-Interaction (HCI) and Visualization (VIS) (Yu and Silva, 2020; Srinivasan et al., 2020a; Liu et al., 2021; Narechinaia et al., 2021; Kim et al., 2021c). At the same time, interest in tasks involving the visual modality has grown strongly in NLP research in recent years (Suhr et al., 2017; Hudson and Manning, 2019; Acharya et al., 2019). While there are common interests and parallel trends in VIS and NLP, research in these fields often adopts different perspectives on what interaction is and how it should be modeled. Broadly speaking, in the VIS community, a lot of research aims to understand why users need to interact with a visualization and what users' intents are when they interact with a visual model (Dimara and Perin, 2020). Therefore, Brehmer and Munzner (2013)
categorize users' data-related intents into visualization tasks and introduce a taxonomy to describe them in general terms and compare them among applications. Recent contributions show that different forms of natural language-based interaction prove suitable to support users in accomplishing various visualization tasks. This concerns not only the popular query interfaces, but also, on a broader scale, the provision of recommendations, annotations, explanations, documentations, or for support in analytical conversation. However, the variety of existing visualization tasks benefiting from natural language interaction beyond simple query interfaces has not yet received much attention in the NLP community. On the other hand, modern NLP methods offer enormous potential for modeling multi-modal dialogues in the visualization tasks.
In this survey, we aim to complement the why-oriented perspective of classifying visualization tasks by intent in VIS with the how-oriented dialogue modeling perspective in NLP for works involving natural language interaction. To substantiate the classification of the papers, we first delimit the scope of the survey and explain the methodology employed to derive the selected papers in Section 2. We discuss the taxonomy of abstract visualization tasks by Brehmer and Munzner (2013) as the basis for the classification in Section 3 by explaining why interaction with a visualization is performed. Section 4 focuses on how interaction is implemented in the works at hand in terms of applied NLP methods as well as characteristic structures in human-machine dialogue (Bunt et al., 2010). Finally, challenges arising from current approaches are pointed out. As such, the need to compile reliable data sets to support the adoption of deep learning-based NLP methods in the field of VIS yields promising space for future creative work.
Human Language Technologies, pages 348 - 374
Figure 1: Investigation of dialogue structures as applied in visualization systems based on the DIT++ taxonomy of dialogue acts by Bunt (2009).
Related Surveys. The survey of Shen et al. (2021) considers 55 visualization-oriented natural language interfaces (V-NLI) at the intersection of NLP and VIS. The work focuses on the applicability of natural language interfaces at the different steps of the information visualization pipeline by Kard et al. (1999). The authors discuss query-based language interfaces in detail. Similarly, Ozcan et al. (2020) put their focus on querying data visualizations using natural language. However, querying a visualization interface is only one possible language-based interaction among many such as annotation, description generation, documentation, and others. Klopfenstein et al. (2017) conduct a more general study on the application of conversational interfaces and derive usage patterns and paradigms for their implementation. Further related work is done at the intersection of Machine Learning (ML) and VIS as by Wu et al. (2021) or Wang et al. (2020), who illuminate where and how ML gains ground in VIS and discuss future directions for applying ML in VIS research. Facing that, we identify substantial ground for a survey that examines current natural language interaction techniques supporting the accomplishment of visualization tasks. Related work in VIS yields comprehensive and well-conducted state-of-the-art surveys focusing on where dialogue systems can be integrated into the information visualization
pipeline proposed by Kard et al. (1999). This work is complementary in that it illuminates from an NLP perspective how visualization-oriented dialogue is structured in terms of initiative, duration, and present communicative functions within the respective visualization task at hand (see Figure 1). By shedding a light on this we hope to arouse interest in the NLP community for the interesting multi-modal dialogue modeling tasks emerging at the intersection of NLP and VIS.
For a paper to be included in the survey paper selection, it must meet the following criteria:
- Language-based interaction must be a designated input/output modality and some kind of language interface must be provided for it, e.g., a text box or a microphone/speaker.
- Language-based interaction must serve to fulfill or support a main visualization task. For example, using natural language for logging into an application is neither a visualization task nor does it support the accomplishment of that task and is, therefore, not valid. In contrast, using natural language to annotate certain aspects of a visualization is considered supportive of achieving the goal of the visualization task and therefore is valid.
In addition to contributions that include concrete implementations of interaction scenarios, theoretical papers that discuss design spaces or considerations of language-based interaction possibilities are included. The aim is to explicitly show not only what has already been implemented, but also which interaction possibilities are conceivable and useful in multi-modal visualization-oriented dialogue. The paper selection is made in a two-stage process. First, a set of seed papers is derived from conference proceedings of the main conferences in HCI, namely SIGCHI, VIS, namely IEEE VIS, Pacific VIS, and EuroVIS, and NLP, namely ACL and EMNLP, starting from the year 2010 until 2021. The papers are filtered using the keywords language, visualization, interface in combination with a semantic embedding map of the abstracts based on Reimers and Gurevych (2019). The exploratory process results in a set of 76 papers. In the second stage, the references of the seed papers are examined and relevant papers that meet the specified
criteria are included in the set. This results in a final set of 119 papers. For a detailed insight into the scope of the survey, we refer to Appendix A.
3 Why Users interact with Visualizations
Brehmer and Munzner (2013) introduce a multilevel typology for abstract description and comparison of visualization tasks between applications. An abstract visualization task represents a high-level description of why interaction with a visualization application is performed, how it is performed, and what the input and output of the task are. The why-branch of Brehmer and Munzner's typology was chosen primarily for three reasons: First, the high level of abstraction allows to cover a high number of visualization tasks and therefore ensures high representativeness. On the other hand, the modular character of the typology is beneficial for breaking down complex tasks into smaller subtasks in which commonalities can then be identified. In addition, the combination with the what and the how branch offers the possibility to describe task chains, which can serve as a blueprint for the design of a dialog with the system. The papers are classified on the basis of the why-branch of the taxonomy because it distinguishes the tasks taking into account the goal to be achieved and thus corresponds to the goal definition as also used in goal-oriented dialogue modeling (Bordes et al., 2016; Li et al., 2017; Liu et al., 2018). The why branch spawns the abstract visualization tasks present, discover, enjoy, and produce illustrated in Figure 2. Following Munzner (2009), we consider language-based interaction as domain- and interface-independent operations performed by users and/or systems by applying natural language in any kind of representation, e.g., written- or spoken text. Table 1 shows an overview of the contributions and the respective visualization task to which they are assigned. In the following subsections, concrete tasks involving natural language interaction are presented for each abstract visualization task of the taxonomy. Each section includes a brief definition of the targeted visualization task and a detailed discussion of current related work that addresses it. For a detailed inspection, we refer to Appendix B.
3.1 Present Task
Brehmer and Munzner (2013) define presentation as 'the use of visualization for the succinct communication of information, for telling a story with
Figure 2: Why-branch of the Multi-Level Typology of Abstract Visualization Tasks inspired by Brehmer and Munzner (2013).
data, guiding an audience through a series of cognitive operations'. During this task, natural language is used to complement the presentation of visual findings and results, for data-driven storytelling, or to explain, evaluate and discuss them.
Visual Storytelling. Visual storytelling considers the communication of knowledge to a broad audience using visual and textual elements that follow a coherent narrative. The main idea is to match linguistic and visual elements and arrange them consistently within a story. Von Landesberger et al. (2021) study how text and visualization interact with each other in a visual storytelling scenario pointing out that visualizations complement the narrative by providing overview, details, and comparison. Automatic story generation is done by Shi et al. (2021) who leverage the generation of a visual story from spreadsheet input. Natural language text drives the story of a visualization presentation in Kwon et al. (2014); Bryan et al. (2017); Metoyer et al. (2018). Users are guided through the story by interacting with the text segments and the system creates visual animations correspondingly in response.
Explanation Generation. Visualizations offer great potential to create understanding for complex issues among different user groups. In contrast to storytelling, explanation generation is not about assigning a sequence of visual elements to a text-based story, but about automatically explaining given visual facts through natural language texts. Combining text and visualization is used to explain complex processes, e.g., in verbalizing the functionality of ML models (Hohman et al., 2019).
Visualization Task	Subtask	#	Representative Paper(s)
Present	Visual Storytelling	6	(Kwon et al., 2014; Metoyer et al., 2018)
Explanation Generation	3	(Sevastjanova et al., 2018; Hohman et al., 2019)
Discover	Keyword Search	15	(Isaacs et al., 2014; Feng et al., 2018; Schleußinger and Henkel, 2018)
Querying	45	(Setlur et al., 2016; Yu and Silva, 2020; Narechinaia et al., 2021)
VQA	3	(Mathew et al., 2021; Chaudhry et al., 2020)
Browsing	7	(Setlur et al., 2020; Luo et al., 2020; Srinivasan and Setlur, 2021)
Enjoy	Augmentation	12	(Srinivasan et al., 2019b; Hullman et al., 2013)
Description Generation	14	(Obeid and Hoque, 2020; Hsu et al., 2021)
Produce	Annotation	7	(Chen et al., 2010b; Ren et al., 2017)
Documentation	1	(Nafari and Weaver, 2015)
Visualization Creation	6	(Cui et al., 2020; Fulda et al., 2016)
Table 1: Classification of papers based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013). For the sake of clarity, papers are classified into the most suitable category only, although some works touch on several categories in terms of content. A comprehensive listing can be found in Appendix B.
Sevastjanova et al. (2018) discuss strategies on how to present language explanations during the model inference process and the interaction techniques to be required, such as details-on-demand, guidance, dialogue, and exploration.
3.2 Discover Task
Using natural language to discover information is one of the most common visualization tasks targeted by V-NLI. Brehmer and Munzner (2013) differentiate between different levels of task granularity such as discover - search - query (see Figure 2). The discovery of concepts, objects, and relationships in a visualization depends on the role that the user and the interface take in the visualization-oriented dialogue, as well as on the concreteness of the user's intent. Intents are formulated in oral or written form. Less concrete user intents lead to a more exploratory character of the search. Concrete intents formalized in a query lead to a specific system response. Vague and fuzzy intents are much more difficult to formalize in a single query and must be inferred by the V-NLI through the application of intelligent recommendations or user guidance.
Keyword Search. Discovering information about a visualization by supplementing it with a keyword search interface is examined by Feng et al. (2018). The authors enable the search of visual concepts in a 2D visualization via text input. Further visualization-oriented keyword search interfaces are applied in Chowdhury et al. (2021); Siddiqui and Hoque (2020); Chung et al. (2010). In contrast to that, visual search interfaces take in keywords but focus on displaying results in a way that facilitates visual exploration, as targeted in Wilson et al. (2010); Schleußinger and Henkel (2018); Peltonen et al. (2017). Search history and coverage are tracked and visualized by Isaacs et al. (2014).
Natural Language Querying. Natural language querying is a scenario in which a user formulates a query to a visual model and the system is tasked with outputting a visual response to that query – referred to as query2viz. Most of the existing V-NLIs focus on this task. Theoretical work on utterance structures in natural language querying has been done by Srinivasan et al. (2019a, 2021b) finding that utterances mainly target attribute, chart type, encoding, aggregation, and design aspects of a visual model. Liu et al. (2021); Sun et al. (2014);
Narechinaia et al. (2021) generate a visualization based on a data table and a natural language query. Yu and Silva (2020) allow query sequences to be specified in a visual exploration workflow.
Ambiguities. Resolving ambiguities and underspecified utterances poses a difficult problem in this visualization task, especially for single-turn query interfaces. Hearst et al. (2019); Tory and Setlur (2019) develop design guidelines for how systems should respond to queries that contain vague modifiers or -user intents by exploring contextual inference strategies. Gao et al. (2015) manage ambiguities in input utterances using visual ambiguity widgets. Setlur et al. (2019) apply inferencing rules based on known syntactic and semantic input structures. Setlur and Kumar (2020) use word co-occurrence in combination with sentiment analysis to determine data attributes and filter ranges associated with the articulated vague property.
Hypothesis Verification. Discovering novel insights from data is usually done by (dis-)validating hypotheses. Choi et al. (2019a,b) study the use of visualizations to prove or disprove natural language hypotheses visually. The user initiates a hypothesis test by formulating it in natural language, and the system indicates the match with the underlying data set by creating a graph that highlights matches/discrepancies in striking green/red colours.
Query Dialogue. Setlur et al. (2016); Aurisano et al. (2016); Bacci et al. (2020) extend the single-turn query2viz interaction to a multi-turn interactive visual exploration also referred to as analytical conversation. Analytical conversation is the support of visual analysis processes by V-NLI with the aim of inspecting visual features through a visualization-oriented human-machine dialogue, as studied by Turkay and Henkin (2018); Aurisano et al. (2015). In contrast to visualization creation (see section 3.4), where visualizations are generated based on natural language text, the manipulation or composition of a visualization in the query dialog is used in the sense of a speech act. The produced or manipulated visualization can be seen here as a dynamically generated visual response to a user query with the goal of providing information in the dialog. Setlur and Tory (2017); Hoque et al. (2018) apply pragmatics to visualization-oriented dialogue modeling by taking the dialogue history into account for computing more adequate future responses. Visualization-oriented dialogue assistants
have been developed in various forms. General-purpose assistants for driving a visual analytics conversation are proposed by Fast et al. (2018); Kassel and Rohs (2018). Assistants implementing instruction following as in plot manipulation or navigation scenarios process and execute commands in a visualization environment (Shao and Nakashole, 2020; Wang et al., 2021). Multi-modal dialogue assistants combine natural language input in oral or written form with touch gestures (Srinivasan and Stasko, 2018; Kim et al., 2021c; Srinivasan et al., 2020a, 2021a). Sperrle et al. (2020, 2021) study adaptive guidance to support a visual analytics process. Collaborative approaches using mixed-initiative interactions for visual analytics are explored by Hu et al. (2018); Langevin et al. (2018). The potential of competitive visualization-oriented dialogue interfaces for educational purposes is theoretically investigated by Reicherts and Rogers (2020) by examining the role of questions in these dialogues. Kumar et al. (2020b) provide a data set of contextualized dialogue acts in a visual exploration scenario as a basis for training dialogue assistants.
Visual Question Answering. VQA is a well-studied task in Language & Vision (Antol et al., 2015; Yang et al., 2016; Anderson et al., 2018) with the goal to answer questions related to the visual content of images. In VIS, the aim is to answer complex questions related to visual models such as charts or scientific illustrations as in Singh and Shekhar (2020); Chaudhry et al. (2020). Infographics as sophisticated arrangements of visual elements and text are supported by VQA in Mathew et al. (2021). Meeting the high informative standards of response generation required to harness the explanatory purposes of visualizations presents itself as a challenging task.
Browsing. Browsing supports users with a vague or fuzzy data-related intent in discovering visualizations. The idea is to narrow down the user intent through language interaction using text input, multi-step questions, or dialogue and suggest appropriate next steps in the interaction with the visualization. Luo et al. (2018) use keyword input to execute personalized visualization recommendations. Other approaches leverage auto-completion in text input (Setlur et al., 2020; Dhamdhere et al., 2017) or use multi-step question procedures to restrict the user's target area (de Araújo Lima and Barbosa, 2020; Luo et al., 2020). Srinivasan and
Setlur (2021) recommend data-related utterances users can use to start a visual analysis or shimmy along. Lee et al. (2021) guide users through a visualization-oriented analytical conversation using insights found in the data similar to Cui et al. (2019).
3.3 Enjoy Task
Brehmer and Munzner (2013) consider enjoying as the 'casual encounter' with a visualization without having a concrete hypothesis to verify. Natural language enhances the perception of a visualization by displaying additional information such as captions that contextualize the visual experience, as applied in immersive experiences, exhibitions, or museums. Visually impaired people experience visualizations through translation into auditory language.
Augmentation. In augmentation, visualizations are complemented by automatically generated textual elements, such as labels or links. Srinivasan et al. (2019b); Hullman et al. (2013) augment visualizations with additional facts to substantiate the message to be transmitted. Kandogan (2012) propose the concept of just-in-time augmentation of visual structures during visual analytics to help users understand the structure of the data. Lai et al. (2020) automatically annotate visualizations based on their textual description. Lalle et al. (2021) highlight corresponding elements of a visualization based on tracked gazes of users as they read a text description associated with the visualization. In contrast to that Xia et al. (2020) augment audio podcasts with visual elements. Gao et al. (2014); Latif and Beck (2019) augment map visualizations by automatically mining and linking site-related facts out of articles to their location on the map. Augmentation is also used to textually describe GUI components automatically as a preliminary step for auditory scene description helping visually impaired people interact with visualizations (Chen et al., 2020a).
Visualization Description Generation. Textual descriptions for visualizations are created to complement visual elements during the encounter with a visualization. Spreafo and Carenini (2020); Qian et al. (2021b); Liu et al. (2020) complement visualizations with text analogously to image captioning (Vinyals et al., 2015; Xu et al., 2015). Murillo-Morales and Miesenberger (2020) generate auditory descriptions to make statistical charts accessible to visually impaired people. Hsu et al.
(2021) captions scientific illustrations with highly informative labels that meet scientific quality standards. Summarization of visualization content into textual form is researched by Demir et al. (2012); Moraes et al. (2014). Bylinskii et al. (2017); Madan et al. (2018) extend this to aggregating infographics into a single descriptive hashtag. Theoretical work on how charts and their descriptions are linked and verbalized by users is carried out in Kim et al. (2021a), where it is found that users tend to retain different amounts of information depending on how prominent the visual feature presented in the caption is.
3.4 Produce Task
Brehmer and Munzner (2013) refer to produce as a 'reference to tasks in which the intent is to generate new artifacts'. Artifacts generated through natural language interaction are, e.g., annotations of objects in a visualization, scene descriptions, or task reports as used, e.g., in medical visual analysis.