Venue: NAACL 2022

# The Why and The How: A Survey on Natural Language Interaction in Visualization

Henrik Voigt\({}^{1}\), Ozge Alacam\({}^{2}\), Monique Meuschke\({}^{3}\), Kai Lawonn\({}^{1}\) and Sina Zarriess\({}^{2}\)

\({}^{1}\)University of Jena \({}^{2}\)University of Bielefeld \({}^{3}\)University of Magdeburg

\({}^{1}\)first.last@uni-jena.de

\({}^{2}\)first.last@uni-bielefeld.de

\({}^{3}\)last@isg.cs.uni-magdeburg.de

###### Abstract

Natural language as a modality of interaction is becoming increasingly popular in the field of visualization. In addition to the popular query interfaces, other language-based interactions such as annotations, recommendations, explanations, or documentation experience growing interest. In this survey, we provide an overview of natural language-based interaction in the research area of visualization. We discuss a renowned taxonomy of visualization tasks and classify 119 related works to illustrate the state-of-the-art of how current natural language interfaces support their performance. We examine applied NLP methods and discuss human-machine dialogue structures with a focus on initiative, duration, and communicative functions in recent visualization-oriented dialogue interfaces. Based on this overview, we point out interesting areas for the future application of NLP methods in the field of visualization.

+
Footnote †: July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Natural language as a modality for interacting with visual models enjoys increasing popularity in human-computer interface research in the fields of Human-Computer-Interaction (HCI) and Visualization (VIS) [48, 67, 3]. At the same time, interest in tasks involving the visual modality has grown strongly in NLP research in recent years [46, 38, 12]. While there are common interests and parallel trends in VIS and NLP, research in these fields often adopts different perspectives on what interaction is and how it should be modeled. Broadly speaking, in the VIS community, a lot of research aims to understand **why** users need to interact with a visualization and what users' _intents_ are when they interact with a visual model [15]. Therefore, hehmer2013categorize users' data-related intents into _visualization tasks_ and introduce a taxonomy to describe them in general terms and compare them among applications. Recent contributions show that different forms of natural language-based interaction prove suitable to support users in accomplishing various visualization tasks. This concerns not only the popular query interfaces, but also, on a broader scale, the provision of recommendations, annotations, explanations, documentations, or for support in analytical conversation. However, the variety of existing visualization tasks benefiting from natural language interaction beyond simple query interfaces has not yet received much attention in the NLP community. On the other hand, modern NLP methods offer enormous potential for modeling multi-modal dialogues in the visualization tasks.

In this survey, we aim to complement the why-oriented perspective of classifying visualization tasks by intent in VIS with the how-oriented dialogue modeling perspective in NLP for works involving natural language interaction. To substantiate the classification of the papers, we first delimit the scope of the survey and explain the methodology employed to derive the selected papers in Section 2. We discuss the taxonomy of abstract visualization tasks by brehmer2013categorizing as the basis for the classification in Section 3 by explaining **why** interaction with a visualization is performed. Section 4 focuses on **how** interaction is implemented in the works at hand in terms of applied _NLP methods_ as well as characteristic _structures_ in human-machine dialogue [23]. Finally, challenges arising from current approaches are pointed out. As such, the need to compile reliable data sets to support the adoption of deep learning-based NLP methods in the field of VIS yields promising space for future creative work.



Related Surveys.The survey of Shen et al. (2021) considers 55 visualization-oriented natural language interfaces (V-NLI) at the intersection of NLP and VIS. The work focuses on the applicability of natural language interfaces at the different steps of the information visualization pipeline by Kard et al. (1999). The authors discuss query-based language interfaces in detail. Similarly, Ozcan et al. (2020) put their focus on querying data visualizations using natural language. However, querying a visualization interface is only one possible language-based interaction among many such as _annotation_, _description generation_, _documentation_, and others. Klopfenstein et al. (2017) conduct a more general study on the application of conversational interfaces and derive usage patterns and paradigms for their implementation. Further related work is done at the intersection of Machine Learning (ML) and VIS as by Wu et al. (2021) or Wang et al. (2020), who illuminate where and how ML gains ground in VIS and discuss future directions for applying ML in VIS research. Facing that, we identify substantial ground for a survey that examines current natural language interaction techniques supporting the accomplishment of visualization tasks. Related work in VIS yields comprehensive and well-conducted state-of-the-art surveys focusing on **where** dialogue systems can be integrated into the information visualization pipeline proposed by Kard et al. (1999). This work is complementary in that it illuminates from an NLP perspective **how** visualization-oriented dialogue is structured in terms of _initiative_, _duration_, and present _communicative functions_ within the respective visualization task at hand (see Figure 1). By shedding a light on this we hope to arouse interest in the NLP community for the interesting multi-modal dialogue modeling tasks emerging at the intersection of NLP and VIS.

## 2 Methodology

For a paper to be included in the survey paper selection, it must meet the following criteria:

* Language-based interaction must be a designated **input/output modality** and some kind of language interface must be provided for it, e.g., a text box or a microphone/speaker.
* Language-based interaction must serve to fulfill or support a main **visualization task**. For example, using natural language for _logging_ into an application is **neither** a visualization task **nor** does it support the accomplishment of that task and is, therefore, **not valid**. In contrast, using natural language to _annotate_ certain aspects of a visualization **is** considered supportive of achieving the goal of the visualization task and therefore **is valid**.

In addition to contributions that include concrete implementations of interaction scenarios, theoretical papers that discuss design spaces or considerations of language-based interaction possibilities are included. The aim is to explicitly show not only what has already been implemented, but also which interaction possibilities are conceivable and useful in multi-modal visualization-oriented dialogue. The paper selection is made in a two-stage process. First, a set of _seed papers_ is derived from conference proceedings of the main conferences in HCI, namely _SIGCHI_, VIS, namely _IEEE VIS, PacificVIS,_ and _EuroVIS,_ and NLP, namely _ACL_ and _EMNLP_, starting from the year 2010 until 2021. The papers are filtered using the keywords _language, visualization, interface_ in combination with a semantic embedding map of the abstracts based on Reimers and Gurevych (2019). The exploratory process results in a set of 76 papers. In the second stage, the references of the seed papers are examined and relevant papers that meet the specified

Figure 1: Investigation of dialogue structures as applied in visualization systems based on the DIT++ taxonomy of dialogue acts by Bunt (2009).

 

criteria are included in the set. This results in a final set of 119 papers. For a detailed insight into the scope of the survey, we refer to Appendix A.

## 3 Why Users interact with Visualizations

Brehmer and Munzner (2013) introduce a multi-level typology for abstract description and comparison of visualization tasks between applications. An abstract visualization task represents a high-level description of _why_ interaction with a visualization application is performed, _how_ it is performed, and _what_ the input and output of the task are. The _why_-branch of Brehmer and Munzner's typology was chosen primarily for three reasons: First, the high level of abstraction allows to cover a high number of visualization tasks and therefore ensures high representativeness. On the other hand, the modular character of the typology is beneficial for breaking down complex tasks into smaller subtasks in which commonalities can then be identified. In addition, the combination with the _what_ and the _how_ branch offers the possibility to describe task chains, which can serve as a blueprint for the design of a dialog with the system. The papers are classified on the basis of the **why**-branch of the taxonomy because it distinguishes the tasks taking into account the _goal_ to be achieved and thus corresponds to the _goal_ definition as also used in goal-oriented dialogue modeling (Bordes et al., 2016; Li et al., 2017; Liu et al., 2018). The why branch spawns the abstract visualization tasks _present, discover, enjoy_, and _produce_ illustrated in Figure 2. Following Munzner (2009), **we consider language-based interaction as domain- and interface-independent operations performed by users and/or systems by applying natural language in any kind of representation, e.g, written- or spoken text**. Table 1 shows an overview of the contributions and the respective visualization task to which they are assigned. In the following subsections, concrete tasks involving natural language interaction are presented for each abstract visualization task of the taxonomy. Each section includes a brief definition of the targeted visualization task and a detailed discussion of current related work that addresses it. For a detailed inspection, we refer to Appendix B.

### Present Task

Brehmer and Munzner (2013) define presentation as _'the use of visualization for the succinct communication of information, for telling a story with data, guiding an audience through a series of cognitive operations'_. During this task, natural language is used to complement the presentation of visual findings and results, for data-driven storytelling, or to explain, evaluate and discuss them.

Visual Storytelling.Visual storytelling considers the communication of knowledge to a broad audience using visual and textual elements that follow a coherent narrative. The main idea is to match linguistic and visual elements and arrange them consistently within a story. Von Landesberger et al. (2021) study how text and visualization interact with each other in a visual storytelling scenario pointing out that visualizations complement the narrative by providing overview, details, and comparison. Automatic story generation is done by Shi et al. (2021) who leverage the generation of a visual story from spreadsheet input. Natural language text drives the story of a visualization presentation in Kwon et al. (2014); Bryan et al. (2017); Metoyer et al. (2018). Users are guided through the story by interacting with the text segments and the system creates visual animations correspondingly in response.

Explanation Generation.Visualizations offer great potential to create understanding for complex issues among different user groups. In contrast to storytelling, explanation generation is not about assigning a sequence of visual elements to a text-based story, but about automatically explaining given visual facts through natural language texts. Combining text and visualization is used to explain complex processes, e.g., in verbalizing the functionality of ML models (Hohman et al., 2019).

Figure 2: **Why-branch of the Multi-Level Typology of Abstract Visualization Tasks inspired by Brehmer and Munzner (2013).

 

Sevastjanova et al. (2018) discuss strategies on how to present language explanations during the model inference process and the interaction techniques to be required, such as _details-on-demand, guidance, dialogue_, and _exploration_.

### Discover Task

Using natural language to discover information is one of the most common visualization tasks targeted by V-NLI. Brehmer and Munzner (2013) differentiate between different levels of task granularity such as _discover - search - query_ (see Figure 2). The discovery of concepts, objects, and relationships in a visualization depends on the role that the user and the interface take in the visualization-oriented dialogue, as well as on the concreteness of the user's intent. Intents are formulated in oral or written form. Less concrete user intents lead to a more exploratory character of the search. Concrete intents formalized in a query lead to a specific system response. vague and fuzzy intents are much more difficult to formalize in a single query and must be inferred by the V-NLI through the application of intelligent recommendations or user guidance.

KeywordSearch.Discovering information about a visualization by supplementing it with a keyword search interface is examined by Feng et al. (2018). The authors enable the search of visual concepts in a 2D visualization via text input. Further visualization-oriented keyword search interfaces are applied in Chowdhury et al. (2021); Siddiqui and Hoque (2020); Chung et al. (2010). In contrast to that, visual search interfaces take in keywords but focus on displaying results in a way that facilitates visual exploration, as targeted in Wilson et al. (2010); Schleussinger and Henkel (2018); Peltonen et al. (2017). Search history and coverage are tracked and visualized by Isaacs et al. (2014).

Natural Language Querying.Natural language querying is a scenario in which a user formulates a query to a visual model and the system is tasked with outputting a visual response to that query - referred to as _query2viz_. Most of the existing V-NLIs focus on this task. Theoretical work on utterance structures in natural language querying has been done by Srinivasan et al. (2019, 2021b) finding that utterances mainly target _attribute, chart type, encoding, aggregation_, and _design_ aspects of a visual model. Liu et al. (2021); Sun et al. (2014);

\begin{table}
\begin{tabular}{l l c p{156.5pt}} \hline
**Visualization Task** & **Subtask** & **\#** & **Representative Paper(s)** \\ \hline
**Present** & _Visual Storytelling_ & 6 & Kwon et al., 2014; Metoyer et al., 2018) \\  & _Explanation Generation_ & 3 & Sevastjanova et al., 2018; Hohman et al., 2019) \\ \hline
**Discover** & _Keyword Search_ & 15 & Isaaccs et al., 2014; Feng et al., 2018; Schleusinger and Henkel, 2018) \\  & _Querying_ & 45 & Setlur et al., 2016; Yu and Silva, 2020; Narechania et al., 2021) \\  & _VQA_ & 3 & Mathew et al., 2021; Chaudhry et al., 2020) \\  & _Browsing_ & 7 & Setlur et al., 2020; Luo et al., 2020; Srinivasan and Seltur, 2021) \\ \hline
**Enjoy** & _Augmentation_ & 12 & Srinivasan et al., 2019; Hullman et al., 2013) \\  & _Description Generation_ & 14 & Obeid and Hoque, 2020; Hsu et al., 2021) \\ \hline
**Produce** & _Annotation_ & 7 & Chen et al., 2010; Ren et al., 2017) \\  & _Documentation_ & 1 & Nafari and Weaver, 2015) \\  & _Visualization Creation_ & 6 & Cui et al., 2020; Fulda et al., 2016) \\ \hline \end{tabular}
\end{table}
Table 1: Classification of papers based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013). For the sake of clarity, papers are classified into the most suitable category only, although some works touch on several categories in terms of content. A comprehensive listing can be found in Appendix B.

 

Narechania et al. (2021) generate a visualization based on a data table and a natural language query. Yu and Silva (2020) allow query sequences to be specified in a visual exploration workflow.

_Ambiguities._ Resolving ambiguities and underspecified utterances poses a difficult problem in this visualization task, especially for single-turn query interfaces. Hearst et al. (2019); Tory and Setlur (2019) develop design guidelines for how systems should respond to queries that contain vague modifiers or -user intents by exploring contextual inference strategies. Gao et al. (2015) manage ambiguities in input utterances using visual ambiguity widgets. Setlur et al. (2019) apply inferencing rules based on known syntactic and semantic input structures. Setlur and Kumar (2020) use word co-occurrence in combination with sentiment analysis to determine data attributes and filter ranges associated with the articulated vague property.

_Hypothesis Verification._ Discovering novel insights from data is usually done by (dis-)validating hypotheses. Choi et al. (2019, 2018) study the use of visualizations to prove or disprove natural language hypotheses visually. The user initiates a hypothesis test by formulating it in natural language, and the system indicates the match with the underlying data set by creating a graph that highlights matches/discrepancies in striking green/red colours.

_Query Dialogue._Setlur et al. (2016); Aurisano et al. (2016); Bacci et al. (2020) extend the single-turn _query2viz_ interaction to a multi-turn interactive visual exploration also referred to as _analytical conversation_. Analytical conversation is the support of visual analysis processes by V-NLI with the aim of inspecting visual features through a visualization-oriented human-machine dialogue, as studied by Turkay and Henkin (2018); Aurisano et al. (2015). In contrast to _visualization creation_ (see section 3.4), where visualizations are generated based on natural language text, the manipulation or composition of a visualization in the _query dialog_ is used in the sense of a speech act. The produced or manipulated visualization can be seen here as a dynamically generated visual response to a user query with the goal of providing information in the dialog. Setlur and Tory (2017); Hoque et al. (2018) apply pragmatics to visualization-oriented dialogue modeling by taking the dialogue history into account for computing more adequate future responses. Visualization-oriented dialogue assistants have been developed in various forms. General-purpose assistants for driving a visual analytics conversation are proposed by Fast et al. (2018); Kassel and Rohs (2018). Assistants implementing instruction following as in plot manipulation or navigation scenarios process and execute commands in a visualization environment (Shao and Nakashole, 2020; Wang et al., 2021). Multi-modal dialogue assistants combine natural language input in oral or written form with touch gestures (Srinivasan and Stasko, 2018; Kim et al., 2021; Srinivasan et al., 2020, 2021). Sperrle et al. (2020, 2021) study adaptive guidance to support a visual analytics process. Collaborative approaches using mixed-initiative interactions for visual analytics are explored by Hu et al. (2018); Langevin et al. (2018). The potential of competitive visualization-oriented dialogue interfaces for educational purposes is theoretically investigated by Reicherts and Rogers (2020) by examining the role of questions in these dialogues. Kumar et al. (2020) provide a data set of contextualized dialogue acts in a visual exploration scenario as a basis for training dialogue assistants.

**Visual Question Answering.** VQA is a well-studied task in Language & Vision (Antol et al., 2015; Yang et al., 2016; Anderson et al., 2018) with the goal to answer questions related to the visual content of images. In VIS, the aim is to answer complex questions related to visual models such as charts or scientific illustrations as in Singh and Shekhar (2020); Chaudhry et al. (2020). Infographics as sophisticated arrangements of visual elements and text are supported by VQA in Mathew et al. (2021). Meeting the high informative standards of response generation required to harness the explanatory purposes of visualizations presents itself as a challenging task.

Browsing.Browsing supports users with a vague or fuzzy data-related intent in discovering visualizations. The idea is to narrow down the user intent through language interaction using text input, multi-step questions, or dialogue and suggest appropriate next steps in the interaction with the visualization. Luo et al. (2018) use keyword input to execute personalized _visualization recommendations_. Other approaches leverage auto-completion in text input (Setlur et al., 2020; Dhamdhere et al., 2017) or use multi-step question procedures to restrict the user's target area (de Araujo Lima and Barbosa, 2020; Luo et al., 2020). Srinivasan and 

Seltur (2021) recommend data-related utterances users can use to start a visual analysis or shimmy along. Lee et al. (2021) guide users through a visualization-oriented analytical conversation using insights found in the data similar to Cui et al. (2019).

### Enjoy Task

Brehmer and Munzner (2013) consider enjoying as the _'casual encounter'_ with a visualization without having a concrete hypothesis to verify. Natural language enhances the perception of a visualization by displaying additional information such as captions that contextualize the visual experience, as applied in immersive experiences, exhibitions, or museums. Visually impaired people experience visualizations through translation into auditory language.

Augmentation.In augmentation, visualizations are complemented by automatically generated textual elements, such as _labels_ or _links_. Srinivasan et al. (2019b); Hullman et al. (2013) augment visualizations with additional facts to substantiate the message to be transmitted. Kandogan (2012) propose the concept of just-in-time augmentation of visual structures during visual analytics to help users understand the structure of the data. Lai et al. (2020) automatically annotate visualizations based on their textual description. Lalle et al. (2021) highlight corresponding elements of a visualization based on tracked gazes of users as they read a text description associated with the visualization. In contrast to that Xia et al. (2020) augment audio podcasts with visual elements. Gao et al. (2014); Latif and Beck (2019) augment map visualizations by automatically mining and linking site-related facts out of articles to their location on the map. Augmentation is also used to textually describe GUI components automatically as a preliminary step for auditory scene description helping visually impaired people interact with visualizations (Chen et al., 2020a).

Visualization Description Generation.Textual descriptions for visualizations are created to complement visual elements during the encounter with a visualization. Spreafico and Carenini (2020); Qian et al. (2021b); Liu et al. (2020) complement visualizations with text analogously to image captioning (Vinyals et al., 2015; Xu et al., 2015). Murillo-Morales and Miesenberger (2020) generate auditory descriptions to make statistical charts accessible to visually impaired people. Hsu et al. (2021) captions scientific illustrations with highly informative labels that meet scientific quality standards. _Summarization_ of visualization content into textual form is researched by Demir et al. (2012); Moraes et al. (2014). Bylinskii et al. (2017); Madan et al. (2018) extend this to aggregating infographics into a single descriptive hashtag. Theoretical work on how charts and their descriptions are linked and verbalized by users is carried out in Kim et al. (2021a), where it is found that users tend to retain different amounts of information depending on how prominent the visual feature presented in the caption is.

### Produce Task

Brehmer and Munzner (2013) refer to produce as a _'reference to tasks in which the intent is to generate new artifacts'_. Artifacts generated through natural language interaction are, e.g., _annotations_ of objects in a visualization, _scene descriptions_, or _task reports_ as used, e.g., in medical visual analysis.

Annotation.Annotating areas of interest, comparing them among each other, and sharing them with colleagues is a common language interaction while working with visualizations (Ren et al., 2017). Chen et al. (2010a, b) leverage touch and click interactions for situated visualization annotation. Latif et al. (2018, 2021) explore the possibilities of linking text and visualization. Sperrle et al. (2019) study the visual annotation of argumentation and how this facilitates analysis. Theoretical work on the sustainable extraction of knowledge from visualization annotations is provided by Vanholst et al. (2021), who propose a classification framework that enables a structured capture and ordering of annotations.

Documentation.Visualization systems are used by experts, e.g., in the medical domain (Meuschke et al., 2021) to plan and discuss a surgery. Reporting, summarizing, and sharing this visualization-related work is an important task that is an additional burden to the surgeon and therefore should be executed by a machine. Nafari and Weaver (2013, 2015) generate natural language questions from queries executed on a visualization resulting in a natural language translation of the interaction. This leaves a step-by-step report of the interaction finding usage as a report of done work.

Visualization Creation.Visualization creation considers the production of a visual model from a 

natural language description - also referred to as _text2viz._ Rashid et al. (2021) generate chart visualizations from natural language text input. Collaborative authoring tools assisting users in visualization creation use natural language as an input modality. Cui et al. (2020) provide a tool that generates infographics using natural language statements as input, similar to Qian et al. (2021a). Fulda et al. (2016) design an interactive production process for generating timelines from unstructured text input. Language-based 3D scene generation, also referred to as _text2scene_, which allows users to describe 3D scenes using text without having to learn software tools, is investigated in Coyne and Sproat (2001); Coyne et al. (2012); Ulinski et al. (2018).

## 4 How Users interact with Visualizations

After discussing **why** users interact with visualizations using natural language, Section 4 provides a complementary discussion of **how** these interactions are modeled. First, in Section 4.1 it is explained which NLP methods are used in these systems. Subsequently, Section 4.2 summarizes the structure of the visualization-oriented dialogues in the analyzed paper set in terms of _initiative, duration_, and present _communication functions_ within the respective visualization task.

### NLP Methods

For each paper in the collection, both the NLP methods used, if any, and if named the specific NLP toolkits used for implementation are elaborated. For the sake of clarity, the methods are roughly divided into two areas: _Natural Language Understanding (NLU)_ and _Natural Language Generation (NLG)_. The majority of the systems apply standard NLP methods like _tokenization, stemming_ or _stopword removal_ to pre-process text inputs, which is why these are not recorded separately. For a detailed inspection, we refer to Appendix C. Figure 3 shows the distribution of applied NLU methods over all papers. _Semantic Parsing_, which relies on rule-based mapping procedures from recognized input tokens to semantic predicates, is predominantly used. Often, _POS-Tagging, Word Embeddings_, and _Named Entity Recognition (NER)_ are additionally applied to increase the accuracy of the mapping. For _Word Sense Disambiguation_ WordNet, VerbNet or ConceptNet are leveraged. _Speech-to-Text_ APIs are a common method used in many systems to enable auditory input. A small number of pioneering systems integrate more sophisticated NLP methods such as _Sentiment Analysis, Vector Search_ or _Co-Reference Resolution_. One main reason for the hesitant use of deep learning methods is the high demands on performance and robustness of visualization systems as Dhamdhere et al. (2017) points out. Fluid interaction between user and system in real time is a crucial factor for the success of a visualization application. Adopting state-of-the-art deep learning models to real-time interactions in visualization, e.g., by using _Knowledge Distillation_Hinton et al. (2015) or _Quantization_Jacob et al. (2018) leaves space for future work. Figure 4 shows the distribution of applied NLG methods over all papers. _Template-based_ language generation is used by the majority of the systems followed by a significantly smaller number of deep learning-based _Seq2Seq Modeling_ approaches. Multi-turn systems are predominantly based on rule-based or probabilistic _Dialogue Management_. Only a few systems use the _Text-to-Speech_ functionality, as most of the generated responses consist of visual elements. In order to advance the adoption of deep learning-based methods in visualization-related text generation, extensive training data sets are required, as pointed out by Kumar et al. (2020a). There is a

Figure 4: Distribution of NLG methods **over all papers**.

Figure 3: Distribution of NLU methods **over all papers**.

 

limited number of data sets for _Visualization Description Generation_(Obeid and Hoque, 2020), _Visual Question Answering_Mathew et al., 2021; Kim et al., 2020) and _Natural Language Querying_Fu et al., 2020; Srinivasan et al., 2021b; Luo et al., 2021). In particular, the compilation of data sets for emerging dialogue scenarios in _Analytical Conversation, Hypothesis Verification_ or collaborative authoring in _Visualization Creation_ would motivate the use of deep learning based NLP methods in these tasks. Therefore, generating high-quality data sets for the aforementioned visualization tasks leaves room for future work.

### Dialogue Structures

The study of structures in visualization-oriented dialogue is done with the idea of identifying task-specific patterns, as shown in Figure 5. The structural analysis is based on the work of Bunt (2009) and highlights, in particular, the _initiative, duration_, and _communicative functions_ present in the modeled dialogues. For each contribution that provides access to sample data illustrating human-machine dialogue within the paper or supplementary material, the presence of the communicative functions _information providing, information-seeking, commissive_, or _directive_ for the user and system is detected. A comprehensive list of allocations is presented in Appendix C. Bunt's DIT++ taxonomy was chosen due to the fact that it focuses on the **function** of the individual speech act. In the context of a visualization task, it is important which function a dialog act fulfills in the successful execution of this task. This manifests itself particularly in the design of dialog agents, where speech acts that are intended to help solve the task must be specified. Other taxonomies focus on the _rhetorical relations_ of speech acts to each other, as in Prasad et al. (2008), or the _emotional_ information a speech act conveys in the dialogue, as in EmotionML Schroder et al., 2011). In contrast to the aforementioned taxonomies, Bunt's taxonomy proves suitable in two respects: It allows to understand how current dialogue situations are functionally structured in the visualization context. In this way, patterns can be identified that are common for the respective visualization tasks. From the generation perspective, it allows to specify dialog actions that need to be prepared in certain visualization task contexts in order to support the solution of the visualization task.

Present Task.In _Visual Storytelling_, users initiate the interactions which are performed as a sequence of multiple turns triggered through the selection of text phrases. The human-machine dialogue is characterized by the actors complementing each other through text and visual animations as speech acts. Similar to storytelling, the human-machine dialogue in _Explanation Generation_ is characterized by a complementing of user input and visualization system output by matching visual and textual elements. Communicating insights found by investigating a visual model is little accompanied by NLP techniques so far compared to other visualization tasks. Template-based story generation systems leave room for innovation in grounding story segments in visualization elements.

Discover Task.The human-machine dialogue in _Keyword Search_ is a short, single-turn dialogue characterized by user-initiated text input that is reciprocated by a visual response from the system, similar to _Natural Language Querying_-Keyword-based visualization search relies on text label-only search without including the on-screen representation. This leaves space for grounding abstract visualization concepts like _outliers_ or _clusters_ in natural language as a promising step towards a generalized visualization search. Keyword search beyond 2D visualizations is an open issue. The modeled dialogue interaction in analytical conversation is a user-initiated dialogue with multiple turns. Visual elements function in the communication as information providers carrying the response. This scenario offers the possibility to apply modern NLP methods to multi-modal dialogues by checking the user's

Figure 5: Analysis of _initiative, duration_ and _communicative functions_ in the sub-task **browsing**.

 

in the _Wish Quality Question Answering_. The question 

Stefan Bieliauskas and Andreas Schreiber. 2017. A conversational user interface for software visualization. _2017 IEEE Working Conference on Software Visualization (VISSOFT)_, pages 139-143.
* Bordes et al. [2016] Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. _arXiv preprint arXiv:1605.07683_.
* Brehmer and Munzner [2013] Matthew Brehmer and Tamara Munzner. 2013. A multi-level typology of abstract visualization tasks. _IEEE Transactions on Visualization and Computer Graphics_, 19:2376-2385.
* Bryan et al. [2017] Chris Bryan, Kwan-Liu Ma, and Jonathan Woodring. 2017. Temporal summary images: An approach to narrative visualization via interactive annotation generation and placement. _IEEE Transactions on Visualization and Computer Graphics_, 23:511-520.
* Bunt [2009] Harry Bunt. 2009. The dit++ taxonomy for functional dialogue markup. In _AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts_, pages 13-24.
* Bunt et al. [2010] Harry Bunt, Jan Alexandersson, Jean Carletta, Jaewoong Choe, Alex Chengyu Fang, Koti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David R. Traum. 2010. Towards an iso standard for dialogue act annotation. In _LREC_.
* Bjkinskii et al. [2017] Zoya Bylinskii, Sami Alsheikh, Spandan Madan, Adria Recasens, Kimberli Zhong, Hanspeter Pfister, Fredo Durand, and Aude Oliva. 2017. Understanding infographics through textual and visual tag prediction. _ArXiv_, abs/1709.09215.
* Chaudhry et al. [2020a] Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020a. Leaf-qa: Locate, encode & attend for figure question answering. _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_, pages 3501-3510.
* Chen et al. [2020b] Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020a. Unblind your apps: Predicting natural language labels for mobile gui components by deep learning. _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_, pages 322-334.
* Chen et al. [2020b] Siming Chen, Jie Li, Gennady L. Andrienko, Natalia V. Andrienko, Yun Wang, Phong H. Nguyen, and Cagatay Turkaya. 2020b. Supporting story synthesis: Bridging the gap between visual analytics and storytelling. _IEEE Transactions on Visualization and Computer Graphics_, 26:2499-2516.
* Chen et al. [2010a] Yang Chen, Scott Barlowe, and Jing Yang. 2010a. Click2annotate: Automated insight externalization with rich semantics. _2010 IEEE Symposium on Visual Analytics Science and Technology_, pages 155-162.
* Chen et al. [2010b] Yang Chen, Jing Yang, Scott Barlowe, and Dong Hyun Jeong. 2010b. Touch2annotate: generating better annotations with less human effort on multi-touch interfaces. _CHI '10 Extended Abstracts on Human Factors in Computing Systems_.
* Chen et al. [2020c] Zhutian Chen, Wai-Shun Tong, Qianwen Wang, Benjamin Bach, and Huamin Qu. 2020c. Augmenting static visualizations with paparivis designer. _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_.
* Choi et al. [2019a] In Kwon Choi, Taylor Childers, Nirmal Kumar Raveendranath, Swati Mishra, Kyle Harris, and Khairi Reda. 2019a. Concept-driven visual analytics: an exploratory study of model- and hypothesis-based reasoning with visualizations. _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_.
* Choi et al. [2019b] In Kwon Choi, Nirmal Kumar Raveendranath, Jared Westerfield, and Khairi Reda. 2019b. Visual (dis) confirmation: Validating models and hypotheses with visualizations. In _2019 23rd International Conference in Information Visualization-Part II_, pages 116-121. IEEE.
* Choi et al. [2019c] Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019c. Visualizing for the non-visual: Enabling the visually impaired to use visualization. _Computer Graphics Forum_, 38.
* Choudhry et al. [2021] Arjun Choudhry, Mandar Sharma, Pramod Chundury, Thomas Kapler, Derek W. S. Gray, Naren Ramakrishnan, and Niklas Elmqvist. 2021. Once upon a time in visualization: Understanding the use of textual narratives for causality. _IEEE Transactions on Visualization and Computer Graphics_, 27:1332-1342.
* Chowdhury et al. [2021] Imran Chowdhury, Abdul Moeid, Enamul Hoque, Muhammad Ashad Kabir, Md. Sabir Hossain, and Mohammad Mainul Islam. 2021. Designing and evaluating multimodal interactions for facilitating visual analysis with dashboards. _IEEE Access_, 9:60-71.
* Chung et al. [2010] Haeyong Chung, Seungwon Yang, Naveed Massjouni, Christopher Andrews, Rahul Kanna, and Chris North. 2010. Vizcept: Supporting synchronous collaboration for constructing visualizations in intelligence analysis. _2010 IEEE Symposium on Visual Analytics Science and Technology_, pages 107-114.
* Coyne et al. [2012] Bob Coyne, Alex Klapheke, Masoud Rouhizadeh, Richard Sproat, and Daniel Bauer. 2012. Annotation tools and knowledge representation for a text-to-scene system. In _Proceedings of COLING 2012_, pages 679-694.
* Cyne and Sproat [2001] Bob Coyne and Richard Sproat. 2001. Wordseye: An automatic text-to-scene conversion system. In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_, pages 487-496.
* Crovari et al. [2020] Pietro Crovari, Sara Pido, Franca Garzolto, and Stefano Ceri. 2020. Show, don't tell. reflections on the design of multi-modal conversational interfaces. In _CONVERSTIONS_ 

Weiwei Cui, Xiaoyu Zhang, Yun Wang, He Huang, B. Chen, Lei Fang, Haidong Zhang, Jian-Guang Lou, and Dongmei Zhang. 2020. Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. _IEEE Transactions on Visualization and Computer Graphics_, 26:906-916.

* Lima and Barbosa (2020) Raul de Araijo Lima and Simone Diniz Junqueira Barbosa. 2020. A question-oriented visualization recommendation approach for data exploration. _Proceedings of the International Conference on Advanced Visual Interfaces_.
* Demir et al. (2012) Seniz Demir, Sandra Carberry, and Kathleen F. McCoy. 2012. Summarizing information graphics textually. _Computational Linguistics_, 38:527-574.

* Dhandhere et al. (2017) Kedar Dhandhere, Kevin S. McCurley, Ralfi Nahmias, Mukund Sundararajan, and Qiqi Yan. 2017. Analyza: Exploring data with conversation. _Proceedings of the 22nd International Conference on Intelligent User Interfaces_.
* Dimara and Perin (2020) Evanthia Dimara and Charles Perin. 2020. What is interaction for data visualization? _IEEE Transactions on Visualization and Computer Graphics_, 26:119-129.
* El-Assady et al. (2020) Mennatallah El-Assady, Rebecca Kehlbeck, Christopher M. Collins, Daniel A. Keim, and Oliver Deussen. 2020. Semantic concept spaces: Guided topic model refinement using word-embedding projections. _IEEE Transactions on Visualization and Computer Graphics_, 26:1001-1011.
* Fast et al. (2018) Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S. Bernstein. 2018. Iris: A conversational agent for complex tasks. _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_.
* Feng et al. (2018) Mi Feng, Cheng Deng, Evan M. Peck, and Lane Harrison. 2018. The effects of adding search functionality to interactive visualizations on the web. _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_.
* Fraser et al. (2020) C. Ailie Fraser, Julia M. Markel, N. James Basa, Mira Dontcheva, and Scott R. Klemmer. 2020. Remap: Lowering the barrier to help-seeking with multimodal search. _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_.
* Fu et al. (2020) Siwei Fu, Kai Xiong, Xiaodong Ge, Siliang Tang, Wei Chen, and Yingcai Wu. 2020. Quda: Natural language queries for visual data analytics. _arXiv preprint arXiv:2005.03257_.
* Fulda et al. (2016) Johanna Fulda, Matthew Brehmer, and Tamara Munzner. 2016. Timelinecurator: Interactive authoring of visual timelines from unstructured text. _IEEE Transactions on Visualization and Computer Graphics_, 22:300-309.
* Gao et al. (2015) Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. _Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology_.
* Gao et al. (2014) Tong Gao, Jessica R. Hullman, Eytan Adar, Brent J. Hecht, and Nicholas A. Diakopoulos. 2014. Newswisews: an automated pipeline for creating custom geovisualizations for news. _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_.
* Hearst and Tory (2019) Marti A. Hearst and Melanic K. Tory. 2019. Would you like a chart with that? incorporating visualizations into conversational interfaces. _2019 IEEE Visualization Conference (VIS)_, pages 1-5.
* Hearst et al. (2019) Marti A. Hearst, Melanie K. Tory, and Vidya Setlur. 2019. Toward interface defaults for vague modifiers in natural language interfaces for visual analysis. _2019 IEEE Visualization Conference (VIS)_, pages 21-25.
* Henkin and Turkay (2020) Rafael Henkin and Casatay Turkay. 2020. Words of estimative correlation: Studying verbalizations of scatterplots. _IEEE Transactions on Visualization and Computer Graphics_, PP.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. _arXiv preprint arXiv:1503.02531_.
* Hohman et al. (2019) Fred Hohman, Arjun Srinivasan, and Steven Mark Drucker. 2019. Telegan: Combining visualization and verbalization for interpretable machine learning. _2019 IEEE Visualization Conference (VIS)_, pages 151-155.
* Hoque et al. (2018) Enamul Hoque, Vidya Setlur, Melanie K. Tory, and Isaac Dykeman. 2018. Applying pragmatics principles for interaction with visual analytics. _IEEE Transactions on Visualization and Computer Graphics_, 24:309-318.
* Hsu et al. (2021) Ting-Yao Hsu, C. Lee Giles, and Ting-Hao Kenneth Huang. 2021. Sciccap: Generating captions for scientific figures. In _EMNLP_.
* Hu et al. (2018) Kevin Zeng Hu, Diana Orghian, and Cesar A. Hidalgo. 2018. Dive: A mixed-initiative system supporting integrated data exploration workflows. _Proceedings of the Workshop on Human-In-the-Loop Data Analytics_.

 

Drew A Hudson and Christopher D Manning. 2019. Gqa: a new dataset for compositional question answering over real-world images. _arXiv preprint arXiv:1902.09506_, 3(8).
* Hullman et al. [2013] Jessica R. Hullman, Nicholas A. Diakopoulos, and Eytan Adar. 2013. Contextifier: automatic generation of annotated stock visualizations. _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_.
* Isaacs et al. [2014] Ellen Isaacs, Kelly Domico, Shane Ahern, Eugene Bart, and Mudita Singhal. 2014. Footprints: A visual search tool that supports discovery and coverage tracking. _IEEE Transactions on Visualization and Computer Graphics_, 20:1793-1802.
* Jacob et al. [2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 2704-2713.
* John et al. [2017] Rogers Jeffrey Leo John, Navneet Potti, and Jignesh M. Patel. 2017. Ava: From data to insights through conversations. In _CIDR_.
* Jung et al. [2021] Crescentia Jung, Shubham Mehta, Athara Kulkarni, Yuhang Zhao, and Yea-Seul Kim. 2021. Communicating visualizations without visuals: Investigation of visualization alternative text for people with visual impairments. _IEEE Transactions on Visualization and Computer Graphics_, PP.
* Kandogan [2012] Eser Kandogan. 2012. Just-in-time annotation of clusters, outliers, and trends in point-based data visualizations. _2012 IEEE Conference on Visual Analytics Science and Technology (VAST)_, pages 73-82.
* Kard et al. [1999] Stuart T Kard, Jock D Mackinlay, and Ben Scheiderman. 1999. _Readings in Information Visualization, using vision to think_. San Francisco: Morgan Kaufmann.
* Kassel and Rohs [2018] Jan-Frederik Kassel and Michael Rohs. 2018. Valletto: A multimodal interface for ubiquitous visual analytics. _Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems_.
* Kassel and Rohs [2019] Jan-Frederik Kassel and Michael Rohs. 2019. Talk to me intelligibly: Investigating an answer space to match the user's language in visual analysis. _Proceedings of the 2019 on Designing Interactive Systems Conference_.
* Kim et al. [2020] Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020. Answering questions about charts and generating visual explanations. In _Proceedings of the 2020 CHI conference on human factors in computing systems_, pages 1-13.
* Kim et al. [2021a] Dae Hyun Kim, Vidya Setlur, and Maneesh Agrawala. 2021a. Towards understanding how readers integrate charts and captions: A case study with line charts. _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_.
* Kim et al. [2021b] Nam Wook Kim, Shkila Cherise Joyner, Amalia Riegelhuth, and Yea-Seul Kim. 2021b. Accessible visualization: Design space, opportunities, and challenges. _Computer Graphics Forum_, 40.
* Kim et al. [2021c] Young-Ho Kim, Bongshin Lee, Arjun Srinivasan, and Eun Kyoung Choe. 2021c. Data@hand: Fostering visual exploration of personal data on smartphones leveraging speech and touch interaction. _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_.
* Klopfenstein et al. [2017] Lorenz Cuno Klopfenstein, Saverio Delpriori, Silvia Malatini, and Alessandro Bogliolo. 2017. The rise of bots: A survey of conversational interfaces, patterns, and paradigms. _Proceedings of the 2017 Conference on Designing Interactive Systems_.
* Kumar et al. [2020a] Abhinav Kumar, Jillian Aurisano, Barbara Di Eugenio, and Andrew Johnson. 2020a. Intelligent assistant for exploring data visualizations. In _The Thirty-Third International Flairs Conference_.
* Kumar et al. [2017] Abhinav Kumar, Jillian Anirsano, Barbara Maria Di Eugenio, Andrew E. Johnson, Abeer Alsaieri, Nigel Flowers, Alberto Gonzalez, and Jason Leigh. 2017. Multimodal coreference resolution for exploratory data visualization dialogue: Context-based annotation and gesture identification.
* Kumar et al. [2020b] Abhinav Kumar, Barbara Di Eugenio, Jillian Aurisano, and Andrew Johnson. 2020b. Augmenting small data to classify contextualized dialogue acts for exploratory visualization. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 590-599.
* Kwon et al. [2014] Burn Chul Kwon, Florian Stoffel, Dominik Jackle, Bongshin Lee, and Daniel Keim. 2014. Visjockey: Enriching data stories through orchestrated interactive visualization. In _Poster compendium of the computation+ journalism symposium_, volume 3, page 3.
* Lai et al. [2020]Chufan Lai, Zhixian Lin, Ruike Jiang, Yun Han, Can Liu, and Xiaoru Yuan. 2020. Automatic annotation synchronizing with textual description for visualization. _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_.
* Lalle et al. [2021] Sebastien Lalle, Dereck Toker, and Cristina Conati. 2021. Gaze-driven adaptive interventions for magazine-style narrative visualizations. _IEEE Transactions on Visualization and Computer Graphics_, 27:2941-2952.
* Langevin et al. [2018] Scott Langevin, David Jonker, Christopher Bethune, Glen Coppersmith, Casey Hilland, Jonathon Morgan, Paul Azunre, and Justin Gawrilow. 2018. Distil: A mixed-initiative model discovery system for subject matter experts. In _International Conference on Machine Learning AutoML Workshop_.
* Latif and Beck [2019] Shahid Latif and Fabian Beck. 2019. Interactive map reports summarizing bivariate geographic data. _Vis. Informatics_, 3:27-37.

 

Shahid Latif, Diao Liu, and Fabian Beck. 2018. Exploring interactive linking between text and visualization. In _EuroVis (Short Papers)_, pages 91-94.
* Latif et al. [2021] Shahid Latif, Zheng Zhou, Yoon Kim, Fabian Beck, and Nam Wook Kim. 2021. Kori: Interactive synthesis of text and charts in data documents. _IEEE Transactions on Visualization and Computer Graphics_.
* Laurrence and Riezler [2016] Carolin (Haas) Lawrence and Stefan Riezler. 2016. Nlmaps: A natural language interface to query openstreetmap. In _COLING_.
* Lee and Parameswaran [2018] Doris Jung Lin Lee and Aditya G. Parameswaran. 2018. The case for a visual discovery assistant: A holistic solution for accelerating visual data exploration. _IEEE Data Eng. Bull._, 41:3-14.
* Lee et al. [2021] Doris Jung Lin Lee, Abdul Quamar, Eser Kandogan, and Fatma Ozcan. 2021. Boomerang: Proactive insight-based recommendations for guiding conversational data analysis. _Proceedings of the 2021 International Conference on Management of Data_.
* Li et al. [2017] Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asil Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. _arXiv preprint arXiv:1703.01008_.
* Litman and Pan [2004] Diane J. Litman and Shimei Pan. 2004. Designing and evaluating an adaptive spoken dialogue system. _User Modeling and User-Adapted Interaction_, 12:111-137.
* Liu et al. [2018] Bing Liu, Gokhun Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. _arXiv preprint arXiv:1804.06512_.
* Liu et al. [2021] Can Liu, Yun Han, Ruike Jiang, and Xiaoru Yuan. 2021. Advvisor: Automatic visualization answer for natural-language question on tabular data. _2021 IEEE 14th Pacific Visualization Symposium (PacificVis)_, pages 11-20.
* Liu et al. [2020] Can Liu, Liwenhan Xie, Yun Han, Datong Wei, and Xiaoru Yuan. 2020. Autocaption: An approach to generate natural language description from visualization automatically. _2020 IEEE Pacific Visualization Symposium (PacificVis)_, pages 191-195.
* Lundgard and Satyanarayan [2021] Alan Lundgard and Arvind Satyanarayan. 2021. Accessible visualization via natural language descriptions: A four-level model of semantic content. _IEEE Transactions on Visualization and Computer Graphics_, PP.
* Luo et al. [2020] Yuyu Luo, Chengliang Chai, Xuedi Qin, Nan Tang, and Guoliang Li. 2020. Interactive cleaning for progressive visualization through composite questions. _2020 IEEE 36th International Conference on Data Engineering (ICDE)_, pages 733-744.
* Luo et al. [2018] Yuyu Luo, Xuedi Qin, Nan Tang, Gioliang Li, and Xinran Wang. 2018. Deepeye: Creating good data visualizations by keyword search. _Proceedings of the 2018 International Conference on Management of Data_.
* Luo et al. [2021] Yuyu Luo, Nan Tang, Guoliang Li, Chengliang Chai, Wenbo Li, and Xuedi Qin. 2021. Synthesizing natural language to visualization (nl2vis) benchmarks from nl2sql benchmarks. In _Proceedings of the 2021 International Conference on Management of Data_, pages 1235-1247.
* Madan et al. [2018] Spandan Madan, Zoya Bylinskii, Matthew Tancik, Adria Recasens, Kimberli Zhong, Sami Alsheikh, Hanspeter Pfister, Aude Oliva, and Fredo Durand. 2018. Synthetically trained icon proposals for parsing and summarizing infographics. _ArXiv_, abs/1807.10441.
* Manuvinakurike et al. [2018] Ramesh Radhakrishna Manuvinakurike, Trung Bui, W. Chang, and Kallirroi Georgila. 2018. Conversational image editing: Incremental intent identification in a new dialogue task. In _SIGDIAL Conference_.
* Mathew et al. [2021] Minesh Mathew, Viraj Bagal, Ruben Perez Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. 2021. Infographicvqa. _ArXiv_, abs/2104.12756.
* Mazumder and Riva [2021] S. Mazumder and Oriana Riva. 2021. Flini: A flexible natural language interface for web navigation. _ArXiv_, abs/2010.12844.
* McTear [2002] Michael F. McTear. 2002. Spoken dialogue technology: enabling the conversational user interface. _ACM Comput. Surv._, 34:90-169.
* Metoyer et al. [2018] Ronald A. Metoyer, Qiyu Zhi, Bart Janczuk, and Walter J. Scheirer. 2018. Coupling story to visualization: Using textual analysis as a bridge between data and interpretation. _23rd International Conference on Intelligent User Interfaces_.
* Meuschke et al. [2021] Monique Meuschke, Bernhard Preim, and Kai Lawonn. 2021. Aneulysis-a system for the visual analysis of aneurysm data. _Computers & Graphics_, 98:197-209.
* Moranes et al. [2014] Priscilla Moranes, Gabriel Sina, Kathby McCoy, and Sandra Carberry. 2014. Generating summaries of line graphs. In _Proceedings of the 8th International Natural Language Generation Conference (INLG)_, pages 95-98.
* Munzner [2009] Tamara Munzner. 2009. A nested model for visualization design and validation. _IEEE Transactions on Visualization and Computer Graphics_, 15(6):921-928.

* Nafari and Weaver [2013] Maryam Nafari and Chris Weaver. 2013. Augmenting visualization with natural language translation of interaction: A usability study. In _Computer Graphics Forum_, volume 32, pages 391-400. Wiley Online Library 

Maryam Nafari and Chris Weaver. 2015: Query2question: Translating visualization interaction into natural language. IEEE Transactions on Visualization and Computer Graphics21, pp. 756-769. Cited by: SS4.
* A. Narechania, Arjun Srinivasan, and J. T. Stasko (2021)Nl4dv: a toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics27, pp. 369-379. Cited by: SS4.
* J. Obeid and E. Hoque (2020)Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In INLG, Cited by: SS4.
* F. Ozcan, A. Quamar, J. Sen, C. Lei, and V. Efthymiou (2020)State of the art and open challenges in natural language interfaces to data. Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. Cited by: SS4.
* J. Peltonen, K. Belorustceva, and T. Ruotsalo (2017)Topic-relevance map: visualization for improving search result comprehension. Proceedings of the 22nd International Conference on Intelligent User Interfaces. Cited by: SS4.
* R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. Joshi, and B. Webber (2008)The penn discourse treebank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Cited by: SS4.
* C. Qian, S. Sun, W. Cui, J. Lou, H. Zhang, and D. Zhang (2021a)Retrieve-then-adapt: example-based automatic generation for proportion-related infographics. IEEE Transactions on Visualization and Computer Graphics27, pp. 443-452. Cited by: SS4.
* X. Qian, E. Koh, F. Du, S. Kim, J. Chan, R. A. Rossi, S. Malik, and T. Lee (2021b)Generating accurate caption units for figure captioning. Proceedings of the Web Conference. Cited by: SS4.
* M. Mahinur Rashid, H. Kawsar Johan, A. Huzzat, R. Ahmed Rahul, T. Bin Zakir, Farhana Meem, Md. Sadam Hossain Mukta, and Swakhkar Shatabda (2021)Text2chart: a multi-staged chart generator from natural language text. ArXivabs/2104.04584. Cited by: SS4.
* L. Reicherts and Y. Rogers (2020)Do make me think! how cuis can support cognitive processes. In Proceedings of the 2nd Conference on Conversational User Interfaces, pp. 1-4. Cited by: SS4.
* N. Reiners and I. Gurevych (2019)Sentence-bert: sentence embeddings using siamese bert-networks. ArXivabs/1908.10084. Cited by: SS4.
* D. Ren, M. Brehmer, B. Lee, T. Hollerer, and E. C. Choe (2017)Chatreccent: annotation for data-driven storytelling. 2017 IEEE Pacific Visualization Symposium (PacificVis), pp. 230-239. External Links: Document, ISSN 1574-1227 Cited by: SS4.
* M. Schleussinger and M. Henkel (2018)Knowde: a visual search interface. In International Conference on Human-Computer Interaction, pp. 191-198. Cited by: SS4.
* M. Schroder, P. Baggia, F. Burkhardt, C. Pelachaud, C. Peter, and E. Zovato (2011)Emotionml-an upcoming standard for representing emotions and related states. In International Conference on Affective Computing and Intelligent Interaction, pp. 316-325. Cited by: SS4.
* P. Seipel, A. Stock, S. priya Santhanam, Artur Baranowski, N. Hochgeschwender, and Andreas Schreiber (2019)Speak to your software visualization--exploring component-based software architectures in augmented reality with a conversational interface. 2019 Working Conference on Software Visualization (VISSOFT), pp. 78-82. Cited by: SS4.
* V. Setluer, S. E. Batterby, M. K. Tory, R. Gossweiler, and A. X. Chang (2016)Eviza: a natural language interface for visual analysis. Proceedings of the 29th Annual Symposium on User Interface Software and Technology. Cited by: SS4.
* V. Setluer, E. Hoque, D. Hyun Kim, and Angel X. Chang (2020)Sneak picture: exploring auto-completion as a data discovery scaffold for supporting visual analysis. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. Cited by: SS4.
* V. Setluer and A. Kumar (2020)Semtifiers: interpreting vague intent modifiers in visual analysis using word co-occurrence and sentiment analysis. 2020 IEEE Visualization Conference (VIS), pp. 216-220. Cited by: SS4.
* V. Setluer and M. K. Tory (2017)Exploring synergies between visual analytical flow and language pragmatics. In AAAI Spring Symposia, Cited by: SS4.
* V. Setluer, M. K. Tory, and A. Djalali (2019)Inferencing underspecified natural language utterances in visual analysis. Proceedings of the 24th International Conference on Intelligent User Interfaces. Cited by: SS4.
* R. Sevastjanova, F. Beck, B. Ell, C. Turkay, R. Henkin, M. Butt, D. A. Keim, and M. El-Assady (2018)Going beyond visualization: verbalization as complementary medium to explain machine learning models. In Workshop on Visualization for AI Explainability at IEEE VIS, Cited by: SS4.
* Y. Shao and N. Nakashole (2020)Chartedilogs: plotting from natural language instructions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3559-3574.

 

Leixian Shen, Enya Shen, Yuyu Luo, Xiaocong Yang, Xuming Hu, Xiongshuai Zhang, Zhiwei Tai, and Jianmin Wang. 2021. Towards natural language interfaces for data visualization: A survey. ArXivabs/2109.03506. External Links: Link Cited by: SS4.
* D. Shi, X. Xu, F. Sun, Y. Shi, and N. Cao (2021)Calliope: automatic visual data story generation from a spreadsheet. IEEE Transactions on Visualization and Computer Graphics27, pp. 453-463. External Links: Document, ISSN 1878-0226, Link Cited by: SS4.
* N. Siddiqui and E. Hoque (2020)Convisqa: a natural language interface for visually exploring online conversations. 2020 24th International Conference Information Visualisation (IV), pp. 440-447. External Links: Link Cited by: SS4.
* T. Adnan Siddiqui (2021)From sketching to natural language: expressive visual querying for accelerating insight. External Links: Link Cited by: SS4.
* H. Singh and S. Shekhar (2020)Stl-cqa: structure-based transformers with localization and encoding for chart question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3275-3284. External Links: Link Cited by: SS4.
* F. Sperrle, A. Jeitler, J. Bernard, D. A. Keim, and M. El-Assady (2020)Learning and teaching in co-adaptive guidance for mixed-initiative visual analytics. In EuroVA@ Eurographics/EuroVis, Cited by: SS4.
* F. Sperrle, H. Schafer, D. Keim, and M. El-Assady. (2021)Learning contextualized user preferences for co-adaptive guidance in mixed-initiative topic model refinement. In Computer Graphics Forum, Vol. 40, Online Library, Wiley Online Library, Cited by: SS4.
* F. Sperrle, R. Sevastjanova, R. Keshbeck, and M. El-Assady. (2019)Viana: visual interactive annotation of argumentation. 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 11-22. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS4.
* A. Spreafico and G. Carenini (2020)Neural data-driven captioning of time-series line charts. In Proceedings of the International Conference on Advanced Visual Interfaces, pp. 1-5. External Links: Link Cited by: SS4.
* A. Srinivasan, M. Dontheva, E. Adar, and S. Walker (2019a)Discovering natural language commands in multimodal interfaces. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pp. 661-672. External Links: Link Cited by: SS4.
* A. Srinivasan, S. M. Drucker, A. Endert, and J. T. Stasko (2019b)Augmenting visualizations with interactive data facts to facilitate interpretation and communication. IEEE Transactions on Visualization and Computer Graphics25, pp. 672-681. External Links: Document, ISSN 1558-1226, Link Cited by: SS4.
* A. Srinivasan, B. Lee, N. Henry Riche, S. M. Drucker, and K. Hinckley (2020)InconTraus: designing consistent multimodal interactions for data visualization on tablet devices. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-13. External Links: Link Cited by: SS4.
* A. Srinivasan, W. Nyapathy, B. Lee, S. M. Drucker, and J. Stasko (2021)Collecting and characterizing natural language utterances for specifying data visualizations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-10. External Links: Link Cited by: SS4.
* A. Srinivasan and V. Setluz (2021)Snowy: recommending utterances for conversational visual analysis.. The 34th Annual ACM Symposium on User Interface Software and Technology, pp. . External Links: Link Cited by: SS4.
* A. Srinivasan and J. T. Stasko (2017)Natural language interfaces for data analysis with visualization: considering what has and could be asked. In EuroVis, Cited by: SS4.
* A. Srinivasan and J. T. Stasko (2018)Orko: facilitating multimodal interaction for visual exploration and analysis of networks. IEEE Transactions on Visualization and Computer Graphics24, pp. 511-521. External Links: Document, ISSN 1558-1226, Link Cited by: SS4.
* A. Srinivasan, J. T. Stasko, D. F. Keefe, and M. K. Toroy (2020)How to ask what to say?: strategies for evaluating natural language interfaces for data visualization. IEEE Computer Graphics and Applications40, pp. 96-103. External Links: Document, ISSN 1558-1226, Link Cited by: SS4.
* A. Srinivasan, M. Lewis, J. Yeh, and Y. Artzi (2017)A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217-223. External Links: Link Cited by: SS4.
* Y. Sun, J. Leigh, A. Johnson, and B. Di Eugenio (2014)Articulate: creating meaningful visualizations from natural language. In Innovative Approaches of Data Visualization and Visual Analytics, IGI Global, Cited by: SS4.
* M. K. Tory and V. Setluz (2019)Do what i mean, not what is say! design considerations for supporting intent and context in analytical conversation. 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 93-103. External Links: Document, ISSN 1558-1226, Link Cited by: SS4.
* C. Turkay and R. Henkin (2018)Towards natural language empowered interactive data analysis. In EuroVis, Cited by: SS4.
* M. Ulinski, B. Coyne, and J. Hirschberg (2018)Evaluating the wordseye text-to-scene system: imaginative and realistic sentences. In Proceedings of 

the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Vanhulst et al. [2021] Pierre Vanhulst, Raphael Tuor, Florian Evequoz, and Denis Lalanne. 2021. Colvis--a structured annotation acquisition system for data visualization. _Information_, 12(4):158.
* Vinyals et al. [2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and D. Erhan. 2015. Show and tell: A neural image caption generator. _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 3156-3164.
* van Landesberger et al. [2021] Tatiana von Landesberger, Shahid Latif, Siming Chen, and Fabian Beck. 2021. A deeper understanding of visualization-text interplay in geographic data-driven stories. _Computer Graphics Forum_, 40.
* Wang et al. [2020] Qianwen Wang, Zhutian Chen, Yong Wang, and Huamin Qu. 2020. Applying machine learning advances to data visualization: A survey on ml4vis. _ArXiv_, abs/2012.00467.
* Wang et al. [2021] Yihan Wang, Yutong Shao, and Ndapandula Nakashole. 2021. Interactive plot manipulation using natural language. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations_, pages 92-98.
* Wilson et al. [2010] Max L. Wilson, Bill Kules, M. C. Schraefel, and Ben Shneiderman. 2010. From keyword search to exploration: Designing future search interfaces for the web. _Found. Trends Web Sci._, 2:1-97.
* Wu et al. [2021] Aoyu Wu, Yun Wang, Xinhuan Shu, Dominik Moritz, Weiwei Cui, Haidong Zhang, Dongmei Zhang, and Huamin Qu. 2021. Ai4vis: Survey on artificial intelligence approaches for data visualization. _IEEE Transactions on Visualization and Computer Graphics_, PP.
* Xia [2020] Haijun Xia. 2020. Crosspower: Bridging graphics and linguistics. _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_.
* Xia et al. [2020] Haijun Xia, Jennifer Jacobs, and Maneesh Agrawala. 2020. Crosscast: Adding visuals to audio travel podcasts. _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_.
* Xu et al. [2015] Ke Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In _ICML_.
* Yang et al. [2016] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. 2016. Stacked attention networks for image question answering. _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 21-29.
* Yu and Silva [2020] Bowen Yu and Claudio T. Silva. 2020. Flowsense: A natural language interface for visual data exploration within a dataflow system. _IEEE Transactions on Visualization and Computer Graphics_, 26:1-11.

## Appendix A Scope

Research on visualization-oriented natural language-based interaction is conducted in the VIS, HCI, and NLP communities. For providing an overview of the number of selected contributions per community, the selection set is grouped based on publication venues related to their respective community. Important related work with high subject relevance being derived from other sources is subsumed in the category _Miscellaneous_. The time span of surveyed works is restricted to be between _2010_ and _2021_. Next to application papers implementing human-machine interaction theoretical works related to language-based interaction modeling are explicitly included. Table 2 shows the distribution of contributions over community-related venues.

The survey is targeted to touch the intersection of the domains of NLP, HCI and VIS. The scope is defined to reflect on how interaction is modeled and implemented in the different communities as well as to point out how researchers combine different ideas originating from the three fields into work that can be deposited in the intersection of them. For the VIS domain, the most common venues are _IEEE Transactions on Visualization and Computer Graphics_ (19) and _Computer Graphics Forum_ (5) containing work that is mostly specialized on query-based natural language interfaces. The area of HCI is ostensibly represented by the venue of the _Conference on Human Factors in Computing Systems (SIGCHI)_ (14) originating works that consider the interaction aspect and focus on language as a tool that transmits information for reaching a goal. In the NLP domain, the most frequent venue is the _Annual Meeting of the Association for Computational Linguistics (ACL)_ (3) including works less

\begin{table}
\begin{tabular}{c c} \hline Venues & Papers \\ \hline Visualization (VIS) & 49 \\ Human-Computer-Interaction (HCI) & 27 \\ Natural Language Processing (NLP) & 9 \\ Miscellaneous & 34 \\
**Total** & **119** \\ \hline \end{tabular}
\end{table}
Table 2: Venues of related work 

visualization-related focusing to a large extent on the dialogue modeling.

## Appendix B Classification

The contributions are classified based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013). Figure 6 illustrates a distribution of papers over the abstract visualization tasks. The visualization task accommodating the highest number of works considering natural language-based interaction is the task _discover_ (70), followed by _enjoy_ (26). Contributions supporting _present_ (9) and _produce_ (14) tasks are less frequent.

Figure 7 shows the distribution of papers over the inner class sub-tasks. For sake of simplicity, papers are categorized into the single most suitable category only, although some works touch on several categories. _Natural Language Querying_ (45) is by far the sub-task with the highest amount of contributions followed by _Keyword Search_ (15) and _Visualization Description Generation_ (14). Less frequently studied tasks are _Explanation Generation_ (3), _Visual Question Answering (VQA)_ (3), and _Documentation_ (1).

Table 3 contains a comprehensive listing of the classification of the single contributions into the taxonomy by Brehmer and Munzner (2013).

Figure 6: Distribution of papers **over tasks**.

Figure 7: Distribution of papers **over sub-tasks**.

 



\begin{table}
\begin{tabular}{|l|c|p{142.3pt}|} \hline
**Visualization Task** & **Subtask** & **References** \\ \hline
**Discover** & _Keyword Search_ & Feng et al. (2018); Chowdhury et al. (2021); Chung et al. (2010); Fraser et al. (2020); Wilson et al. (2010); Schleussinger and Henkel (2018); Isaacs et al. (2014); Peltonen et al. (2017); Siddiqui and Hoque (2020) \\  & _VQA_ & Singh and Shekhar (2020); Mathew et al. (2021); Chaudhry et al. (2020) \\  & _Querying_ & Srinivasan et al. (2020b); Srinivasan and Stasko (2017); Kassel and Rohs (2019); Crovari et al. (2020); Tory and Setlur (2019); Hearst and Tory (2019); Liu et al. (2021); Hoque et al. (2018); Setlur and Tory (2017); Siddiqui (2021); Bacci et al. (2020); Narechania et al. (2021); Yu and Silva (2020); Setlur et al. (2016); Sun et al. (2014); Aurisano et al. (2016); Srinivasan et al. (2021, 2019a); Gao et al. (2015); Setlur et al. (2019); Hearst et al. (2019); Setlur and Kumar (2020); Choi et al. (2019b, a); Sperrle et al. (2020, 2021); El-Assady et al. (2020); Kumar et al. (2020a); Avirsano et al. (2015); Turkay and Henkin (2018); Mazumder and Riva (2021); Lawrence and Riezler (2016); Shao and Nakashole (2020); Wang et al. (2021); Manuvinakurike et al. (2018); Fast et al. (2018); Kassel and Rohs (2018); Bieliauskas and Schreiber (2017); Seipel et al. (2019); Lee and Parameswaran (2018); Kumar et al. (2020b); Srinivasan and Stasko (2018); Kim et al. (2021c); Srinivasan et al. (2020a); Kumar et al. (2017); Srinivasan et al. (2021a); Hu et al. (2018); Langevin et al. (2018); Reicherts and Rogers (2020); John et al. (2017) \\  & _Browsing_ & Setlur et al. (2020); Lee et al. (2021); Luo et al. (2018); de Araújo Lima and Barbosa (2020); Luo et al. (2020); Srinivasan and Setlur (2021); Cui et al. (2019); Dhamdhere et al. (2017) \\ \hline
**Enjoy** & _Augmentation_ & Srinivasan et al. (2019b); Hullman et al. (2013); Xia et al. (2020); Gao et al. (2014); Kandogan (2012); Chen et al. (2020c, a); Lai et al. (2020); Bylinskii et al. (2017); Madan et al. (2018); Lalle et al. (2021); Latif and Beck (2019) \\  & _Description Generation_ & Demir et al. (2012); Moraes et al. (2014); Spreafico and Carenini (2020); Qian et al. (2021b); Murillo-Morales and Miesenberger (2020); Kim et al. (2021a); Lundgard and Satyanarayan (2021); Kim et al. (2021b); Jung et al. (2021); Choi et al. (2019c); Obeid and Hoque (2020); Hsu et al. (2021); Henkin and Turkay (2020); Liu et al. (2020) \\ \hline \end{tabular}
\end{table}
Table 3: Classification of papers based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013).



 

Analysis Details

The language-based interaction implemented in the visualization applications is analyzed considering their _initiative, duration_ and _communicative functions_ present in the human-machine dialogue based on the DIT++ taxonomy of dialogue acts by Bunt (2009). The idea is to create an overview of how the modeled interactions in the respective tasks and sub-tasks are structured. The variables considered in the study and the definitions used for them are explained below. Only contributions that present systems that implement human-machine interaction are part of this examination, theoretical works are excluded. Table 6 contains a comprehensive listing of all contributions evaluated as well as their respective investigation results.

### NLP Methods

For all papers in the selection, the NLP methods used, if any, and if named the NLP toolkits used for implementation are elaborated. For the sake of clarity, the methods are roughly divided into two areas: _Natural Language Understanding (NLU)_ and _Natural Language Generation (NLG)_. Due to the fact, that the majority of the systems use standard NLP methods such as _tokenization, stemming_, or _stopword removal_ in text pre-processing, these are not recorded separately.

Figure 8 shows the distribution of applied NLU methods over the four visualization tasks. It can be seen, that in the _discover_ task the largest variety of methods is applied. Predominantly used are _Semantic Parsing, POS-Tagging_, and _Speech-to-Text_ methods followed by _Language Modeling_ and _Word Sense Disambiguation_. Interfaces in _produce_ to a greater extend rely on _Word Embedding_ and _Named Entity Recognition (NER)_. The _enjoy_ task similar to _discover_ employs a variety of methods. At _present_, NLU methods are only used to a minor extent.

Figure 9 illustrates the distribution of applied NLG methods over the four abstract visualization tasks. _Template-based NLG_ methods are predominantly used to generate text in all tasks at hand. In the _enjoy_ task _Seq2Seq Modeling_ based on deep learning technologies is primarily used presumably due to the proximity to the common NLP task of _Image Captioning_. The same probably applies to the task of text summarization, which is also carried out. _Dialogue Management_ is only applied in _discover_, mostly relying on rule-based or probabilistic modeling methods, e.g., by leveraging _Finite-State-Machine (FSM)_ approaches to manage the sequence of dialogue acts. A small number of systems in _discover_ and _enjoy_ rely on _Text-to-Speech_ technologies in the interaction.

**NLP Toolkits.** An overview of the applied toolkits in NLU is shown in Table 4. Especially _Stanford Core NLP, ANTLR, SpaCy_, and _NLTK_ are found to accomplish several tasks. _Word2Vec_ is the most popular embedding method, followed by _FastText_. The _Web Speech API_ is used primarily because many visualization applications use web technologies. It is striking that many systems rely on _N-gram language models_. In terms of _Word Sense Disambiguation_, WordNet experiences great popularity.

An overview of the applied toolkits in NLG is illustrated in Table 4. The markup language for chatbots _AIML_ as well as the _Rasa_ toolkit are adopted for _Template-based_ text generation, as well as handcrafted _Context-Free-Grammars_. _LL* Parsers_ are predominantly applied in the generation of auto-completions. _Seq2Seq Modeling_ is experiencing increasing interest expressed through the adoption

Figure 8: Distribution of NLU methods **per task**.

Figure 9: Distribution of NLG methods **per task**.

 

of deep learning models such as different variants of _LSTM_ and _Transformers_. It is striking that, in addition to ready-made toolkits such as _Rasa_ and mark-up languages such as _AIML_ for _Dialogue Management_, _Finite-State-Machines_ are also predominantly used.

### Initiative

In an interaction, the initiative is taken by the actor that leads or controls the dialogue, e.g., via questions. McTear (2002) classifies initiative into user initiative, system initiative, and mixed- initiative. Litman and Pan (2004) mark, that the initiative within a dialogue determines the set of possible questions and responses of user and system and therefore the outline of the dialogue. Considering the initiator, the classification of McTear (2002) is used as a basis for the classification including the three general categories:

User InitiativeLanguage-based interactions are classified as user-initiated when the direction of the dialog is determined by the user's actions, in this case, written- or spoken utterances or other text input. The conversation is usually conducted through commands or questions. In addition to prior works (McTear, 2002; Litman and Pan, 2004) considering visualization-oriented dialogue the _data-related intent_ depicts an important factor for a user to take the initiative. Exemplary, this happens when users initiate a conversation by formulating a query to discover new insights about

\begin{table}
\begin{tabular}{p{56.9pt} p{313.8pt}} \hline
**NLU Method** & **Toolkits and Technologies** \\ \hline Semantic Parsing & _ANTLR (4), Context-Free-Grammars(CFG) (3), NLTK (3), Stanford Core NLP (2), AIML (2), IBM Watson, NL4DV Toolkit, SpaCy, Google Cloud Natural Language API, OpenCalais EMPRE_ \\ \hline POS-Tagging & _Stanford Core NLP (6), SpaCy (3), ClearNLP, Rasa, Composite JS_ \\ \hline Word Embedding & _Word2Vec (8), FastText (3), GloVe (2), TF-IDF(2), Sent2Vec, BERT Embedding_ \\ \hline Speech-to-Text & _Web Speech API (8), Microsoft Speech API (3), Google Speech API (2), Apple Speech Framework_ \\ \hline NER & _Stanford Core NLP (3), Chrono JS, Google NLP Toolkit, Wikifier, OpenCalais API, TimeML, TERNIP_ \\ \hline Language Modeling & _N-Gram Language Model (6), BERT (4), Bidirectional LSTM (2)_ \\ \hline Word Sense Disambiguation & _WordNet (6), VerbNet, Concept Net, FrameNet_ \\ \hline Dependency Parsing & _Stanford Core NLP (3), Apache OpenNLP, SpaCy_ \\ \hline Knowledge Representation & _RDF (2), Wolfram Alpha Unit Taxonomy, SIMON_ \\ \hline Constituency Parsing & _ANTLR (2), Stanford Core NLP_ \\ \hline Keyword Extraction & _TF-IDF (3)_ \\ \hline Co-Reference Resolution & _CogCompNLP_ \\ \hline Sentiment Analysis & _Stanford Core NLP, LSTM_ \\ \hline Vector Search & _Word2Vec (2), TF-IDF_ \\ \hline \end{tabular}
\end{table}
Table 4: NLP Toolkits and Technologies used for NLU and how often they are used in the visualization applications.

\begin{table}
\begin{tabular}{l l} \hline
**NLQ Method** & **Toolkits and Technologies** \\ \hline Template-based NLG & _AIML, Rasa, LL* Parser,Context-Free-Grammars_ \\  & _(CFG), IBM Watson_ \\ \hline Seq2Seq Modeling & _LSTM (3), LSTM+Attention (2), CNN+Conditional Random Fields (CRF) (2), Transformer, M4C, LayoutLM, Image Transformer, Bidirectional LSTM_ \\ \hline Dialogue Management & _Finite-State-Machines (FSM)_ \\ \hline Text-to-Speech & _Microsoft TTS_ \\ \hline Text-Summarization & _PageRank Algorithm_ \\ \hline \end{tabular}
\end{table}
Table 5: NLP Toolkits and Technologies used for NLG and how often they are used in the visualization applications.

 

a visualization.

System InitiativeSystem-initiated language-based interactions are determined by natural language utterances generated by the system. The system creates the outline of the interaction towards a previously determined goal. The user is led towards the goal and if the goal is achieved the visualization task is completed. An exemplary case is a system guiding a user in a step-by-step tutorial through the execution of a task, e.g., the identification and elimination of outliers in a visualization.

Mixed InitiativeIn mixed-initiated language-based interaction users and systems at different times and to different proportions contribute to the determination of the interaction. In a visualization context both follow a data-related intent but the way there is characterized by negotiation, proposals, and agreement and disagreement. Exemplary this is the case during interactive clustering where user and system propose different divisions of the data space to each other negotiating a good classification for the underlying data.

To carry out the classification an interaction is considered to be _single-initiated_ (= user initiative or system initiative only) if **during the whole completion of the visualization task the same actor is initiating**. If the initiative **changes at least once** the interaction is classified as _mixed-initiative_ interaction. The scaling of the visualizations is normalized to 100 percent.

Figure 10 shows the distribution of initiative over all included papers in the study. In the set of contributions, user-initiated interactions are predominant, followed by mixed-initiative interactions. Only about ten percent of the interactions are system-initiated.

The distribution of the initiative within the individual visualization tasks is illustrated in Figure 11. In _discover_ and _produce_ user-initiated interactions are predominant. The _present_ task contains a balanced ratio of user- and mixed-initiated interactions. The _enjoy_ task is the only task, where system-initiated interactions represent the majority.

Figure 12 shows the distribution of initiative within the single sub-task categories. Users initiate the interactions in _Keyword Search, Explanation Generation, VQA, Visualization Creation_, and _Documentation_. System initiative is present in _Augmentation, Visualization Description Generation, and Browsing_. Mixed initiative interaction is modeled in _Annotation, Storytelling_ and less frequently in _Natural Language Querying, Browsing_, and _Augmentation_.

### Duration

Within dialogue modeling, natural language-based interactions are modeled as a sequence of dialogue turns. In their survey Deriu et al. (2020) propose a characterization of dialogue system types as task-oriented dialogue systems, conversational

Figure 11: Distribution of initiative **over tasks**.

Figure 12: Distribution of initiative **over sub-tasks**.

 

agents, and interactive QA systems. The basis for this classification is differences in the dialogue structures supported by the different systems, especially in their duration and task-orientedness. Interactive QA systems are considered task-related single- or multi-turn systems. Task-oriented dialogue systems are considered multi-turn systems with short interaction lengths due to the optimization goal. Conversational agents are classified as non-task-oriented multi-turn systems with long interaction lengths. The decision for single- or multi-turn dialogue systems in a visualization-oriented dialogue is a conceptual one that V-NLI designers have to make concerning the quality measure that is set on the system. Single turn systems, e.g., hold higher risks in failing to resolve ambiguities or vague expressions from a single query than multi-turn systems that can pose requests, but also deliver the result in the quickest possible way.

Depending on the visualization task and sub-task at hand, the interaction structures differ. To carry out a uniform duration classification we consider **the length of the human-machine dialogue, that is modeled by the application** as the decisive criterion. Therefore, **interactions that include more than a single utterance in the calculation of the next response** (e.g. by including the dialogue history in context management) and **interactions that support more than one dialogue turn for users and system respectively** are considered _multi-turn_. An interaction is considered to be _single-turn_**if user and system utter at maximum one utterance respectively in a coherent dialogue**.

Figure 13 shows that the modeled interactions in almost two-thirds are single turn and one-third are multi-turn.

On the task level, in _present_ and _enjoy_ interactions are predominantly single-turned (see Figure 14). In the _discover_ and _produce_ task the ratio of multi- to single turn interactions is rather balanced.

Figure 15 illuminates the distribution of duration on a sub-task level. _Explanation Generation_ and _VQA_ are modeled in single turn interactions similar to _Keyword Search, Augmentation_, and _Browsing_. Multi-turn interactions are found predominantly in _Natural Language Querying, Visualization Creation_, and _Documentation_.

### Communicative Functions

Butn (2009) introduces a taxonomy for dialogue act classification. Following that, each individual speech act in a conversation is classified due to the communicative function it carries. On a high level, these general-purpose functions distinguish speech acts as that every individual turn carries either _information providing, information seeking, commissive_ or _directive_ functionality. Especially since visualization-oriented interactions are multimodal designers of V-NLI have to decide which communicative functions are adopted by visual elements as a complementary modality to language

Figure 14: Distribution of duration **over tasks**.

Figure 13: Distribution of duration **over all papers**.

Figure 15: Distribution of duration **over sub-tasks**.

 

in multi-modal dialogue. The examination of communicative functions in the applications at hand is carried out with the idea in mind of gaining an overview of who holds which share of which communicative function in the modeled dialogues. The aim is to help to better characterize and compare the dialogues in the individual tasks and sub-tasks. For contributions that provide access to exemplary human-machine dialogues either within the paper or the supplemental material the presence of each of the communicative functions _information providing, information seeking, commissive_, or _directive_ is detected for user and system respectively (see Table 6). The representation of the identified communicative functions is in absolute quantities for the respective sub-task under consideration.

Figure 16 shows the characteristic distribution of communicative functions for the task _present_ and its respective sub-tasks. It turns out that in these interactions systems predominantly provide users with information. In _Visual Storytelling_ the user and the system complement each other in different modalities, textual and visual, to jointly present visual insights in the form of a multi-modal story. _Explanation Generation_ is characterized by users who are looking for an explanation for a certain behaviour, which can be understood more easily with the help of a visualization and a generated text description acting as a guide.

Figure 17 illuminates the shares of communicative functions in the visualization task _discover_ and the respective sub-tasks. It shows that interactions in _Keyword Search_ and _Visual Question Answering_ are predominantly characterized by users seeking information and systems providing those to the user. In _Keyword Search_ commissives are occasionally uttered by the system to respond to user-induced directives in dialogue. _Visual Question Answering_ follows the classic question-answer scheme in which users bundle their search for information in a question and systems provide textual answers that can be substantiated by the visualization. _Natural Language Querying_ and _Browsing_ contain a more variable profile of communicative functions which also accommodate higher numbers of directives such as, e.g., system-generated _suggestions_ used to pose recommendations to the user in _Browsing_ or _commands_ in _Natural Language Querying_ applied

Figure 16: Distribution of communicative functions in task **present**.

Figure 17: Distribution of communicative functions in task **discover**.

 

by users to make the system execute an action. Interestingly, commissive utterances occur especially in longer analytical conversations, for example, to confirm the loading of a data set or to acknowledge the perception of a command given by the user. When looking at the distribution of communicative functions in _Browsing_, it becomes clear that the system tries to facilitate the user's entry into visual exploration by providing additional information or directives.

The _enjoy_ task is characterized through systems providing the user with additional information as well as occasional directives in the _Augmentation_ task (see Figure 18). Users occasionally ask for information, but the bulk of the interaction consists of the system presenting information to the user or suggesting directives for future interaction. In _Visualization Description Generation_, the special focus of systems is on providing information to visually impaired people. Describing scenes from a visualization in detail so that visually impaired people can perceive them in their full detail requires high-quality text generation that goes beyond standard image captioning.

Interactions in the context of artifact _production_ deliver diverse profiles of communicative functions, as shown in Figure 19. _Annotation_ is characterized by users providing information, e.g., in form of text labels and systems that direct the user, e.g., by making _suggestions_ where to put those. Interactions in _Visualization Creation_ face user and system contributing information as well as systems delivering additional _suggestions_ for the next step in the creation process. In _Documentation_, systems provide textual information in form of a report during or after a user interaction with a visual model.

Figure 19: Distribution of communicative functions in task **produce**.

Figure 18: Distribution of communicative functions in task **enjoy**.

 





* [14] M. C. Dickson, R. K. Sridharan, and J. A. Friedman, _The New York: An Introduction to the Theory of Computing_, volume 207 of _Lecture Notes in Computer Science_, vol. 359 of _Lecture Notes in Computer Science_, vol.