SlowGuess/NAACL_2022_OCR huggingface dataset:
The contributions are classified based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013). Figure 6 illustrates a distribution of papers over the abstract visualization tasks. The visualization task accommodating the highest number of works considering natural language-based interaction is the task discover (70), followed by enjoy (26). Contributions supporting present (9) and produce (14) tasks are less frequent.
Figure 6: Distribution of papers over tasks.
Figure 7 shows the distribution of papers over the inner class sub-tasks. For sake of simplicity, papers are categorized into the single most suitable category only, although some works touch on several categories. Natural Language Querying (45) is by far the sub-task with the highest amount of contributions followed by Keyword Search (15) and Visualization Description Generation (14). Less frequently studied tasks are Explanation Generation (3), Visual Question Answering (VQA) (3), and Documentation (1).
Figure 7: Distribution of papers over sub-tasks.
Table 3 contains a comprehensive listing of the classification of the single contributions into the taxonomy by Brehmer and Munzner (2013).
Table 3: Classification of papers based on the Multi-Level Typology of Abstract Visualization Tasks by Brehmer and Munzner (2013).
Visualization Task	Subtask	References
Discover	Keyword Search	(Feng et al., 2018; Chowdhury et al., 2021; Chung et al., 2010; Fraser et al., 2020; Wilson et al., 2010; Schleußinger and Henkel, 2018; Isaacs et al., 2014; Peltonen et al., 2017; Siddiqui and Hoque, 2020)
VQA	(Singh and Shekhar, 2020; Mathew et al., 2021; Chaudhry et al., 2020)
Querying	(Srinivasan et al., 2020b; Srinivasan and Stasko, 2017; Kassel and Rohs, 2019; Crovari et al., 2020; Tory and Setlur, 2019; Hearst and Tory, 2019; Liu et al., 2021; Hoque et al., 2018; Setlur and Tory, 2017; Siddiqui, 2021; Bacci et al., 2020; Narecha-nia et al., 2021; Yu and Silva, 2020; Setlur et al., 2016; Sun et al., 2014; Aurisano et al., 2016; Srinivasan et al., 2021b, 2019a; Gao et al., 2015; Setlur et al., 2019; Hearst et al., 2019; Setlur and Kumar, 2020; Choi et al., 2019b,a; Sperle et al., 2020, 2021; El-Assady et al., 2020; Kumar et al., 2020a; Aurisano et al., 2015; Turkay and Henkin, 2018; Mazumder and Riva, 2021; Lawrence and Riezler, 2016; Shao and Nakashole, 2020; Wang et al., 2021; Manuvinakurike et al., 2018; Fast et al., 2018; Kassel and Rohs, 2018; Bieliauskas and Schreiber, 2017; Seipel et al., 2019; Lee and Parameswaran, 2018; Kumar et al., 2020b; Srinivasan and Stasko, 2018; Kim et al., 2021c; Srinivasan et al., 2020a; Kumar et al., 2017; Srinivasan et al., 2021a; Hu et al., 2018; Langevin et al., 2018; Reicherts and Rogers, 2020; John et al., 2017)
Browsing	(Setlur et al., 2020; Lee et al., 2021; Luo et al., 2018; de Araújo Lima and Barbosa, 2020; Luo et al., 2020; Srinivasan and Setlur, 2021; Cui et al., 2019; Dhamdhere et al., 2017)
Enjoy	Augmentation	(Srinivasan et al., 2019b; Hullman et al., 2013; Xia et al., 2020; Gao et al., 2014; Kandogan, 2012; Chen et al., 2020c,a; Lai et al., 2020; Bylinskii et al., 2017; Madan et al., 2018; Lallé et al., 2021; Latif and Beck, 2019)
Description Generation	(Demir et al., 2012; Moraes et al., 2014; Sprefico and Carenini, 2020; Qian et al., 2021b; Murillo-Morales and Miesenberger, 2020; Kim et al., 2021a; Lundgard and Satyanarayan, 2021; Kim et al., 2021b; Jung et al., 2021; Choi et al., 2019c; Obeid and Hoque, 2020; Hsu et al., 2021; Henkin and Turkay, 2020; Liu et al., 2020)
Continued on next page
Present	Storytelling	(Bryan et al., 2017; Kwon et al., 2014; Metoyer et al., 2018; Choudhry et al., 2021; Shi et al., 2021; Chen et al., 2020b)
Explanation Generation	(Sevastjanova et al., 2018; Hohman et al., 2019; von Landesberger et al., 2021)
Produce	Annotation	(Chen et al., 2010b,a; Vanhulst et al., 2021; Sperle et al., 2019; Latif et al., 2018; Ren et al., 2017; Latif et al., 2021)
Documentation	(Nafari and Weaver, 2013, 2015)
Visualization Creation	(Rashid et al., 2021; Cui et al., 2020; Qian et al., 2021a; Fulda et al., 2016; Xia, 2020)
C Analysis Details
The language-based interaction implemented in the visualization applications is analyzed considering their initiative, duration and communicative functions present in the human-machine dialogue based on the DIT++ taxonomy of dialogue acts by Bunt (2009). The idea is to create an overview of how the modeled interactions in the respective tasks and sub-tasks are structured. The variables considered in the study and the definitions used for them are explained below. Only contributions that present systems that implement human-machine interaction are part of this examination, theoretical works are excluded. Table 6 contains a comprehensive listing of all contributions evaluated as well as their respective investigation results.
C.1 NLP Methods
For all papers in the selection, the NLP methods used, if any, and if named the NLP toolkits used for implementation are elaborated. For the sake of clarity, the methods are roughly divided into two areas: Natural Language Understanding (NLU) and Natural Language Generation (NLG). Due to the fact, that the majority of the systems use standard NLP methods such as tokenization, stemming, or stopwords removal in text pre-processing, these are not recorded separately.
Figure 8: Distribution of NLU methods per task.
Figure 8 shows the distribution of applied NLU methods over the four visualization tasks. It can be seen, that in the discover task the largest variety of methods is applied. Predominantly used are Semantic Parsing, POS-Tagging, and Speech-to-Text methods followed by Language Modeling and Word Sense Disambiguation. Interfaces in produce to a greater extent rely on Word Embedding and Named Entity Recognition (NER). The enjoy task similar to discover employs a variety of methods.
At present, NLU methods are only used to a minor extent.
Figure 9: Distribution of NLG methods per task.
Figure 9 illustrates the distribution of applied NLG methods over the four abstract visualization tasks. Template-based NLG methods are predominantly used to generate text in all tasks at hand. In the enjoy task Seq2Seq Modeling based on deep learning technologies is primarily used presumably due to the proximity to the common NLP task of Image Captioning. The same probably applies to the task of text summarization, which is also carried out. Dialogue Management is only applied in discover, mostly relying on rule-based or probabilistic modeling methods, e.g., by leveraging Finite-State-Machine (FSM) approaches to manage the sequence of dialogue acts. A small number of systems in discover and enjoy rely on Text-to-Speech technologies in the interaction.
NLP Toolkits. An overview of the applied toolkits in NLU is shown in Table 4. Especially Stanford Core NLP, ANTLR, SpaCy, and NLTK are found to accomplish several tasks. Word2Vec is the most popular embedding method, followed by FastText. The Web Speech API is used primarily because many visualization applications use web technologies. It is striking that many systems rely on Ngram language models. In terms of Word Sense Disambiguation, WordNet experiences great popularity.
An overview of the applied toolkits in NLG is illustrated in Table 4. The markup language for chatbots AIML as well as the Rasa toolkit are adopted for Template-based text generation, as well as handcrafted Context-Free-Grammars.  $LL^{*}$  Parsers are predominantly applied in the generation of auto-completions. Seq2Seq Modeling is experiencing increasing interest expressed through the adoption
NU Method	Toolkits and Technologies
Semantic Parsing	ANTLR (4), Context-Free-Grammars(CFG) (3), NLTK (3), Stanford Core NLP (2), AIML (2), IBM Watson, NL4DV Toolkit, SpaCy, Google Cloud Natural Language API, OpenCalais API, Conditional Random Fields (CRF), Wit.ai, Stanford SEMPRE
POS-Tagging	Stanford Core NLP (6), SpaCy (3), ClearNLP, Rasa, Compromise JS
Word Embedding	Word2Vec (8), FastText (3), GloVe (2), TF-IDF(2), Sent2Vec, BERT Embedding
Speech-to-Text	Web Speech API (8), Microsoft Speech API (3), Google Speech API (2), Apple Speech Framework
NER	Stanford Core NLP (3), Chrono JS, Google NLP Toolkit, Wikifier, OpenCalais API, TimeML, TERNIP
Language Modeling	N-Gram Language Model (6), BERT (4), Bidirectional LSTM (2)
Word Sense Disambiguation	WordNet (6), VerbNet, ConceptNet, FrameNet
Dependency Parsing	Stanford Core NLP (3), Apache OpenNLP, SpaCy
Knowledge Representation	RDF (2), Wolfram Alpha Unit Taxonomy, SIMON
Constituency Parsing	ANTLR (2), Stanford Core NLP
Keyword Extraction	TF-IDF (3)
Co-Reference Resolution	CogCompNLP
Sentiment Analysis	Stanford Core NLP, LSTM
Vector Search	Word2Vec (2), TF-IDF
of deep learning models such as different variants of LSTM and Transformers. It is striking that, in
Table 4: NLP Toolkits and Technologies used for NLU and how often they are used in the visualization applications.
NLG Method	Toolkits and Technologies
Template-based NLG	AIML, Rasa, LL* Parser, Context-Free-Grammars (CFG), IBM Watson
Seq2Seq Mod-eling	LSTM (3), LSTM+Attention (2), CNN+Conditional Random Fields (CRF) (2), Transformer, M4C, LayoutLM, Image Transformer, Bidirectional LSTM
Dialogue Man-agement	Finite-State-Machines (FSM) (2), AIML, Rasa, IBM Watson
Text-to-Speech	Microsoft TTS
Text-Summarization	PageRank Algorithm
Table 5: NLP Toolkits and Technologies used for NLG and how often they are used in the visualization applications.
addition to ready-made toolkits such as Rasa and mark-up languages such as AIML for Dialogue Management, Finite-State-Machines are also predominantly used.
C.2 Initiative
In an interaction, the initiative is taken by the actor that leads or controls the dialogue, e.g., via questions. McTear (2002) classifies initiative into user initiative, system initiative, and mixed- initiative. Litman and Pan (2004) mark, that the initiative within a dialogue determines the set of possible questions and responses of user and system and therefore the outline of the dialogue. Considering the initiator, the classification of McTear (2002) is used as a basis for the classification including the three general categories:
User Initiative. Language-based interactions are classified as user-initiated when the direction of the dialog is determined by the user's actions, in this case, written- or spoken utterances or other text input. The conversation is usually conducted through commands or questions. In addition to prior works (McTear, 2002; Litman and Pan, 2004) considering visualization-oriented dialogue the data-related intent depicts an important factor for a user to take the initiative. Exemplary, this happens when users initiate a conversation by formulating a query to discover new insights about
a visualization.
System Initiative. System-initiated language-based interactions are determined by natural language utterances generated by the system. The system creates the outline of the interaction towards a previously determined goal. The user is led towards the goal and if the goal is achieved the visualization task is completed. An exemplary case is a system guiding a user in a step-by-step tutorial through the execution of a task, e.g., the identification and elimination of outliers in a visualization.
Mixed Initiative. In mixed-initiated language-based interaction users and systems at different times and to different proportions contribute to the determination of the interaction. In a visualization context both follow a data-related intent but the way there is characterized by negotiation, proposals, and agreement and disagreement. Exemplary this is the case during interactive clustering where user and system propose different divisions of the data space to each other negotiating a good classification for the underlying data.
To carry out the classification an interaction is considered to be single-initiated (= user initiative or system initiative only) if during the whole completion of the visualization task the same actor is initiating. If the initiative changes at least once the interaction is classified as mixed-initiative interaction. The scaling of the visualizations is normalized to 100 percent.
Figure 10: Distribution of initiative over all papers.
Figure 10 shows the distribution of initiative over all included papers in the study. In the set of contributions, user-initiated interactions are predominant, followed by mixed-initiative interactions. Only about ten percent of the interactions are system-
Figure 11: Distribution of initiative over tasks.
The distribution of the initiative within the individual visualization tasks is illustrated in Figure 11. In discover and produce user-initiated interactions are predominant. The present task contains a balanced ratio of user- and mixed-initiated interactions. The enjoy task is the only task, where system-initiated interactions represent the majority.
Figure 12: Distribution of initiative over sub-tasks.
Figure 12 shows the distribution of initiative within the single sub-task categories. Users initiate the interactions in Keyword Search, Explanation Generation, VQA, Visualization Creation, and Documentation. System initiative is present in Augmentation, Visualization Description Generation, and Browsing. Mixed initiative interaction is modeled in Annotation, Storytelling and less frequently in Natural Language Querying, Browsing, and Augmentation.
C.3 Duration
Within dialogue modeling, natural language-based interactions are modeled as a sequence of dialogue turns. In their survey Deriu et al. (2020) propose a characterization of dialogue system types as task-oriented dialogue systems, conversational
agents, and interactive QA systems. The basis for this classification is differences in the dialogue structures supported by the different systems, especially in their duration and task-orientedness. Interactive QA systems are considered task-related single- or multi-turn systems. Task-oriented dialogue systems are considered multi-turn systems with short interaction lengths due to the optimization goal. Conversational agents are classified as non-task-oriented multi-turn systems with long interaction lengths. The decision for single- or multi-turn dialogue systems in a visualization-oriented dialogue is a conceptual one that V-NLI designers have to make concerning the quality measure that is set on the system. Single turn systems, e.g., hold higher risks in failing to resolve ambiguities or vague expressions from a single query than multi-turn systems that can pose requests, but also deliver the result in the quickest possible way.
Depending on the visualization task and sub-task at hand, the interaction structures differ. To carry out a uniform duration classification we consider the length of the human-machine dialogue, that is modeled by the application as the decisive criterion. Therefore, interactions that include more than a single utterance in the calculation of the next response (e.g. by including the dialogue history in context management) and interactions that support more than one dialogue turn for users and system respectively are considered multi-turn. An interaction is considered to be single-turn if user and system utter at maximum one utterance respectively in a coherent dialogue.
Figure 13: Distribution of duration over all papers.
Figure 13 shows that the modeled interactions in almost two-thirds are single turn and one-third are multi-turn.
Figure 14: Distribution of duration over tasks.
On the task level, in present and enjoy interactions are predominantly single-turned (see Figure 14). In the discover and produce task the ratio of multi- to single turn interactions is rather balanced.
Figure 15: Distribution of duration over sub-tasks.
Figure 15 illuminates the distribution of duration on a sub-task level. Explanation Generation and VQA are modeled in single turn interactions similar to Keyword Search, Augmentation, and Browsing. Multi-turn interactions are found predominantly in Natural Language Querying, Visualization Creation, and Documentation.
C.4 Communicative Functions
Bunt (2009) introduces a taxonomy for dialogue act classification. Following that, each individual speech act in a conversation is classified due to the communicative function it carries. On a high level, these general-purpose functions distinguish speech acts as that every individual turn carries either information providing, information seeking, commissive or directive functionality. Especially since visualization-oriented interactions are multimodal designers of V-NLI have to decide which communicative functions are adopted by visual elements as a complementary modality to language
in multi-modal dialogue. The examination of communicative functions in the applications at hand is carried out with the idea in mind of gaining an overview of who holds which share of which communicative function in the modeled dialogues. The aim is to help to better characterize and compare the dialogues in the individual tasks and sub-tasks. For contributions that provide access to exemplary human-machine dialogues either within the paper or the supplemental material the presence of each of the communicative functions information providing, information seeking, commissive, or directive is detected for user and system respectively (see Table 6). The representation of the identified communicative functions is in absolute quantities for the respective sub-task under consideration.
Figure 16: Distribution of communicative functions in task present.
Figure 16 shows the characteristic distribution of communicative functions for the task present and its respective sub-tasks. It turns out that in these interactions systems predominantly provide users with information. In Visual Storytelling the user and the system complement each other in different modalities, textual and visual, to jointly present visual insights in the form of a multi-modal story. Explanation Generation is characterized by users who are looking for an explanation for a certain behaviour, which can be understood more easily with the help of a visualization and a generated text description acting as a guide.
Figure 17 illuminates the shares of communicative functions in the visualization task discover and the
Figure 17: Distribution of communicative functions in task discover.
respective sub-tasks. It shows that interactions in Keyword Search and Visual Question Answering are predominantly characterized by users seeking information and systems providing those to the user. In Keyword Search commissives are occasionally uttered by the system to respond to user-induced directives in dialogue. Visual Question Answering follows the classic question-answer scheme in which users bundle their search for information in a question and systems provide textual answers that can be substantiated by the visualization. Natural Language Querying and Browsing contain a more variable profile of communicative functions which also accommodate higher numbers of directives such as, e.g., system-generated suggestions used to pose recommendations to the user in Browsing or commands in Natural Language Querying applied
by users to make the system execute an action. Interestingly, commissive utterances occur especially in longer analytical conversations, for example, to confirm the loading of a data set or to acknowledge the perception of a command given by the user. When looking at the distribution of communicative functions in Browsing, it becomes clear that the system tries to facilitate the user's entry into visual exploration by providing additional information or directives.
Figure 18: Distribution of communicative functions in task enjoy.
The enjoy task is characterized through systems providing the user with additional information as well as occasional directives in the Augmentation task (see Figure 18). Users occasionally ask for information, but the bulk of the interaction consists of the system presenting information to the user or suggesting directives for future interaction. In Visualization Description Generation, the special focus of systems is on providing information to visually impaired people. Describing scenes from a visualization in detail so that visually impaired people can perceive them in their full detail requires high-quality text generation that goes beyond standard image captioning.
Interactions in the context of artifact production deliver diverse profiles of communicative functions, as shown in Figure 19. Annotation is characterized by users providing information, e.g., in form of text labels and systems that direct the user, e.g., by making suggestions where to put those. Interactions in Visualization Creation face user and system con
Figure 19: Distribution of communicative functions in task produce.
tributing information as well as systems delivering additional suggestions for the next step in the creation process. In Documentation, systems provide textual information in form of a report during or after a user interaction with a visual model.