SlowGuess/NAACL_2022_OCR huggingface dataset:
Annotation. Annotating areas of interest, comparing them among each other, and sharing them with colleagues is a common language interaction while working with visualizations (Ren et al., 2017). Chen et al. (2010a,b) leverage touch and click interactions for situated visualization annotation. Latif et al. (2018, 2021) explore the possibilities of linking text and visualization. Sperle et al. (2019) study the visual annotation of argumentation and how this facilitates analysis. Theoretical work on the sustainable extraction of knowledge from visualization annotations is provided by Vanhulst et al. (2021), who propose a classification framework that enables a structured capture and ordering of annotations.
Documentation. Visualization systems are used by experts, e.g., in the medical domain (Meuschke et al., 2021) to plan and discuss a surgery. Reporting, summarizing, and sharing this visualization-related work is an important task that is an additional burden to the surgeon and therefore should be executed by a machine. Nafari and Weaver (2013, 2015) generate natural language questions from queries executed on a visualization resulting in a natural language translation of the interaction. This leaves a step-by-step report of the interaction finding usage as a report of done work.
Visualization Creation. Visualization creation considers the production of a visual model from a
natural language description – also referred to as text2viz. Rashid et al. (2021) generate chart visualizations from natural language text input. Collaborative authoring tools assisting users in visualization creation use natural language as an input modality. Cui et al. (2020) provide a tool that generates infographics using natural language statements as input, similar to Qian et al. (2021a). Fulda et al. (2016) design an interactive production process for generating timelines from unstructured text input. Language-based 3D scene generation, also referred to as text2scene, which allows users to describe 3D scenes using text without having to learn software tools, is investigated in Coyne and Sproat (2001); Coyne et al. (2012); Ulinski et al. (2018).
4 How Users interact with Visualizations
After discussing why users interact with visualizations using natural language, Section 4 provides a complementary discussion of how these interactions are modeled. First, in Section 4.1 it is explained which NLP methods are used in these systems. Subsequently, Section 4.2 summarizes the structure of the visualization-oriented dialogues in the analyzed paper set in terms of initiative, duration, and present communication functions within the respective visualization task.
4.1 NLP Methods
For each paper in the collection, both the NLP methods used, if any, and if named the specific NLP toolkits used for implementation are elaborated. For the sake of clarity, the methods are roughly divided into two areas: Natural Language Understanding (NLU) and Natural Language Generation (NLG). The majority of the systems apply standard NLP methods like tokenization, stemming or stopwords removal to pre-process text inputs, which is why these are not recorded separately. For a detailed inspection, we refer to Appendix C. Figure 3 shows the distribution of applied NLU methods over all papers. Semantic Parsing, which relies on rule-based mapping procedures from recognized input tokens to semantic predicates, is predominantly used. Often, POS-Tagging, Word Embeddings, and Named Entity Recognition (NER) are additionally applied to increase the accuracy of the mapping. For Word Sense Disambiguation WordNet, VerbNet or ConceptNet are leveraged. Speech-to-Text APIs are a common method used in many systems to enable auditory input. A small number of pioneering
Figure 3: Distribution of NLU methods over all papers.
Figure 4: Distribution of NLG methods over all papers.
systems integrate more sophisticated NLP methods such as Sentiment Analysis, Vector Search or Co-Reference Resolution. One main reason for the hesitant use of deep learning methods is the high demands on performance and robustness of visualization systems as Dhamdhere et al. (2017) points out. Fluid interaction between user and system in real time is a crucial factor for the success of a visualization application. Adopting state-of-the-art deep learning models to real-time interactions in visualization, e.g., by using Knowledge Distillation (Hinton et al., 2015) or Quantization (Jacob et al., 2018) leaves space for future work. Figure 4 shows the distribution of applied NLG methods over all papers. Template-based language generation is used by the majority of the systems followed by a significantly smaller number of deep learning-based Seq2Seq Modeling approaches. Multi-turn systems are predominantly based on rule-based or probabilistic Dialogue Management. Only a few systems use the Text-to-Speech functionality, as most of the generated responses consist of visual elements. In order to advance the adoption of deep learning-based methods in visualization-related text generation, extensive training data sets are required, as pointed out by Kumar et al. (2020a). There is a
limited number of data sets for Visualization Description Generation (Obeid and Hoque, 2020), Visual Question Answering (Mathew et al., 2021; Kim et al., 2020) and Natural Language Querying (Fu et al., 2020; Srinivasan et al., 2021b; Luo et al., 2021). In particular, the compilation of data sets for emerging dialogue scenarios in Analytical Conversation, Hypothesis Verification or collaborative authoring in Visualization Creation would motivate the use of deep learning based NLP methods in these tasks. Therefore, generating high-quality data sets for the aforementioned visualization tasks leaves room for future work.
4.2 Dialogue Structures
The study of structures in visualization-oriented dialogue is done with the idea of identifying task-specific patterns, as shown in Figure 5. The structural analysis is based on the work of Bunt (2009) and highlights, in particular, the initiative, duration, and communicative functions present in the modeled dialogues. For each contribution that provides access to sample data illustrating human-machine dialogue within the paper or supplementary material, the presence of the communicative functions information providing, information-seeking, commissive, or directive for the user and system is detected. A comprehensive list of allocations is presented in Appendix C. Bunt's  $\mathrm{DIT}++$  taxonomy was chosen due to the fact that it focuses on the function of the individual speech act. In the context of a visualization task, it is important which function a dialog act fulfills in the successful execution of this task. This manifests itself particularly in the design of dialog agents, where speech acts that are intended to help solve the task must be specified. Other taxonomies focus on the rhetorical relations of speech acts to each other, as in Prasad et al. (2008), or the emotional information a speech act conveys in the dialogue, as in EmotionML (Schröder et al., 2011). In contrast to the aforementioned taxonomies, Bunt's taxonomy proves suitable in two respects: It allows to understand how current dialogue situations are functionally structured in the visualization context. In this way, patterns can be identified that are common for the respective visualization tasks. From the generation perspective, it allows to specify dialog actions that need to be prepared in certain visualization task contexts in order to support the solution of the visualization task.
Figure 5: Analysis of initiative, duration and communicative functions in the sub-task browsing.
Present Task. In Visual Storytelling, users initiate the interactions which are performed as a sequence of multiple turns triggered through the selection of text phrases. The human-machine dialogue is characterized by the actors complementing each other through text and visual animations as speech acts. Similar to storytelling, the human-machine dialogue in Explanation Generation is characterized by a complementing of user input and visualization system output by matching visual and textual elements. Communicating insights found by investigating a visual model is little accompanied by NLP techniques so far compared to other visualization tasks. Template-based story generation systems leave room for innovation in grounding story segments in visualization elements.
Discover Task. The human-machine dialogue in Keyword Search is a short, single-turn dialogue characterized by user-initiated text input that is reciprocated by a visual response from the system, similar to Natural Language Querying. Keyword-based visualization search relies on text label-only search without including the on-screen representation. This leaves space for grounding abstract visualization concepts like outliers or clusters in natural language as a promising step towards a generalized visualization search. Keyword search beyond 2D visualizations is an open issue. The modeled dialogue interaction in analytical conversation is a user-initiated dialogue with multiple turns. Visual elements function in the communication as information providers carrying the response. This scenario offers the possibility to apply modern NLP methods to multi-modal dialogues by checking the user's
intent and dynamically adapting the user's experience by using the feedback of multiple turns. The implemented systems in Visual Question Answering follow a user-initiated, single-turn dialogue approach in which a user question is answered based on textual or visual information the visualization holds. In Browsing the initiative in the systems varies between user-, mixed- and system-initiated (see Figure 5) with variable duration. Most approaches focus on high-quality auto-completion to lead users, leaving space for innovation in guidance-based dialogue approaches.
Enjoy Task. The predominant features of human-machine dialogue in Augmentation are system-initiated single-turn interactions where written or spoken text is used to augment the visual representation. The augmentation of visualizations leaves room for a stronger inclusion of multi-modal interaction triggers such as gazes and gestures in the dialogue conception. Visualization Description Generation is characterized by system-initiated single turn systems. Summarizing visualizations is a challenge because it requires a high-quality scene description due to the high explanatory potential of visualizations. Particularly visually impaired people benefit from well-designed auditory visualization descriptions, which are a motivation for further improvements.
Produce Task. During Annotation, users initiate interactions, which can be continued by system suggestions or completions. Producing artifacts based on a visualization so far relies on template-based authoring tools. Guidance and competition in educational contexts, as well as the collaboration of user and system during artifact production, seem to be promising directions for production-supporting human-machine dialogue conception in the future. Authoring tools take the initiative in Visualization Creation by suggesting answers or partial task completions. The cooperation with the user appears often in form of a multi-turn production process. Documentation is done as a complement of the user's actions, in that the system provides the user with a report of the work performed after or during the user-initiated interaction with a visualization.
In this survey, for a renowned taxonomy of abstract visualization tasks, we classified 119 approaches
of language-based interaction that support users in pursuing data-related intents. In particular, we shed a light on how the human-machine dialogue is constructed in these works and which NLP methods current V-NLI use. Considering the progress of NLP methods in the field of visualization, we can summarize our work on two main outcomes: A compilation of data sets for the individual visualization tasks seems promising to advance the use of deep learning-based NLP methods; When introducing them, special attention must be paid to performance and robustness aspects due to the high requirements in the VIS area. Finally, the support of visualization tasks through natural language interaction offers a large number of interesting areas for the application of state-of-the-art NLP methods, inviting the NLP and VIS communities to work creatively in the emerging intersection of both fields.
We thank the Michael Stifel Center Jena for funding this work, which is part of the Carl Zeiss Foundation-funded project 'A Virtual Workshop for Digitization in the Sciences' (062017-02).
Manoj Acharya, Karan Jariwala, and Christopher Kanan. 2019. Vqd: Visual query detection in natural scenes. arXiv preprint arXiv:1904.02794.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6077-6086.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31.
Jillian Aurisano, Abhinav Kumar, Alberto Gonzales, Khairi Reda, Jason Leigh, Barbara Di Eugenio, and Andrew Johnson. 2015. Show me data": Observational study of a conversational interface in visual data exploration. In IEEE VIS, volume 15, page 1.
Jillian Aurisano, Abhinav Kumar, Alberto Gonzalez, Jason Leigh, Barbara DiEugenio, and Andrew Johnson. 2016. Articulate2: Toward a conversational interface for visual data exploration. In IEEE Visualization.
Francesca Bacci, Federico Maria Cau, and Lucio Davide Spano. 2020. Inspecting data using natural language queries. Computational Science and Its Applications - ICCSA 2020, 12254:771 - 782.
Stefan Bieliauskas and Andreas Schreiber. 2017. A conversational user interface for software visualization. 2017 IEEE Working Conference on Software Visualization (VISSOFT), pages 139-143.
Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2016. Learning end-to-end goal-oriented dialog. arXiv preprint arXiv:1605.07683.
Matthew Brehmer and Tamara Munzner. 2013. A multilevel typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics, 19:2376-2385.
Chris Bryan, Kwan-Liu Ma, and Jonathan Woodring. 2017. Temporal summary images: An approach to narrative visualization via interactive annotation generation and placement. IEEE Transactions on Visualization and Computer Graphics, 23:511-520.
Harry Bunt. 2009. The dit++ taxonomy for functional dialogue markup. In AAMAS 2009 Workshop, Towards a Standard Markup Language for Embodied Dialogue Acts, pages 13-24.
Harry Bunt, Jan Alexandersson, Jean Carletta, JaeWoong Choe, Alex Chengyu Fang, Koiti Hasida, Kiyong Lee, Volha Petukhova, Andrei Popescu-Belis, Laurent Romary, Claudia Soria, and David R. Traum. 2010. Towards an iso standard for dialogue act annotation. In LREC.
Zoya Bylinskii, Sami Alsheikh, Spandan Madan, Adrià Recasens, Kimberli Zhong, Hanspeter Pfister, Frédo Durand, and Aude Oliva. 2017. Understanding infographics through textual and visual tag prediction. ArXiv, abs/1709.09215.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. Leaf-qa: Locate, encode & attend for figure question answering. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 3501-3510.
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020a. Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning. 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pages 322-334.
Siming Chen, Jie Li, Gennady L. Andrienko, Natalia V. Andrienko, Yun Wang, Phong H. Nguyen, and Cagatay Turkay. 2020b. Supporting story synthesis: Bridging the gap between visual analytics and storytelling. IEEE Transactions on Visualization and Computer Graphics, 26:2499-2516.
Yang Chen, Scott Barlowe, and Jing Yang. 2010a. Click2annotate: Automated insight externalization with rich semantics. 2010 IEEE Symposium on Visual Analytics Science and Technology, pages 155-162.
Yang Chen, Jing Yang, Scott Barlowe, and Dong Hyun Jeong. 2010b. Touch2annotate: generating better annotations with less human effort on multi-touch interfaces. CHI '10 Extended Abstracts on Human Factors in Computing Systems.
Zhutian Chen, Wai-Shun Tong, Qianwen Wang, Benjamin Bach, and Huamin Qu. 2020c. Augmenting static visualizations with paparvis designer. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
In Kwon Choi, Taylor Childers, Nirmal Kumar Raveendranath, Swati Mishra, Kyle Harris, and Khairi Reda. 2019a. Concept-driven visual analytics: an exploratory study of model- and hypothesis-based reasoning with visualizations. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems.
In Kwon Choi, Nirmal Kumar Raveendranath, Jared Westerfield, and Khairi Reda. 2019b. Visual (dis) confirmation: Validating models and hypotheses with visualizations. In 2019 23rd International Conference in Information Visualization–Part II, pages 116–121. IEEE.
Jinho Choi, Sanghun Jung, Deok Gun Park, Jaegul Choo, and Niklas Elmqvist. 2019c. Visualizing for the non-visual: Enabling the visually impaired to use visualization. Computer Graphics Forum, 38.
Arjun Choudhry, Mandar Sharma, Pramod Chandury, Thomas Kapler, Derek W. S. Gray, Naren Ramakrishnan, and Niklas Elmqvist. 2021. Once upon a time in visualization: Understanding the use of textual narratives for causality. IEEE Transactions on Visualization and Computer Graphics, 27:1332-1342.
Imran Chowdhury, Abdul Moeid, Enamul Hoque, Muhammad Ashad Kabir, Md. Sabir Hossain, and Mohammad Mainul Islam. 2021. Designing and evaluating multimodal interactions for facilitating visual analysis with dashboards. IEEE Access, 9:60-71.
Haeyong Chung, Seungwon Yang, Naveed Massjouni, Christopher Andrews, Rahul Kanna, and Chris North. 2010. Vizcept: Supporting synchronous collaboration for constructing visualizations in intelligence analysis. 2010 IEEE Symposium on Visual Analytics Science and Technology, pages 107-114.
Bob Coyne, Alex Klapheke, Masoud Rouhizadeh, Richard Sproat, and Daniel Bauer. 2012. Annotation tools and knowledge representation for a text-to-scene system. In Proceedings of COLING 2012, pages 679-694.
Bob Coyne and Richard Sproat. 2001. Wordseye: An automatic text-to-scene conversion system. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 487-496.
Pietro Crovari, Sara Pidò, Franca Garzotto, and Stefano Ceri. 2020. Show, don't tell: reflections on the design of multi-modal conversational interfaces. In CONVERSATIONS.
Weiwei Cui, Xiaoyu Zhang, Yun Wang, He Huang, B. Chen, Lei Fang, Haidong Zhang, Jian-Guang Lou, and Dongmei Zhang. 2020. Text-to-viz: Automatic generation of infographics from proportion-related natural language statements. IEEE Transactions on Visualization and Computer Graphics, 26:906-916.
Zhe Cui, Sriram Karthik Badam, Mehmet Adil Yalçın, and Niklas Elmqvist. 2019. Datasite: Proactive visual data exploration with computation of insight-based recommendations. Information Visualization, 18:251-267.
Raul de Araujo Lima and Simone Diniz Junqueira Barbosa. 2020. A question-oriented visualization recommendation approach for data exploration. Proceedings of the International Conference on Advanced Visual Interfaces.
Seniz Demir, Sandra Carberry, and Kathleen F. McCoy. 2012. Summarizing information graphics textually. Computational Linguistics, 38:527-574.
Jan Deriu, Álvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2020. Survey on evaluation methods for dialogue systems. Artificial Intelligence Review, 54:755 - 810.
Kedar Dhamdhere, Kevin S. McCurley, Ralfi Nahmias, Mukund Sundararajan, and Qiqi Yan. 2017. Analyza: Exploring data with conversation. Proceedings of the 22nd International Conference on Intelligent User Interfaces.
Evanthia Dimara and Charles Perin. 2020. What is interaction for data visualization? IEEE Transactions on Visualization and Computer Graphics, 26:119-129.
Mennatallah El-Assady, Rebecca Kehlbeck, Christopher M. Collins, Daniel A. Keim, and Oliver Deussen. 2020. Semantic concept spaces: Guided topic model refinement using word-embedding projections. IEEE Transactions on Visualization and Computer Graphics, 26:1001-1011.
Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S. Bernstein. 2018. Iris: A conversational agent for complex tasks. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
Mi Feng, Cheng Deng, Evan M. Peck, and Lane Harrison. 2018. The effects of adding search functionality to interactive visualizations on the web. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems.
C. Ailie Fraser, Julia M. Markel, N. James Basa, Mira Dontcheva, and Scott R. Klemmer. 2020. Remap: Lowering the barrier to help-seeking with multimodal search. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.
Siwei Fu, Kai Xiong, Xiaodong Ge, Siliang Tang, Wei Chen, and Yingcai Wu. 2020. Quda: Natural language queries for visual data analytics. arXiv preprint arXiv:2005.03257.
Johanna Fulda, Matthew Brehmer, and Tamara Munzner. 2016. Timelinecurator: Interactive authoring of visual timelines from unstructured text. IEEE Transactions on Visualization and Computer Graphics, 22:300-309.
Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie Karahalios. 2015. Datatone: Managing ambiguity in natural language interfaces for data visualization. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology.
Tong Gao, Jessica R. Hullman, Eytan Adar, Brent J. Hecht, and Nicholas A. Diakopoulos. 2014. Newviews: an automated pipeline for creating custom geovisualizations for news. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
Marti A. Hearst and Melanie K. Tory. 2019. Would you like a chart with that? incorporating visualizations into conversational interfaces. 2019 IEEE Visualization Conference (VIS), pages 1-5.
Marti A. Hearst, Melanie K. Tory, and Vidya Setlur. 2019. Toward interface defaults for vague modifiers in natural language interfaces for visual analysis. 2019 IEEE Visualization Conference (VIS), pages 21-25.
Rafael Henkin and Cagatay Turkay. 2020. Words of estimative correlation: Studying verbalizations of scatterplots. IEEE Transactions on Visualization and Computer Graphics, PP.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Fred Hohman, Arjun Srinivasan, and Steven Mark Drucker. 2019. Telegram: Combining visualization and verbalization for interpretable machine learning. 2019 IEEE Visualization Conference (VIS), pages 151-155.
Enamul Hoque, Vidya Setlur, Melanie K. Tory, and Isaac Dykeman. 2018. Applying pragmatics principles for interaction with visual analytics. IEEE Transactions on Visualization and Computer Graphics, 24:309-318.
Ting-Yao Hsu, C. Lee Giles, and Ting-Hao Kenneth Huang. 2021. Scicap: Generating captions for scientific figures. In EMNLP.
Kevin Zeng Hu, Diana Orghian, and César A. Hidalgo. 2018. Dive: A mixed-initiative system supporting integrated data exploration workflows. Proceedings of the Workshop on Human-In-the-Loop Data Analytics.