Before revision: Multimodal approaches where interactive visualization and natural language are used in tandem are emerging as promising techniques for data analysis. A significant barrier in effectively designing such multimodal techniques is the lack of a systematicunderstanding of how people verbalize visual representations of data . Motivated by these gaps, this paper devises and applies a transferable, semi-automated methodology to systematically study the relation between visualization and natural language through two crowd-sourced experiments and natural language analysis. We describe these experiments, analyze the resulting corpus of utterances with natural language processing techniques and derive an empirically supported semantic lexicon for aligning visualizations and verbalizations of data.
After revision: Natural language and visualization are being increasingly deployed together for supporting data analysis in different ways, from multimodal interaction to enriched data summaries and insights. Yet, researchers still lack systematic, foundational knowledge on how people relate visual and textual information in data analysis contexts. We describe two studies aimed at developing a broader understanding about the relevant characteristics of data and charts when people read and reason about data visualizations. The first study asks participants to verbalize what they see in scatterplots that depict various levels of correlations. The second study then asks participants to choose visualizations that, in their interpretation, match verbal descriptions of correlation. Using natural language processing techniques and derive an empirically supported semantic lexicon for aligning visualizations and verbalizations of data.