VisText: A Benchmark for Semantically Rich Chart Captioning
Year: 2023

Abstract:
Captions that describe or explain charts help improve recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, current approaches for automatically generating such captions struggle to articulate the perceptual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts’ construction, report key statistics, and identify perceptual and cognitive phenomena. In VisText, a chart is available as three representations: a rasterized image, a backing data table, and a scene graph — a hierarchical representation of a chart’s visual elements akin to a web page’s Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to produce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we identify six broad categories of errors that our models make that can inform future work.

heading: Future Work
text: Further, we categorize six common captioning errors that can inform the future development of chart captioning models on the VisText dataset. The VisText dataset and source code are available at: https://github. com/mitvis/ vistext. 2 Related work
Heuristic-Based Chart Captioning. Automatically generating natural language descriptions of data tables dates back to Reiter and Dale (1997). Demir et al. (2008, 2010, 2012) survey this early work and describe the process of extracting insights from a chart by evaluating a list of propositions and composing selected propositions together to produce a natural language summary. More recently, data visualization researchers have explored heuristics that calculate summary statistics and templates to assemble natural language “data facts” (Srini-
vasan et al. , 2019) or descriptions (Cui et al. , 2019). While useful, these approaches yield standardized descriptions that lack the variation and linguistic construction that characterize semantically rich captions (Lundgard and Satyanarayan, 2022). Chart Captioning as Image Captioning. With rapid advances of neural image captioning (Vinyals et al. , 2015; Anderson et al. , 2018), researchers have begun to adapt these methods for captioning charts. For instance, Balaji et al. (2018) develop a deep learning pipeline that ingests a PNG chart image, classifies the chart type, detects and classifyies textual content present in the chart, and uses this information to generate a textual description. Chen et al. (2019a,b, 2020) propose a simpler workflow using ResNet to encode the chart image and an LSTM with Attention to decode it into a natural language description. Both approaches share a pair of limitations. The captions they produce convey relatively simplistic information about the chart (e. g. , title, axis labels, etc. ) or articulate concepts in visual rather than data terms (e. g. , “Dark Magenta has the lowest value”). While both approaches contribute associated datasets, their charts and captions are synthetically generated and may not represent real-world counterparts. SciCap (Hsu et al. , 2021) addresses this limitation by scraping real-world charts from 290,000 arXiv papers; however, the baseline models trained on this dataset struggle to generate semantically rich captions. Chart Captioning as Text Translation. Perhaps closest to our contribution is recent work modeling chart captioning as a data-to-text problem. For instance, Spreafico and Carenini (2020) train an encoder-decoder LSTM architecture to generate a natural language caption from time series data. Similarly, Obeid and Hoque (2020) and Kantharaj et al. (2022) explore how transformer architectures can translate tabular structures into captions. These data-to-text methods are more successful than chart-as-image captioning, yielding captions that better capture relevant information from the charts and have higher BLEU scores. Nevertheless, we observe two limitations with these data-to-text approaches that motivate our contribution. First, data-to-text methods are heavily reliant on access to a chart’s data table. In practice, data tables are only rarely published alongside charts and methods that recover equivalent information via OCR experience a significant drop in performance (Kantharaj et al. , 2022). Second, the associated datasets
7269
do not contain sufficient training examples of captions that express semantically rich insights about the depicted data (i. e. , the perceptual and cognitive phenoma that distinguish charts as a medium as distinct from data tables (Lundgard and Satyanarayan, 2022)). As a result, while the generated captions are compelling, they are largely limited to reporting statistics which sighted and blind readers prefer less than captions that convey complex trends and patterns (Lundgard and Satyanarayan, 2022). 3 The VisText Dataset
We designed the VisText dataset in response to two limitations existing datasets present for generating semantically rich chart captions. First, existing datasets represent charts as either rasterized images or as data tables. While useful, these representations trade off perceptual fidelity and chart semantics in mutually exclusive ways — images capture the perceptual and cognitive phenomena that are distinctive to charts (e. g. , trends or outliers) but pixels cannot express the rich semantic relationships between chart elements (e. g. , estimating plotted data values using axis labels). While the vice-versa is true (Lundgard and Satyanarayan, 2022), tables also present additional caveats. There is not always a one-to-one relationship between the semantics of a data table and chart (i. e. , one data table may be the source for several distinctly different charts). Moreover, data tables are rarely published alongside charts; and, automatic data table extraction is error-prone due to the diversity of chart types and visual styles as well as the difficulty of reasoning about visual occlusion (Kantharaj et al. , 2022; Luo et al. , 2021; Jung et al. , 2017)). Second, if existing datasets provide captions that describe perceptual or cognitive features, these captions comprise only a small portion of the dataset. At best, LineCap (Mahinpei et al. , 2022) offers 3,528 such captions for line charts only, while Chart-to-Text (Kantharaj et al. , 2022) estimates that roughly 15% of the sentences in its captions across a variety of chart types express such content. In contrast, VisText provides 12,441 crowdsourced English captions that articulate statistical, perceptual, and cognitive characteristics of bar, line, and area charts. In VisText, charts are available as not only data tables and rasterized images but also as scene graphs. Scene graphs are hierarchical representations that better preserve perceptual fidelity and chart semantics, are often the format for
publishing web-based charts, and can be recovered from chart images (Poco and Heer, 2017). 3. 1 Data Table Collection
The data tables found in VisText are sourced from the Statista dataset of the Chart-to-Text benchmark (Kantharaj et al. , 2022). The tables were collected by crawling Statista. com in December 2020 and contain real-world data related to technology, trade, retail, and sports. We process these tables to make them amenable for chart generation, including stripping formatting symbols (e. g. , $ and %), standardizing data strings, and identifying the measure type of each column (i. e. , quantitative, categorical, or temporal). Data tables are discarded if they do not contain at least one quantitative field and one categorical or temporal field, or if other errors occur during the processing steps. We further down select to data tables containing between 2 to 20 columns and 10 to 500 rows. If a data table has over 500 rows, we randomly sample rows. In larger data tables, this step potentially affects how salient a trend is. 3. 2 Chart Generation and Representation
Charts in the Chart-to-Text Statista dataset all feature the same layout and visual appearance. In contrast, we aim for richer visual diversity by generating charts using the Vega-Lite visualization library (Satyanarayan et al. , 2016) via the Python Altair package (VanderPlas et al. , 2018). To facilitate collecting high-quality captions, we focus on univariate charts: charts that depict one quantitative observation against a categorical or temporal variable. This focus is informed by recent work in the data visualization research community which has chosen single-series line charts as the target of study for natural language descriptions (Kim et al. , 2021; Stokes et al. , 2022). VisText also includes single-series bar and area charts as they typically exhibit similar perceptual features to line charts. For each data table, we iterate through pairs of univariate fields. If the pair contains a temporal field, we randomly generate an area or line chart; if the pair contains a categorical field, we randomly generate a horizontal or vertical bar chart. For diversity in layout and visual appearance, we randomly rotate axis labels and apply one of fourteen themes provided by the Vega-Lite library. These themes mimic the visual style of common chart platforms or publishers (e. g. , ggplot2 or the LA Times). 7270
In VisText, each chart is represented as a rasterized image, stored as an RGBA-encoded PNG file, as well as a scene graph. A scene graph is a textual representation of the rendered chart similar to a web page’s Document Object Model (DOM). Scene graphs encode the position, value or content, and semantic role of all visual elements within a chart, including the individual marks (i. e. , bars or points along the line), titles, axes gridlines, etc. Thus, scene graphs express the perceptual features of rasterized images in a more computationallytractable form. Scene graphs are a standard data structure for representing vector-based graphics — the most common format for publishing visualizations — and, thus, can be trivially recovered (e. g. , by traversing the SVG text string). We extract the scene graph directly from the rendered chart using the Vega-Lite API. As most text generation models expect a linear set of input tokens, we flatten the scene graph via a depth-first traversal. To scale to large language models, we need to further reduce the size of the scene graph. Thus, we preserve the following elements which we hypothesize as being most critical for generating semantically rich captions: title, title coordinates, axis labels, axis label coordinates, axis tick coordinates, mark coordinates, and mark sizes. VisText includes both the original (hierarchical) and reduced (linearized) scene graphs. 3. 3 Caption Generation and Collection
Our captioning process is guided by the framework developed by Lundgard and Satyanarayan (2022), which identifies four levels of semantic content: L1 content enumerates aspects of the chart’s construction (e. g. , axis ranges); L2 content reports summary
statistics and relations (e. g. , extrema); L3 content synthesizes perceptual and cognitive phenomena (e. g. , complex trends); and, L4 content describes domain-specific insights (e. g. , sociopolitical context). In subsequent studies, the authors find that while sighted readers typically prefer higher levels of semantic content, blind readers are split about the usefulness of L1 and L4 content. Thus, given these differing preferences, we define a single caption to express multiple levels of content separated across clauses or sentences. We only consider the first three levels of this model, and leave L4 content to future work. Following guidelines prescribed by the National Center for Accessible Media (NCAM), our captions begin with L1 content and then turn to L2 and L3 content (Gould et al. , 2008). We algorithmically generate L1 content and use a crowdsourced protocol to collect L2 and L3 content. This approach follows (Lundgard and Satyanarayan, 2022)’s computational considerations as well as results from Morash et al. (2015) who find that, even with instructions and guidelines, crowd workers do not describe a chart’s structural elements sufficiently for blind readers. Thus, synthetically generating L1 content allows us to ensure that captions convey complete descriptions of the chart’s structural elements. L1 content comprises 1 sentence conveying the chart type and title, and then 1 – 2 sentences describing the axes (including the titles, ranges, and scales). We use template randomization to generate a diverse range of L1 captions to mimic human variability and reduce the capacity of the model to overfit to a single L1 style. Three templates are defined for the first sentence and twenty-six template combinations for the subsequent sentences. During generation, we randomly select a pair of templates and fill in in-
7271
formation from the abstract chart specification. For additional diversity, we randomly drop scale information and swap template words with synonyms. Templates and synonym replacements are listed in Appendix E. 2. To crowdsource L2 and L3 content, we extend the protocol used by Lundgard and Satyanarayan (2022). After soliciting consent, we introduce the task: participants are presented with a chart image and corresponding L1 description; they are asked to write a description about the trends and patterns they observe without drawing on background knowledge or repeating L1 information. The introduction provides examples and explanations of valid and invalid responses. After acknowledging these examples, participants are asked to complete 5 random iterations of the task. To maximize the quality of our crowdsourced captions, we manually curated the charts and L1 descriptions used in the study. We discarded any charts that were challenging to read (e. g. , colors were too similar, marks were not easily readable, etc. ). Participants were recruited on the Prolific. co platform, took approximately 14 minutes to complete the study, and were compensated $3. 25 ($14/hour). Additional details on our crowdsourcing process are in Appendix E. 3. We manually verified charts where participants failed an attention check and discarded invalid descriptions. Additionally, we manually inspected captions for personally identifiable information or offensive content. Using heuristics, we removed captions where respondents described charts as unclear or illegible and replaced newline characters with spaces. Although we attempted to fix incorrect spelling and casing errors using a similar heuristic-based approach, we observed that this process could improperly affect axis and chart names. As a result, these errors remain in our dataset. 3. 4 Dataset Analysis Figure 2 shows the distribution and means of the lengths of chart representations and captions. Synthetically generated L1 captions have roughly 1. 5x more characters than crowdsourced L2/L3 captions (µ = 255 vs. µ = 177) but the average number of sentences are comparable (2. 5 vs. 2). The VisText dataset consists of captions for 3,189 area charts, 6,238 bar charts, and 3,014 line charts — the roughly twice-as-many bar charts as area or line charts corresponds to the randomization of temporal fields during chart generation (Sec. 3. 2). As some charts have multiple crowdsourced captions, we randomly split our dataset into training, validation, and test sets using the chart IDs to prevent data leakage across sets. This resulted in an approximate ratio of 80:10:10. Finally, to understand the distribution of semantic content, we manually coded 2% (230) of crowdsourced captions. We followed a protocol inspired by Lundgard and Satyanarayan (2022) by breaking sentences down into independent statements and mapping these statements to their semantic content level. We marked statements as not categorizable if they did not map to the framework — for instance, if captions expressed commentary from crowd workers such as “this chart is hard to read. ” Our analysis revealed 11 L1 statements (2. 4%), 180 L2 statements (39. 7%), 253 L3 statments (55. 7%), and 10 not categorizable statements (2. 2%). While a handful express L1 content, the bulk of statements (95%) express L2 or L3 content, with approximately 1. 4x L3 statements than L2. 4 Chart Captioning Models
To demonstrate the affordances of the VisText dataset, we train three classes of models. First, we fine-tune large language models to translate from textual chart representations to natural lan-
7272
guage captions. These models evaluate the feasibility and impact of scene-graph models compared to prior data-table approaches (Kantharaj et al. , 2022). Second, as VisText provides multiple chart representations, we adapt image-guided translation (Sulubacak et al. , 2020; Cho et al. , 2021) to develop two multimodal chart captioning models: image-scene-graph and image-data-table. Finally, since VisText offers captions at different semantic levels and prior work has shown significant differences in readers’ preferences (Lundgard and Satyanarayan, 2022), we explore prefix-tuned models that selectively output L1, L2/L3, or L1+L2/L3 captions. Training details are in Appendix D. 4. 1 Text-Based Chart Captioning
Informed by prior work (Kantharaj et al. , 2022), we investigate text translation models for generating chart captions. In particular, Kantharaj et al. found that models that translate data tables to chart captions significantly outperform image captioning models. However, when data tables were not available, the authors found a significant drop in their models’ ability to extract relevant information from the chart — an effect that was only slightly ameliorated by using OCR methods to extract text from chart images. In contrast, VisText’s scene graphs can be more readily recovered from charts when data tables are not available — for instance, by processing the SVG format of web-based visualizations. Moreover, scene graphs offer a potentially richer source of information than data tables as they encode visual properties of the chart (e. g. , coordinates and colors) and are less noisy than tokens recovered via OCR. Thus, to evaluate the feasibility and efficacy of scene graphs, we train a scene-graph text translation model and a baseline data-table model for comparison. For each model, we fine-tune a pretrained ByT5 transformer model (Xue et al. , 2022) on the VisText dataset. We choose ByT5 over T5 transformers (Raffel et al. , 2020) because it uses a token-free, byte-encoding that eliminates the use of a tokenizer. As a result, it is robust to noisy inputs, minimizes the need for text preprocessing, and eliminates the out-of-dictionary problem. This allows our model to handle common typographical and chart reading errors in the crowdsourced L2 and L3 captions and increases generalizability to previously-unseen words that could be present in chart and axes titles. 4. 2 Image-Guided Chart Captioning
Following recent advancements in image-guided machine translation (Sulubacak et al. , 2020), we train image-guided captioning models using the VisText dataset. Images have improved text-based machine translation models by providing visual information complementary to natural language inputs. Similarly, chart images can contain visuals complementary to the textual specification. For instance, visual affordances that are important for perceiving a trend (e. g. , gestalt relations, relative sizes/areas, etc. ) may be obfuscated in the scene graph but better captured in the chart image. We train three image-guided chart captioning models: image, image-scene-graph, and image-data-table. All models leverage the vision-language transformer model VL-T5 (Cho et al. , 2021). VL-T5 is pretrained on image captioning and visual grounding tasks and was successfully applied to machine translation, making it suitable for chart captioning. We extract visual features for each VisText chart image using a Bottom-Up Feature Extractor (Anderson et al. , 2018). To explore the value of images to chart captioning, our image model only takes in the image features, while image-scene-graph and image-data-table concatenate the image features with the chart’s textual representations (scene graph or data table). 4. 3 Semantic Prefix-Tuning
In real-world chart captioning settings, users want to vary the level of semantic content in their captions. For instance, while some blind users want verbose captions that describe the chart visuals, sighted users may only want captions that help them expose data trends (Lundgard and Satyanarayan, 2022). To develop models capable of such customization, we leverage prefix-tuning strategies alongside VisText’s semantic caption breakdown. Prefix-tuning specifies a task alongside the input, permitting a single large language model to perform many different tasks. In our setting, we use prefix-tuning to specify the level of semantic content to include in the caption (Li and Liang, 2021). We train each of our models with and without semantic prefix-tuning. With semantic prefix-tuning, we treat chart captioning as a multi-task fine-tuning problem, where the model is trained to generate the L1 and L2/L3 captions separately. In every epoch, the model sees each VisText chart twice, once with
7273
the L1 prefix and caption and once with the L2/L3 prefix and caption. 5 Evaluation and Results
To evaluate the VisText dataset and our chart captioning models, we measure the readability and accuracy of generated captions and their similarity to the VisText target caption. We also qualitatively analyze the descriptiveness of generated L2/L3 captions and categorize common errors. 5. 1 Quantitative Model Performance
We evaluate the results of our text-based and imageguided captioning models with and without prefixtuning. We also compare to a current state-of-theart chart captioning model that uses data table chart representations and a T5 generation model (Kantharaj et al. , 2022). To measure the quality of output captions, we evaluate each model on machine translation and language generation metrics (Table 1). Chart images do not support captioning. The image model performs the worst of all the chart captioning models. Its low perplexity and high error rates indicate it is highly confident in its inaccurate captions. While chart images contain the same information encoded in the chart’s textual representations, it is presumably not adequately extracted by the model. Both the image model backbone (Cho et al. , 2021) and the visual feature extractor (Anderson et al. , 2018) are trained on natural images, making chart images out-of-distribution inputs that are likely to be poorly represented by these vision models. As the chart captioning task grows, model backbones, architectures, and feature extractors could be customized to chart images, which may improve image-based chart captioning. All models produce high quality L1 captions. In our chart captioning setting, relation generation (Wiseman et al. , 2017) measures how often the chart title, axis names, and axis scales in the input appear in the caption. Every model (except image) achieves a similarly-high relation generation score, indicating that every model can generate detailed L1 captions. Scene graphs perform as well as data tables. Models trained on scene graph representations achieve similar performance across the evaluative metrics to models trained on data tables. As scene graphs can be more easily extracted from web-
based charts images, they may be the preferred representation for future chart captioning models. Image-guiding does not improve captioning. Our image-guided captioning models do not experience the significant increase in performance other image-guided translation tasks report. While in image-guided translation, images contain substantial additional information beyond the text, the image and textual representations in chart captioning often contain highly similar information. The small amount of additional information in images might benefit complex captioning tasks on multivariate charts or infographics; however, the current VisText captions rarely reference visual information not present in the scene graph or data table. Prefix-tuning is free. Adding semantic prefixtuning to our models does not significantly change their performance. Models trained with and without prefix-tuning are exposed to the same set of charts, so it is consistent that prefix-tuning would not impact the quality of output captions. Given prefix-tuned models are able to output L1, L2/L3, and L1+L2/L3 captions, prefix-tuning may be preferred if users require semantic customization. 5. 2 Qualitative Caption Evaluation To augment our quantitative evaluation, we qualitatively assess the descriptiveness and accuracy of the generated chart captions. Since L1 caption accuracy can be measured at scale via relation generation, we focus our evaluation on L2/L3 predictions. Prior analysis tasked annotators with comparing the accuracy, coherence, and fluency of generated captions compared to a target caption (Kantharaj et al. , 2022). Instead, our approach follows an inductive qualitative data analysis approach: iteratively analyzing captions in a “bottom-up” fashion to identify emergent patterns in how generated captions compare to the ground truth (Bingham and Witkowsky, 2021). We randomly sample 176 generated captions from the scene-graph model with prefix-tuning and break them into their independent L2 and L3 statements, resulting in 181 (48. 27%) L2 statements and 194 (51. 73%) L3 statements. Approximately half (241 / 512) of the L2 and L3 statements made in the generated captions are factually accurate. Moreover, many of the full sentences are written in a natural, human-like manner and generated captions frequently include both compound and complex sentences. On average, every generated caption has one L3 statement and zero to
7274
two L2 statements. Often this takes the form of a L3 general trend statement (e. g. , “The median annual family income in Canada has increased from 2000 to 2018”) accompanied by an L2 minimum and maximum statement (“The highest was in 2015 at 80k and the lowest was in 2000”). For the remaining half of analyzed captions, we identified the following recurring types of errors:
Identity Errors. We identify 86 identity errors (22. 93% of analyzed statements). An identity error occurs when an L2 or L3 statement incorrectly reports the independent variable for a given (often correctly identified) trend. For bar charts, this error means incorrectly reporting the categorical label associated with a bar (e. g. , in Appendix Figure 5c: “The most popular music activity is vinyl albums and vinyl singles” should be “The most popular music activity is tickets for festivals”). For area and line charts, this error means incorrectly identifying the temporal point or range of the trend. With bar charts, in particular, we observed that the identities were often “off-by-one” (i. e. , identifying a minimum or maximum value, but attributing it to the second-highest or second-lowest category). Value Errors. A value error occurs when the quantitative data value of a statement is incorrect. Of the captions we analyzed, 3. 20% (12) of statements contained a value error. For instance, as shown in Appendix Figure 4c, for the caption “The total gate revenue from sporting events worldwide by region from 2006 to 2015 has increased from around 15 billion dollars to around 15 billion dollars”, the value should be around 18 billion dollars. If it is ambiguous whether an error is an Identity or Value Error, we classify it as the former. Direction Errors. A direction error occurs when the direction (which can be increasing, decreasing, or stable) of a trend in an L3 statement is incorrect. We uncovered 32 direction errors (8. 53% of analyzed statements). For instance, in the caption “The per capita consumption of sweet corn in the US has increased from 2000 to 2019” (Appendix Figure 3c), the trend is actually decreased. In most direction errors, the identity (i. e. , temporal range) is correct. Stability Errors. A stability error occurs when the magnitude of a direction or the variance in a trend is incorrect. This can often refer to how much a trend is increasing or decreasing, such as rapidly or slowly, as well as whether it’s a steady change or highly-fluctuating change. In Appendix Figure 4b, “The comparable sales growth of Sam’s Club in the United States from fiscal year 2006 to 2020 has been steadily decreasing from 2006 to 2020. ” should read “The comparable sales growth of Sam’s Club in the United States from fiscal year 2006 to 2020 has been highly-fluctuatingly decreasing from 2006 to 2020. ” 1. 07% (4) of the statements we analyzed contained this error. Repetition. Repetition is when a caption repeats a previously-generated claim, regardless of its correctness. 117 (31. 2%) statements contained repetition, making it the most common error we encountered. For example, in Appendix Figure 4a, we see "The average age at widow hood in the Netherlands has increased from 2008 to 2018. The average age at widow hood in the Netherlands has increased from 2008 to 2018. " Repetition is a known problem for text generation models with transformer backbones, like our chart captioning models (Fu et al. , 2021). 7275
Nonsensical Errors. If a L2 or L3 statement cannot be understood in context of the chart, or makes a fundamental mistake in interpretation, we label it as nonsensical error. We encountered 20 nonsensical errors in addition to the 395 statements we analyzed. For example, in Appendix Figure 5b, "The most popular visitors was Harry Potter in 1999 and 2009. " misinterprets the chart. It might instead correctly read "The destination with the most visitors after the TV/movie’s release was New Zealand for The Lord of the Rings"