UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Authors: Ahmed Masry
Parsa Kavehzadeh
Do Xuan Long
Enamul Hoque
Shafiq Joty
Venue: EMNLP 2023 Main (2023)
Keywords: Charts
Pretraining
TL;DR: We present a pretrained universal chart understanding model along with a large-scale pretraining dataset.

Abstract: Charts are widely used for data analysis, providing visual representations and insights into complex data. To facilitate chart-based data analysis using natural language, several downstream tasks have been introduced recently such as chart question answering and chart summarization. However, existing methods for these tasks often rely on pretraining on language or vision-language tasks, neglecting the explicit modeling of chart structures (e.g., how chart elements are related to each other). To address this, we first build a large corpus of charts covering diverse topics and visual styles. We then present UniChart, a pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual elements of charts and then uses a chart-grounded text decoder for text generation. We propose several chart-specific pretraining tasks that include: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart understanding and reasoning skills. Our experiments demonstrate that pretraining UniChart on a large corpus with chart-specific objectives, followed by fine-tuning, yields state-of-the-art performance on four downstream tasks. Moreover, our model exhibits superior generalizability to unseen chart corpus, surpassing previous approaches that lack chart-specific objectives and utilize limited chart resources.

UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning
Ahmed Masry
Parsa Kavehzadeh Xuan Long Do Enamul Hoque", Shafiq Joty"
York University, Canada
Nanyang Technological University, Singapore, "Salesforce AI
{ parsaka, enamulh] @ @yorku.ca
srjoty @ ntu. ntu.edu.sg
dhmee.elmaasry246533 @ gmail. com, xuanlong.do do @ @u.nus.edu
Abstract
charts have been proposed recently, including chart question answering (Masry al., 2022; Kantharaj et al., 2022; Lee et al., 2022), natural language gen- eration for visualizations (Obeid and Hoque, 2020; Shankar et al., 2022) and automatic data story gen- eration (Shi et al., 2020).
Charts are widely used for data analysis, pro- viding visual representations and insights into complex data. To facilitate chart- based data analysis using natural language, several down- stream tasks have been introduced recently such as chart question answering and chart summarization. However, existing methods these tasks often rely on pretraining on lan- guage or vision-language tasks, neglecting the explicit modeling of chart structures (e.g., how chart elements are related to each other). To ad- dress this, we first build a large corpus of charts covering diverse topies and visual styles. We then present UniChart, pretrained model for chart comprehension and reasoning. UniChart encodes the relevant text, data, and visual ele- ments of charts and then uses a chart-grounded text decoder for text generation. We propose several chart-specific pretraining tasks that in- clude: (i) low-level tasks to extract the visual elements (e.g., bars, lines) and data from charts, and (ii) high-level tasks to acquire chart un- derstanding and reasoning skills. Our exper- iments demonstrate that pretraining UniChart on a large corpus with chart-specific objectives, followed by fine-tuning, yields state-of-the-art performance on four downstream tasks. More- over, our model exhibits superior generalizabil- ity unseen chart corpus, surpassing previous approaches that lack chart-specing objectives and utilize limited chart resources.
A dominant strategy to tackle these downstream tasks is to utilize pretrained models (Su et al., 2020; Li al., 2020b; Kim et al., 2021; Cho et al., 2021) trained on langauge and vision tasks (Du al., 2022). However, although effective, such mod- els may not optimal for chart-specific tasks be- cause they are trained on large text corpus and/or image-text pairs without any specific focus on chart comprehension. In reality, charts differ from natu- ral images in that they visually communicate the data using graphical marks (e.g., bars, lines) and text (e.g., titles, labels, legends). Readers can dis- cover important patterns, trends, and outliers from such visual representation (Munzner, 2014). Exist- ing pretrained models do not consider such unique structures and communicative goals charts. For instance, Pix2Struct (Lee et al., 2022) is a pre- trained image-to-texs model designed for situated language understanding. Its pretraining objective focuses screenshot parsing based HTML codes of webpages, with primary emphasis on layout understanding rather than reasoning over the visual elements. MatCha (Liu et al., 2022h) extends Pix2Struct by incorporating math reason- ing and chart data extraction tasks, but it still lacks training objectives for text generation from charts and it was trained
1 Introduction
Information visualizations such as bar charts and line charts are commonly used for analyzing data, inferring key insights and making informed deci- sions (Hoque et al., 2022). However, understand- ing important patterns and trends from charts and answering complex questions about them can be cognitively taxing. Thus, facilitate users in ana- lyzing charts, several downstream NLP tasks over
limited number of charts.
In this work, we present UniChart, pretrained model designed specifically for chart comprehen- sion and reasoning. UniChart is pretrained large corpus of charts and it aims to serve as a Uni- versal model for various chart-related downstream tasks (Fig. 1). Inspired by the model architecture from Kim et al. (2022), UniChart consists of two modules: (7) a chart encoder, which takes the chart
"Equal contribution.
Now affiliated with NUS.

---

Input Image and Prompts
UniChart
Quiput
The trand abiza decitw in bath nose incame levela with the lighost bith rat in 2006 wonion bolow the povorty level at 95, and the
woree incoino 200 pencort
Chart Image Encoder
44.
The line chin shows the rate Liikzd Brates from
numbere reprosavt
bittes
worrnn In 2010, the aith for woren below povelty lowel 74, for sommen with income 100 to persent the poisery level Df was Iie over
with the higheet tirm alowy decine barth rakey aal recors women helnw pownrty
isstoiing two krhmost <lonswors
hopaneended quastion amsoaregg What beich rode tur U.S. from 2015 io 20190 shanneer
Text Decoder
Ciinnneeeee e Troome Tekos
Tncoos 100 150 percent of
powerty weed Incore 300 pertent powerty lavee or sroem & 2010 1M
2015 D0R1 81 22 40 2010 B5 D4 45 2020 (P0 (S 146 2008) 196 179
2015 ca & 2017 922 46 2010 78
sc0l tabl genoraiioo obnswer.
2007 1922 190 46 2002 1911 600 2000 15b
Our UniChart model with different pretraining objectives. The model consists of two main modules: Chart Image Encoder, and Text Decoder. Four different pretraining objectives specified in different colors; data table generation. chart summarization, numerical and visual reasoning, and open-ended question answering.
Figure I:
image as input, and (2) text decoder, trained to decode the expected output based on the encoded image and the text input fed in the decoder as task prompt. We performed pretraining on a diverse set of 61 1K charts that we collected from multiple real-world sources. Our pretraining objectives in- clude both low-level tasks focused on extracting visual elements and data from chart images, as well as high-level tasks, intended to align more closely with downstream applications. One key challenge for pretraining was that most charts in the corpus do not come with informative summaries, which are critical for various downstream lasks. To address this challenge, we used knowledge distillation tech- niques to leverage large language models (LLMs) for opportunisticalyy collecting chart summaries, which were then used during pretraining.
to charts; (ii) a large-scale chart corpus for pretrain- ing, covering a diverse range of visual styles and topies; (iii) extensive automatic and human evalu- ations that demonstrate the state-of-the-ar perfor- mance of UniChart across various chart-specific benchmark task while optimizing time and mem- ory efficiency. We have made our code and chart corpus publicly available at https//github.com/iis nlp/UniChart.
2
Related Work
2.1
Vision-language Pretraining
Pretrained models have dominated in many vision and language tasks (Du et al., 2022). Building a pre- trained vision-language model typically involves three steps. First, textual input is usually encoded using BERT-based encoder (Lu et al., 2019; Rad- ford al., 2021; Li et al., 2021, 2022). Second, for the input image, some prior studies utilize Fast- RCNN (Ren et al., 2015) to encode the sequence of object regions as the image features (Li et al., 2019; Lu et al., 2019; Chen et al., 2020). However, this method may neglect some crucial regions in an image. Recent approaches favor encoding the im- age whole (Huang et al., 2020, 2021; Li et al., 2021, 2022) by using ResNet (He et al., 2016) or ViT (Dosovitskiy et al., 2021). Third, to fuse the textual and visual features, prior work mostly either designs a fusion encoder (Tan and Bansal, 2019; Su et al., 2020; Cho et al., 2021; Kim et al., 2021) or a dual encoder (Radford et al., 2021; Jia et al., 2021; Li et al., 2022). Finally, multiple common cross-modal pretraining tasks have been designed such image-text matching (Chen et al., 2020; Li al., 2020a), cross-modal contrastive learning (Radford et al., 2021; Jía et al., 2021) and genera- tion tasks such as visual question answering (Cho et al., 2021; Wang et al., 2021).
We conducted extensive experiments and analy- sis on various chart-specific downstream tasks to evaluate the effectiveness of our approach. Specifi- cally, we evaluated UniChart on two chart question answering datasets, ChartQA (Masry et al., 2022) and OpenCQA (Kantharaj et al., 2022), and found that it outperformed the sfute-o-the-wee models in both cases. For chart summarization, UniChart achieves superior performance in both human and automatic evaluation measures such as BLEU (Post, 2018) and ratings from ChatGPT (OpenAI, 2022). Moreover, UniChart achieved state-of-the--at re- sults in the Chart-to-TT Table downstream task. Fi- nally, our model showed improved time and mem- ory efficiency compared the previous ofate-of- the-art model, MatCha, being more than 11 times faster with 28% fewer parameters.
Our primary contributions are: (i) A pretrained model for chart comprehension with unique low- level and high-level pretraining objectives specific

---

Our work is also related to multimodal docu- ment understanding lasks that involve analyzing the textual content, layout, and visual elements of documents (Xu et al., 2020b,a; Wang et al., 2022; Huang et al., 2022; Kim et al., 2022; Tang et 2022). These tasks can be addressed using encoder-only and encoder-decoder architectures. Encoder-only models rely on OCR engines to ex- tract text from document images and use BERT- like encoders augmented with specialized embed- dings to encode layout and visual features (Xu et al., 2020b,a; Wang et al., 2022; Huang et al., architectures
2019). We evaluate our model on these four chart- related tasks, as they involve the interaction be- tween language and vision and have publicly avail- able datasets. There are a few other tasks such as in- fographics understanding (Mathew et al., 2022) and question answering with science diagram (Kemb- havi et al., 2016), however, in this work, we only focus on chart-related tasks.
Chart Pretraining Corpus
To build a large and diverse corpus with various styles, topics, and storage formats, we crawled charts from various online sources. Additionally, we utilized publicly available chart datasets suit- able for pretraining. The collected charts can be categorized into two types: charts with underlying data tables and charts without data tables.
2022). In contrast, encoder- combine transformer-based encoders with autore- gressive text decoders for text generation tasks re- lated to documents (Tang et al., 2022; Kim et al., 2022: Lee et al., 2022). While Tang al. (2022) incorporates an OCR tool to supplement the vision encoder, Kim et al. (2022) and Lee et al. (2022) operate in an end-to-end manner without external OCR engines. In line with the latter approach, our model adopts an end-to-end decoder- archi- tecture (Kim et al., 2022).
3.1 Charts with Data Tables
Charts with an underlying data table are collected in three ways: (i) utilize existing datasets, (ii) ex- tract SVG charts, and (iii) data augmentation.
Utilize Existing Datasets Our goal was train the model based on real-world data, thus, we did not consider the ones that are generated from synthetic data (Kafle et al., 2018; Kahou et al., 2018). In particular, we used the following five chart datasets for which the underlying data tables were available: (i) Statista (statista.com) (Shankar et al., 2022), (ii) Our World In Data OWID (our- worldindata.org) (Masry al., 2022), (iii) Organ- isation for Economic Co-operation and Develop- ment or OECD (oeed.org) (Masry et al., 2022). (iv) PlotQA (Methani et al., 2020), and (v) a subset of the ChartInfo (ChartInfo, 2022) dataset that pro- vides bounding box annotations for data encoding marks (c.g., bars in bar chart).
In general, the above work focuses on training on large image-text pairs or text corpus, lacking focus on chart understanding. One exception is MatCha (Liu et al., 2022b), pretrained chart model based on Pix2Struct (Lee et al., 2022), which achieved SoTA on chart question answering and summariza- tion tasks. However, MatCha's 's pretraining tasks mainly focus on data table generation without fo- cusing text generation tasks. The model is also pretrained with reasoning tasks using the textual datasets which might limit its visual reasoning abil- ity. Our model trained on a larger corpus with chart-specific pretraining objectives, including vi- sual reasoning and text generation, making it more versatile for various chart-related tasks.
Extract SVG Charts: We extracted charts in SVG format from the Chartblocks and Plotly datasets of the Beagle corpus (Battle et al., 2018). These charts do not come with data tables, but the data can be extracted accurately from the SVG ele- ments. The steps for preparing these charts are: (1) identify axis labels and legends using specific class names of HTML attribute, (2) extract bounding boxes of chart elements (e.g., bars, line) using SVG attribute properties (c.g., size and Locatio on of <rect>), (3) construct the underlying data table by iterating through each of the <g> elements find data values of each data attribute. When data labels are absent, we utilize the scale information based on the axis labels and tick marks of the chart
2.2
Chart-related Downstream Tasks
There has been growing interest in solving var- ious chart-related tasks. Chart question answer- ing (ChantA) tackles questions about charts, with benchmarks like (Methani et al., 2020) and (Masry et al., 2022) targeting factoid questions involy- ing visual and arithmetic reasoning. Open-ended question answering (OpenCQA) task requires an explanatory answer by reasoning with the chart content (Kantharaj et al., 2022). Finally, Chart-to- Text generates natural language summaries from input charts (Shankar et al., 2022), while Chart-to- Table generates underlying data tables (Choi et al.,

---

and the bounding box information of data encoding marks to recover the data values.
3.2 Charts without Data Tables
Many online charts are available only images, without corresponding data tables. However, they can still be valuable for large-scale pretraining as we can extract chart elements and rich textual con- tents (e.g., titles, surrounding texts, captions) us- ing object detection and optical character recog- nition (OCR) techniques. We collected image chart datasets such as LineCap (Mahinpei et al., 2022) and Neural Caption Generation (Spreafico and Carenini, 2020) since they provide high-quality summaries. We also used the Pew dataset from (Shankar et al., 2022) and further augmented it by an crawling additional IK charts. Finally, we used the ExcelChart400K dataset (Luo et al., 2021) which only provides bounding boxes without under- lying data tables. We also considered other existing image chart datasets such as Vis30K (Chen et al., 2021) and Vislmage (Deng al., 2020), but they are suitable as they usually have poor resolution and lack meaningful textual content (e.g., titles).
Data Augmentation We further augmented the corpus by creating charts from publicly avail- able data tables. We used the The Web Data Com- mons (WDC) (WDC, 2022), which used Common Crawl' to collect a large amount of structured data. The charts are created in the following steps:
(i) Data pre-procee pre-processing: Since many tables in WDC contain more than three columns, we decom- posed so that tables are suitable for creating desired chart types (e.g.. bars, lines, and pie charts). In par- ticular, we automatically analyze the data type of each column (e.g, numeric categorical) and then randomly choose one column with numeric data values and one/two column(s) with categorical data. We also limit the maximum number of rows of the table to 8 so that the corresponding chart can fit within reasonable screen space.
(ii) Chart generation: To generate visually diverse charts, used the D3 (Bostock et al., 201 1) library that provides great flexibility in terms of creating diverse visualization styles. We also employed Vega-Lite (Satyanarayan et al., 2016) which creates charts based on declarative JSON syntax. We used simple heuristics for determining chart types from the data table (Mackinlay et al., 2007). We created four types of charts: (I) vertical simple bar charts with one numeric data column, (2) vertical grouped bar charts, (3) pie charts, and (4) line charts (both single series and multi-scries).
3.3 Augmentation by Knowledge Distillation for Chart-to-text Generation Tasks
Chart-related downstream tasks such as chart sum- marization (Shankar et al., 2022) and open-ended question answering (Kantharaj et al., 2022) require generating informative and relevant texts. How- ever, for most of the charts in the pretraining cor- pus, there are either no associated summaries or the summaries that are collected opportunistically such as the Statista dataset (Shankar et al., 2022) lack quality (e.g., too short and not very infor- mative). Training on such substandard "ground- truth" summaries can negatively affect the overall model performance as shown in text summariza- tion (Kryscinski et al., 2019; Clark et al., 2021). Indeed, Goyal et al. (2022) and Liu et al. (2023b) have recently shown that human raters prefer sum- maries generated by LLMs, especially the ones that instruction-tuned such InstructGPT (Ouyang et al.. 2022), compared to the reference summaries in various text summarization datasets. Conse- quently, the instruction-tuncd LLMs have been successfully used as annotator in several recent studies (DING et al.. 2023; Qin et al., 2023).
(iii) Visual diversification: To create visually di- verse charts resembling real-world variations, we manipulated the following visual style properties: (I) Colors and shapes: Color schemes from Color- Brewer² and Tableau² 3 were chosen for categorical data attributes. We also varied shape properties such bar thickness, line types (e.g.. continuous vs dotted), and legend shape types (e.g., rect, cir- cle). (2) Position and distance: We also varied bar positions and distances with respect to axis la- bels. (3) Guides: Charts may contain additional guides such as grids, so we generate charts with and without grids to diversify styles.
Fig. 2 depicts visually diverse set of charts created using this augmentation process. In total, we created a total of 189,839 charts (Table 4).
Inspired by these findings, we leveraged Instruct- GPT to generate coherent and relevant text. Specifi- cally, we prompted text-davi new-iinnii00 by pro- viding the underlying data table as input and one exemplar (i.e., I 1-shot in-context learning). Since
'ttpss//coommoncrawwooggg
bttp:://colorbbeeeer.oorg/ tablcau.com

---

generating summaries for thousands of charts by calling OpenAI API quite costly, we devised a knowledge distillation approach. We first used text-davincii 33 to create a small dataset of 3700 summaries for different chart types. Next, we finetumed Flan-T5 XL (Chung et al., 2022) on this dataset. Finally, we utilized the finetuned Flan-TS model to generate summaries for charts that do not have an associated summary. More details about this approach can be found in Appendix A.2.
Chart Image Encoder In order effectively encode chart image, an encoder needs identify and interpret three different types of chart compo- nents: (1) textual elements (axis labels and leg- ends), (2) visual elements (e.g., bars, lines), and (3) the layout that arranges textual and visual el- ements within chart. Since this has similarity with document image (e.g.. receipts) understanding, our chart image encoder builds upon the encoder of one of the recent state-of-the- document image understanding models, Donut (Kim al., 2022). Donut offers an OCR-free architecture. The model is pretrained using an OCR-pscudo task, where it sequentially generates the encoded text in document image, following the order from the top-left corner to the bottom-right corner of the image. As a result, we did not have to run an exter- nal OCR module like CRAFT (Back et al., 2019) and Parseq (Bautista and Atienza, 2022), which improved time and memory efficiency throughout our training pipeline. Donut employs Swin Trans- former (Liu et al., 2021) architecture the im- age encoder. To encode the chart image features, the images are split into non-overlapping patches, which are then processed using shifted window- based multi-headed self-attention and MLP layers to produce the image embeddings.
3.4 Datasets Analysis
Our chart pretraining corpus has over 611K charts covering a diverse range of bar charts, line charts, and pie charts (Table 4). Data tables of Simple charts have two columns (simple bar charts or single-series line charts), whereas Complex charts involve least three columns (e.g., stacked or group bar charts, line charts with multiple lines). The first two chart groups in Table 4 come with an underlying data table which cover over 80% of the corpus. The bottom group contains five datasets which only provide charts in image for- mat without a data table" and cover about 20% of the corpus. Bar charts make up the majority portion (58.51%), followed by line charts (32.94%) and pic charts (9.39%). About 60% of the charts have mul- tiple columns in their data tables, while 40% of the charts have only two columns. 5 The corpus also covers a diverse range of topics including tech- nology, economy, politics, health, and society. To ensure a fair evaluation, we excluded charts found in the validation and test sets of the downstream tasks from our pretraining corpus. Details about the linguisties of the corpus textual elements can be found in Appendix A.3.
Text Decoder Similar Donut (Kim al., 2022), we use the BART (Lewis et al., 2019) de- coder for generating the output. The textual (task- specific) prompts are fed to the decoder and the decoder has to generate the output by conditioning on the prompted context (see Fig. 1).
4.2 Pretraining Objectives
Our pretraining objectives include low-level tasks that more focused on retrieving the underlying data from the chart images and high-level tasks that align closely with the downstream tasks.
4 Method
We propose UniChart, unified pretrained model for chart comprehension and reasoning. This sec- tion first introduces the UniChart architecture fol- lowed by its pretraining objectives.
Data Table Generation A chart creates vi- sual representation of a data table by mapping each data attribute (e.g.. 'country', 'population') to cor- responding visual attributes (e.g.. x positions, he hettt of graphical marks (e.g, bars). An ef- fective chart comprchension and reasoning model should be able to deconstruct the structured under- lying data table by recovering such mappings. To this end, we propose the data table generation task in which we ask the model to generate the flattened data table given a chart image.
4.1 Model Architecture
UniChart consists of two main modules: a chart im- age encoder and text decoder shown in Fig. 1.
"The ExcelChart400K dataset only provides bounding box annotations of chart elements and we used this dataset for data value estimation task during pretraining.
"Since we do not have access to the chart types of Pew datasct, we manually tagged random 200 samples from each of these datasets to estimate the chart type distribution.
A vast amount of charts available online are

---

tence containing the answer in the summary then serves as the answer its generated question.
Gracration
Summnatzaioo
1,294
Satiea, ORED,, OWID
144,147
675/420
126
12A/00
155
2,414,599
157,100
155,000
Chart Summarization Image captioning is fundamental problem in Al in which the machines need to summarize the main content of the image in the textual form. This task has been studied ex- tensively (Vinyals et al., 2015; Herdade et al., 2019; Hu et al., 2021; Li et al., 2022). We follow previ- ous work (Vinyals et al., 2015; Xia et al., 2021) to pretrain our model on this task to further en- hance the model's capability in generating textual descriptions from the chart image. As discussed in $3.3, we used mostly the summaries generated from GPT models provided by OpenAl either di- rectly or through a knowledge distillation step.
LindCap
2
2,821
Nourd Captin
130
305
Beagle
3,972
5]
Charilado
1,76
21,009
Dut Aug
199.792
2218,468
189,002
159,502
EuoCChrr
100,977
Toul
001,086
5,354.47
481J897
481,303
Table Numher of examples for each task pretraining.
stored as bitmap images without access to the un- derlying data. It is important learn how to re- cover data values when the chart data is not avail- able. Therefore, we also introduce the data value estimation task, in which the model is asked to gen- erate the scale of the graphical marks (e.g., bars, line points) as a percentage of the chart plot area. We obtain these scales by dividing the bars or line points heights (bounding boxes) by the height of the chart plot area and rounding the result to two decimal places. At the final stage. we use charts for which both data tables and object bounding boxes are available as well as charts for which at least the bounding box annotations are available, e.g., ExcelCharts from (Luo et al., 2021).
4.3 Downstream Tasks
In addition to zero-shot evaluation, we also adapt UniChart by finetuning it on a downstream task. We consider four downstream tasks: (1) Fac- toid Chart Question Answering: we use ChartQA (Masry et al., 2022), which is benchmark con- sisting of factoid question-answer pairs for charts with particular focus on visual and logical rea- soning questions; (2) Complex Chart Question An- swering: we consider OpenCQA (Kantharaj et al., 2022), another QA benchmark which answers are explanatory descriptions; (3) Chart Summariza- tion: we use Chart-to-Text (Shankar et al., 2022), a large-scale benchmark for chart summarization; (4) Chart-to-Table: we use ChartQA for both fine- tuning and evaluation. Morcover, we evaluate the pretrained model in zero-shot setup on the We- bCharts dataset (Choi al., 2019), collection of 300 charts obtained from the web.
Numerical & Visual Reasoning Many down- stream applications over charts may involve numer- ical and visual reasoning with the chart elements such as chart QA and summarization. For example, the model may need to apply series of mathe- matical and logical operations such as addition, subtraction and comparisons to answer question. To inject such reasoning skills into the model, we design template-ber based numerical reasoning tasks where the model is trained to execute/perform the most common mathematical operations over the chart data values. We manually analyzed the exist- ing task datasets (c.g., CharlQA) to find the most common operations (e.g., sum, average, difference, etc.) and constructed 90 templates that we utilize to generate synthetic question and answer pairs. All the templates are provided in Appendix A.8.
4.4 Experiments Setup
To minimize the computational resource require- ments, we initialize our model from the base Donut weights (Kim et al., 2022). Our pretraining process consists of two stages. In the first stage, we set the input image resolution to 512x512 and pretrain for 300K steps. In the second stage, we increase the input image resolution to 960x960 and pretrain for an additional 100K steps. Table 6 shows the hyperparameters we used in pretraining and fine- tuning our model on each downstream task. All our experiments were carried out using one 4-A100 (40GB), one 4-A100 (80GB), and one 4- 4-V100 (32 GB) GPU machines.
Open-ended Question Answering It is very common for users to ask open-ended questions over charts (Kantharaj et al., 2022). Such questions often ask for answers that require high-level reason- ing and explanations. To improve the capability of the model in answering open-enn spen-ended questions, follow previous work (Shi al., 2022) to generate synthetic open-ended QA pairs. Specifically, a T5 model (Raffel et al., 2020) pretrained on SQuAD (Rajpurkar et al., 2016) is employed to generate an open-ended question for each summary. The sen-

---

ChanQA
OpenCQA
Chart-to-Tedd Text
Chart-to-Table
(RA)
(BLEU)
(BLEC)
(RVS8 (
Model
#Parums
aug.
human
avg.
OpenCQA
Pew
Statista
CharQA
WebCharts
VisionTaPPa (Masry el al., 2022)
61.44
45.52
'T (Masry al., 2022)
222M
56.96
25.12
41.04
9.28
10.49
35.29
VL-T5 (Masry 2022)
56.88
26.24
41.56
14.73
Pix2Struct (Lee et al., 2022)
282M
81.6
30.5
56.0
10.3
38.0
MarCha (Liu al., 2022b)
282M
90.2
38.2
64.2
12.2
39.4
85.21
183.49
44.37 17.94
UniChari
201M
88.56
43.92
66.24
14.88
12.48
38.21
94.01 I 191.10
60.73 | 43.21
Table Evaluation results on four public benchmarks: ChartQA, Chart-to- Text, OpenCQA, and Charl-to-Table are calculated after finetuning UniChart pretrained checkpoint except for WebCharts (zero-shot).
results
5 Evaluation
Summary
Human
ChatGPT
p-value
UniChart ZeroShot
3,97
3.18
1.70c-10
5.1 Baselines & Evaluation Metrics
UniChart Finetuned
2.86
2.32e-8
MatCha (I.iu et al., 2022h)
2,50
2.18
0.0020
We compare our model against five baselines: (1) T5 (Raffel et al., 2020), unified seq2seq Trans- former model that achieved state-of-the-art (SoTA) results on various text-to-text tasks, including ques- tion answering and summarization; (2) VL-T5 (Cho al., 2021), a T5-based model that unifies Vision- Language (VL) tasks as text generation conditioned on multimodal inputs and achieved SoTA results on OpenCQA (Kantharaj et al., 2022); (3) VisionTapas (Masry et al., 2022), an extension of TaPas (Herzig et al., 2020), SoTA table encoder, adapted for QA over charts; (4) Pix2Struct (Lee et al., 2022), pretrained image-to-text model for visual language understanding and achieved SoTA results on doc- ument understanding tasks; and (5) MarCha (Liu et al., 2022b), an adapted version of Pix2Struct for charts that is further pretrained on math reasoning and chart data extraction tasks, achieving SoTA results on Chart-to-Texw (Shankar et al., 2022) and ChartQA (Masry et al., 2022).
Gold (Shankar 2022)
3.19
2.73
2.13e-6
Table 3: Average Informativeness scores from Human and ChatGPT-based evaluation.
metrics to evaluate the Chart-to-Table task.
5.2 Main Results
As shown in Table 2, UniChart outperforms previ- ous state-of-the-an models, MatCha and VL-T5, on the ChartQA and Chart-to- (Pew) datasets, although shows slightly lower performance on Chart-to- Text (Statista). The performance gap is more prominent on the challenging human-written questions in the ChartQA benchmark (Masry et al., 2022), where our model's pretraining objectives tailored to visual and numerical reasoning give it significant advantage. UniChart also achieved a higher BLUE score compared to the SoTA VI.-T5 model on OpenCQA benchmark, which demon- strates our model's capability in generating ex- planatory answers for questions about charts. Fi- nally, UniChart surpasses MatCha's 's performance on two datasets, demonstrating its generalizability across diverse visual styles, even in zero-shot setup on unseen charts (WehCharts). Overall, these results establish UniChart as the SoTA model for chart comprehension and reasoning tasks.
To evaluate our approach, we follow previous works (Lee et al., 2022; Shankar et al., 2022;
Masry et al., 2022; Kantharaj al., 2022; Liu al., 2022b) and utilize Relaxed Accuracy (RA) for CharlQA and BLEU (Post, 2018) for text- generation tasks (Chart-to-Text and OpenCQA). However, the BLEU score has limitations as primarily focuses on n-gram matching between the generated and reference texts, overlooking im- portant factors such as semantic similarity, infor- mativeness, and factual correctness (Goyal et al., 2022). Therefore, conduct human evaluation and ChatGPTTdriive study to assess and compare these crucial aspects in the outputs of different mod- els ($5.3). Finally, we use Relative Number Set Similarity (RNSS) (Masry et al., 2022) and Rela- tive Mapping Similarity (RMS) (Liu et al., 2022a)
To further assess the impact of our different pre- training objectives on our model's performance, we conducted ablation studies. We observe that re- moving various pertaining objectives led to slight decrease in performance (Table 8). The decrease in performance particularly noticeable when the Numerical Reasoning pretaining task is removed, highlighting the importance of this task in imbuing numerical abilities into our model. More details of this experiment can be found in Appendix A.4.

---

5.3 Human and ChatGPT Evaluation
and ChatGPT, with the null hypothesis that the rat- ings are two independent samples. The p-values in each row Table 3 demonstrate that it is very infrequent that two rating samples are independent based on the observed ratings. Also in terms of different semantic contents, the ZeroShot model tends to contain more sentences with high-level visual patterns and trends. A previous study finds that such high-level insights lead to more reader takeaways compared to the text describing low- level visual encodings like axes and colors (Stokes et al., 2022). Overall, the results above suggest that UniChart model's summaries are more informative with high-level insights and factually accurate than the SoTA (MatCha).
As discussed in $5.1, reference-based metrics like BLEU have relatively low correlations with hu- man judgments (Belz. and Reiter, 2006; Tan et al., 2015; Liu et al., 2023a), and generated texts with very high such scores can be of very poor quality (Smith et al., 2016). Therefore, we decided to con- duct human evaluation to measure the quality of summaries generated by different models. We fo- cus on following criteria in the chart summarization task:(1) Informativeness: (2) Factual Correctness; and(3) Semantic Levels that characterize the con- tent of the summary. More details about the criteria be found in Appendix A.5.
We randomly picked 150 sample charts from Chart2text Statista test split and asked 3 human annotators rate four summaries for cach chart based on informativeness out of 5. The order of exposure of summaries to the annotator was ran- domized to avoid any potential bias. Summaries for each chart were rated by one annotator except for the first 100 charts for which we had two an- notators to measure the agreement. We computed Krippendorff's ss alpha (Krippendorff, 201 1) to mea- sure inter-annotator agreement and found mod- erate level of agreement with alpha coefficient of 0.54. We further utilize ChatGPT for evaluating the same 150 samples, as LLMs have demonstrated their effectiveness as evaluators for text generation tasks (Luo et al., 2023; Liu et al., 2023a; Gao et al., 2023; Fu et al., 2023). We define the informative- ness criteria and rating scheme to ChatGPT and then employ ChatGPT to generate evaluation steps. We then send these evaluation steps along with the data table of the chart and the summary ChatGPT obtain ratings (see Appendix A.5 for details).
5.4 Time and Memory Efficiency
UniChart exhibits significant time efficiency com- pared to MatCha, as shown in Fig. 4. The gap in speed is more evident on tasks that require the generation of long output sequences (e.g.. Chart-to- Text). This difference in speed can be attributed MatCha's use of long input sequence (4K) with a quadratic increase in complexity while UniChart's vision encoder relies on sliding windows with a local attention mechanism that scales linearly with the input image size. Moreover, UniChart boasts smaller parameter count (201M) compared to MatCha (282M), further contributing to its effi- ciency. As a result, UniChart is highly suitable for real-world applications that prioritize fast inference speeds. More details are provided in Appendix A.7.
5.5 Error Analysis and Challenges
We conducted a manual analysis of our model's outputs to identify key challenges faced by existing models.
Table 3 shows the result of human evaluation on chart summarization based on informativeness cri- teria. We notice that annotators preferred ZeroShot version of our model which generates summaries that are more similar to those generated by GPT, rather than gold summaries. The finetuned ver- sion of UniChart was also rated higher compared to SoTA MatCha (Liu et al., 2022b). The finetumed UniChart model also produces fewer factual er- compared to Matcha and the ZeroShot version (Appendix A.5 and Table 7). We observe that the ratings provided by ChatGPT are roughly consis- tent with the human annotators' scores in terms of informativeness criteria. Moreover, we conducted a statistical test (p-value) for ratings from humans
Densely populated charts: Our model struggles with extracting insights from chart images that con- tain numerous data elements densely packed in limited area. This is evident in Figure Fig. 9 (Q3) where our model generates a hallucinated summary due to the complexity of the chart. Increasing model parameters and input image resolution could potentially improve performance in these cases. Numerical reasoning: Despite efforts to incor- porate mathematical skills, our model still encoun- ters difficulties with complex arithmetic caleula- tions (Q2 in Fig. 9). Addressing this challenge involves decoupling arithmetic calculations and rea- soning steps by employing external program execu- tors that perform the calculations using the equa-

---

tions generated by our model (Gao et al., 2022). Factual correctness in generated summaries: Factual correctness still poses a challenge for au- toregressive language models (Lin et al., 2022; Ope- nAI, 2022; Zhao et al., 2023). Although our fine- tuned UniChart model produced fewer factual er- rors compared to MatCha (see Table 7), it still gen- erates some incorrect statements (see Q4 in Fig. 9). This issue can be attributed factual errors in the pretraining captions generated by ChatGPT.
not investigate the effect hyperparameter tuning might have on the performance on the different downstream tasks. Also, although we have noticed the convergence of UniChart at the end of the sec- ond stage pretraining, we can not confirm whether further pretraining may improve the performance of our model.
Ethics Statement
During the dataset collection process, we made comply with the terms and conditions of the different websites we used crawl our data. Statista provide a permissive license to use their publiely available data for scientific purposes. Pew Research Centre also provide a permissive license to use their data with the condition that we attribute it to the Centre. OECD* allows the users to down- load and publish their data as long as they give ap- propriate credit to the OECD website. For OWID all their data are provided under the Creative Com- mons BY license which gives the permission for downloading and publication. Web Data Com- mons" 10 which we used in the data augmentation process, allows the usage of their data under the conditions of the Apache License Software which gives the right to download and publish. Finally, all the remaining datasets (PlotQA, Beagle, ChartInfo. ExceIChar(40KK, LineCap, and Neural Captions) publicly available datasets which were released in earlier scientific publications.
6
Conclusion
We present UniChart, a general purpose pretrained model designed for a broad range of chart-related tasks. Our model incorporates chart-specific pre- training tasks and is trained on a large and diverse collection of charts and corresponding summaries collected opportunistically using LLMs. We con- ducted both human and ChatGPT evaluations to show the superiority of our method. While our model sets the state-of-the-ar record on four differ- ent downstream tasks and showed improved time and memory efficiency, the evaluation also reveals opportunities for improvement. We believe that our model and pretraining data will be valuable resources for future research and encourage further exploration in this relatively new area.
Limitations
While UniChart exhibits state-of-the-art perfor- mance several benchmarks, it suffers from sev- eral limitations. Despite the remarkable abilities on the ChartQA dataset, the model still struggles to answer questions that involve compositional mathe- matical operations. Moreover, we have noticed that the model may hallucinate and produce factually incorrect statements on the text generation tasks such Chart-to-Tes, and OpenCQA.
Due to the generative nature of our models, they may be abused for misinforming the public by gen- erating factually incorrect responses. Moreover, we can not guarantee that our models may not pro- duce texts that may contain hate speech harmful content.
Acknowledgement
Despite the generalizability of our model on seen chart image styles (WebCharts), there's still a noticeable drop in performance compared to the performance on the tasks on which the model is finetuned (e.g., ChartQA). Hence, there's s still a need for better generalizable chart models for the diverse charts on the Web. One direction is to en- large our pretraining datasets by crawling millions of chart images from the Web. Since most charts the Web do not provide high-quality captions or the underlying data table, self-supervised pretraining objectives are needed to benefit from these charts.
The authors would like to thank the anonymous viewers for their helpful comments. This research was supported by the Natural Sciences Engi- neering Research Council (NSERC) of Canada and Canada Foundation for Innovation (CFI).
www.ssaaissta.oo/gttnnngg hhttps:/wwww.wee statista. www.gettinggggeooooooooom... wtwp/ggting-stared/uubihishiiiii started/publishing cratista-content-trneeee of-o tuunmm-----------d--iccciiiiioi publication-right
conditions/ http://wwww.pooeesrrar///////ooe
https//wwww..cdddrreeeeannnnnntt
reprodtuce-your-dtta 10 hhtpps//wehhtttcommmnn..oy
Due to the limited computing resources, we did

---

References
human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Voiume I: Long Papers), ), pages 7282-7296, Online. Association for Computational Linguisties.
Youngmin Back, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019. Character region awareness for text detection. CoRR, abs/1904.01941.
Leilani Baule, Peitong Duan, Zachery Miranda, Dana Mukusheva, Remco Chang, and Michael Stonebraker. 2018. Beagle: Automated extraction and interpreta- tion of visualizations from the web. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1-8.
Dazhen Deng, Yihong Wu, Xinhuan Shu, Jiang Wu, Mengye Xu, Siwei Fu, Weiwei Cui, and Yingeai Wu. 2020. Visimages: corpus of visualizations in the images of visualization publications. arXiv preprint arXiv:2007.0455
Darwin Bautista and Rowel Atienza. 2022. Scene text recognition with permuted autoregressive sequence models. In European Conference Computer Vi- sion, pages 178-196, Cham. Springer Nature Switzer- land.
BOSHENG DING, Chengwei Qin, Linlin Liu, Yew Ken Chia, Lidong Bing, Boyang Li, and Shafiq Joty. 2023. Is gpt-3 a good data annotator? In Proceedings of the s Annual Meeting of the Association for Computa- tional Linguistics, ACL/23, Toronto, Canada. ACL.
Anja Belz and Ehud Reiter. 2006. Comparing auto-
Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations
matic and human evaluation of NLG systems. In
11th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 313- 320, Trento, Italy, Association for Computational Linguisties.
Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3: Data-driven documents. IEEE Transac- tions on Visualization & Computer Graphics (Proc. Info
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. A survey of vision-language pre-trained mod- els. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5436-5443, International Joint Conferences on Artificial Intelligence Organization. Survey Track.
ChartInfo. 2022. Competition on harvesting raw tables from infographics.
Jian Chen, Meng Ling, Rui Li, Petra Isenberg, Tobias Isenberg. Michael Sedlmair, Torsten Moller, Robert Laramee, Han Wei Shen, Katharina Wunsche, et al. 2021. Vis30k: A collection of figures and tables from jeee visualization conference publications. IEEE Transactions on Visualization and Computer Graph- ies.
Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. 2023. Gpiscore: Evaluate as you desire.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. arXiv preprint arxiv:2211.J0433
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng. and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021. Unifying sssion-and-langee tion. In ICML.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3.
tasks via text genera-
J. Choi, Sanghun Jung, Deok Gun Park, Choo, and N. Elmqvist. 2019. Visualizing for the non-visual: Enabling the visually impaired Computer Graphics Forum, 38.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
visualization.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finent finetuned language models. arXiv preprint arXiv:2210. 11416.
Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019. Image captioning: Transforming objects into words. Advances Neural Information Processing Systems, 32.
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah Smith. 2021. All that's 'human' is not gold: Evaluating
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via

---

pre-training. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4333, Online. Association for Computa- tional Linguisties.
Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- scok Hwang, Sangdoo Yun, Dongyoon Han, and Scunghyun Park. 2022. Ocr-free document under- standing transformer. In European Conference on Compuder Vision, pages 498-517, Springer.
Enamul Hoque, Parsa Kavehzadeh, and Ahmed Masry, 2022. Chart question answering: State of the art and future directions. Journal of Computer Graphics Forum (Proc. EuroVis), pages 555-572.
Wonjae Kim, Bokyung Son, and Iidoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 5583-5594 PMLR.
Xiaowei Hu, Xi Yin, Kevin Lin, Lei Zhang. Jianfeng Gao, Lijuan Wang, and Zicheng Liu. 2021. Vivo: Visual vocabulary me-training for novel object cap- tioning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1575-1583.
Klaus Krippendorff. 2011. Computing krippendorff's alpha- rilhabreli.bility.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutimv3; Pre-training for doc- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia, MM '22, page 4083-4091, 1, New York, NY, USA. Association for Computing Machin- ery.
Wojeiech Kryscinski, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE pages 540-551, Hong Kong, China. Association for Computational Linguis- tics.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang. Bei Liu, Dongmei Fu, and Jianlong Fu. 2021. Seeing out of the box: End-to-end pre-training for vision- language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22976-1298
Kenton Lee, Mandar Joshi, Iulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandel- wal, Peter Shaw, Ming-Was Heig-Weii Chang, and Kristina Toutanova. 2022. Pix2struct: Screenshot parsing as pretraining for visual language understanding. arXiv preprimt 2rx10:2210.033347
Zhicheng Huang. Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint 20X00:2004.00849.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: demoising for natural language generation, translation, and com- prehension. CoRR, abs/1910.13400
pre-training
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018 Dvqa: Understanding data visualiza- tions via question answering. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 5648-5656.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal en- coder for vision and language by cross-modal pre- training. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 1 11336- 11344.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018 Figureqa: An annotated figure dataset for visual reasoning. 6th International Conference on Learning Reprexentations, ICLR 2018- Workshop Track Proceedings, pages 1-20.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision-languag understanding and generation. In JCML.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS.
Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan, Enamul Hoque, and Shafiq Joty. 2022. Opencqa: Open-ended question answering with charts. In Proceedings of EMNLP (to appear).
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min- joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth dozen images. In Euro- pean conference on computer vision, pages 235-251. Springer.
I Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and lan- guage.

---

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. 2020b. What does BERT with vision look at? In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 5265-5275, Online. Association for Computational Linguistics.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as a factual inconsistency evaluator for text summarization.
Jock Mackinlay, Pat Hanrahan, and Chris Stolte. 2007. Show me: Automatic presentation for visual analysis. IEEE transactions on visualization and computer graphics, 13(6): I 137-1144.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 321 14-3222, Dublin, Ireland. Association for Computational Linguistics.
Anita Mahinpei, Zona Kostic, and Chris Tanner. 2022. Linecap: Line charts for data visualization caption- ing models. In 2022 IEEE Visualization and Visual Analytis ( (VIS), pages 35-39. IEEE
Fangyu Liu, Julian Martin Eisenschlos. Francesco Pic- cinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2022a. Deplot: One-shot visual lan- guage reasoning by plot-to-table translation. arXiv preprint arXiv:221.
Ahmed Masry, Do Long. Jia Qing Tan, Shafig Joty, and Enamul Hoque. 2022. ChartQA: benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263- 2279, Dublin, Ireland. Association for Computational Linguisties.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang. Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022b. Matcha: Enhancing visual language pretrain- ing with math reasoning and chart derendering, arXiv preprint arXiv:2212.09662
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthe- nis Karatzas, Ernest Valveny, and CV Jawahar. 2022. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vi- sion, pages 1697-1706.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra. and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV). Yixin Liu, Alex Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan, Ruilin Han, Simeng Han, Shafiq Joty, Chien-Sheng Jason Wu, Caiming Xiong, and Dragomir Radev. 2023b. Revisiting the gold stan- dard: Grounding summarization evaluation with ro- bust human evaluation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics, ACI,23, Toronto, Canada, ACI..
Yang Liu, Dan Iter. Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: Nlg evaluation using gpt-4 with better human align- ment.
Tamara Munzner. 2014. Visualization Analysis and Design. CRC Press.
Jason Obeid and Enamul Hoque. 2020. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. In Proceedings of the 13th International Conference Natural Lan- guage Generation, pages 138-147, Dublin, Ireland. Association for Computational Linguistics.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 10012-10022.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini A Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tie representations for aision-and-language tasks. Ad- vances in neural information processing systems, 32.
Alan Lundgard and Arvind Satyanarayan. 2021. Ac- cessible visualization via natural language descrip- tions: A four-level model of semantic content. IEEE transactions on visualization and computer graphics, 28(1):1073 1083.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Brussels, Belgium. Association for Computa- tional Linguistics.
Junyu Luo, Zekun Li. Jinpeng Wang, and Chin- Yew Lin. 2021. Chartocr: Data extraction from charts images via deep hybrid framework. 2021 IEEE Winter Con- ference on Applications of Computer Vision (WACV), pages 1916-1924.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver?

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 -866.. PMIR.
Chase Stokes, Vidya Sethur, Bridget Cogley, Arvind Satyanarayan, and Marti A Hearst. 2022. Striking balance: Reader takeaways and preferences when integrating text and charts. IEEE Transactions on Visualization and Computer Graphics, 29( 12:33 1243.
Weijie Su, Xizhou Zhu, Yue Cao. Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. Vl-bert: Pre-training of generic disual-linguise representations. In Inter- national Conference on Learning Representations.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research,
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross- cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
166140):1-67.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ ques- tions for machine comprehension of text. CoRR, abs/1606.05250.
Liling Tan, Jon Dehdari, and Josef van Genabith. 2015. An awkward disparity between BLEU/ RIBES scores and human judgements in machine translation. In Proceedings of the 2nd Workshop on Asian Transla- tion ( (WAT2015), pages 74 81. Kyoto, Japan. Work- shop on Asian Translation.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cmn: Towards real-time object detection with region proposal networks. In Ad- vances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Zineng Tang, Ziyi Yang, Guoxin Wang. Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. 2022. Unifying vision, text, and layout for universal document processing. arXiv preprint arXh::2212.0262.
Arvind Satyanarayan, Dominík Morizz, Kanit Wong- suphasawat, and Jeffrey Heer. 2016. Vega-lite: A grammar of interactive graphics. IEEE transactions on visualization and computer graphics, 23(1):341-
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3156-31644
350.
Kamtharaj Shankar, Leong Rixie Tiffany Ko, Lin Xi- ang. Masry Ahmed, Thakkar Megh, Hoque Enamul, and Joty Shafiq. 2022. Chart-to-tex:: A large-scale benchmark for chart summarization. In In Proceed- ings of the Annual Meeting of the Association for Computational Linguistics (ACL). 2022.
Jiapeng Wang, lianwen Jin, and Kai Ding. 2022. Lilt: A simple yet effective ihnguage-independen layout transformer for structured document understanding. arXiv preprint arXiv:2202.
Danqing Shi, Xinyue Xu, Fuling Sun, Yang Shi, and Nan Cao. 2020. Calliope: Automatic visual data story generation from spreadsheet. IEEE Trans- actions on Visualization and Computer Graphics, 45324455..463.
Zirui Wang, Jiahui Yu, Adams Wei Yu. Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision. arXiv preprint htt.:2008.10044
Peng Shi, Patrick Ng, Feng Nan, Henghui Zhu, Jun Wang, Jiarong Jiang, Alexander Hanbo Li. Rishay Chakravarti, Donald Weidner, Bing Xiang, and Zhiguo Wang. 2022. inveration-focusee table-based intermediate pre- are-training for free-form question an- swering. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):1 1311)-111112113333 1320.
WDC. 2022. Web data commons, extracting structured data from the common crawl.
Qiaolin Xia, Haoyang Huang, Nan Duan, Dongdong Zhang, Lei Ji, Zhifang Sui, Edward Cui, Taroon Bharti, and Ming Zhou. 2021. Xgpt: Cross-modal generative pre-training for image captioning. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 786-797. Springer.
Aaron Smith, Christian Hardmeier, and Joerg Tiede- mann. 2016. Climbing mont BLEU: The strange world of reachable high-BLEU translations. In Pro- ceedings of the 19th Annual Conference of the Eu- ropean Association for Machine Translation, pages 269-281.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinci Florencio. Cha Zhang, Wanxiang Che, et al. 2020a. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv
Andrea Spreafico and Giuseppe Carenini. 2020. Neural data-driven captioning of time-series line charts. In Proceedings of the International Conference on Ad- vanced Visual Interfaces. AVI '20 New York, NY, USA. Association for Computing Machinery.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020b. Layoutlm: Pre-training of text and layout for document image understanding.

---

Proceedings of ACM SIGKDD International Con- ference on Knowledge Discovery and Data Mining, pages 1192-1200.
A
Appendices
A.1
Data Augmentation
Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023. Verify-and-edit: knowledge-enhanced chain-of-thought framework. arXiv preprint 2305.3830...326..
During the data augmentation process, we mainly utilized two of the most popular visualization li- braries: D3 (Bostock al., 2011) and Vegalite (Satyanarayan et al., 2016). Moreover, we have introduced range of visual variations in terms of color scheme, elements dimensions, shapes, back- ground, ,etc (see Fig. 2). This makes our generated chart images closely resemble the real-world charts found on the Web.
A.2
Data Augmentation by Knowledge Distillation
We select the small dataset (3700 charts) from PlotQA and the augmented charts from WDC, since these datasets are accompanied by the un- derlying data tables which serve as suitable chart representation for LLMs. Also, they cover wide range of topics which contributes to the diversity in the generated summaries ($3.1). Fig. 3 shows process for generating summaries for the charts that have underlying data tables using InstructGPT model (Ouyang et al., 2022). The input mainly consists one demonstration (table-caption pair) followed by the desired chart data table. The output is the generated summary. Using this mechanism, we generated small dataset of 3,700 samples. We then finetuned Flan-TS XL (Chung al., 2022) this dataset. To our knowledge, Flan-T5 was the SoTA open-sourced instruction-tuncl model during the development our dataset. After finetuning on our task, we (qualitatively) observed similar per- formance as soett-davicii-O-- At the final step, we used the finetuned Flan-T5 model to gen- crate summaries for all the charts that do not have an associated summary (e.g.. PlotQA, augmented charts, OWID and OECD charts). In this process, added around 470K summaries for charts in our pretraining corpus. Fig. 5 shows some examples generated by the finetuned Flan-T5.
To benefit more from the capability of GPT mod- els generating high-quality summaries, we fur- ther prompt ChatGPT (gpt-3. 1empt-3.5-tee 5....ttus 5-turbb) (Ope- nAI, 2022) to generate summaries for the charts from Statista and Pew Research and put these in our pretraining corpus instead of the original sum- maries in the Chart-to-Text benchmark (Shankar et al., 2022). In most cases, we found the sum- maries from ChatGPT to be more elaborate and better writing style. For the Pew Research Centre

---

Chart Type
Linguistic Stutistics
Datasets
Bar
Line
Pic
#Charts
#Vocab
Avg. Character
Avg. Token
Two-Col
Multi-Col
Two-Col
Multi-Col
Statista
71.9%
15.3%
8.3%
2%
2%
19,143
24,392
111.37
21.88
OWID
$1.9%
0.0%
9%
38. 98.91%
0.0%
60,624
3,721
85.89
16.96
OFCD
49.1%
0.0%
3.1%
47.7%
0.0%
64,380
1,606
65.47
14.67
PlotQA
11.2%
55.6%
6.7%
26.2%
0.0%
157,070
2,230
155.32
33.21
Beugle
29.8%
27.3%
24.7%
17.9%
0.0%
3,972
11,361
78.76
20.55
ChartInfo
31.7%
51.0%
8.6%
8.6%
0.0%
1,796
13,329
120.75
26.11
Data Augmentation
13.3%
49.3%
11.7%
11.1%
14.3%
189,836
117,244
85.62
21.16
ExcelChart0OK
11.5%
32.7%
12.0%
22.3%
27.7%
106,897
515,922
138.68
27.72
PowRescurch
11.4%
55.5%
4.4%
21.9%
6.5%
5,295
38,165
477.33
98.08
LineCap 0.0%
0.0%
15.9%
84.0%
0.0%
2,821
16,570
102.11
24.62
Neural Caption 0.0%
0.0%
100%
0.0%
0.0%
100
389
117.56
27.43
Total
21.95%
36.56%
9.23%
23.71 %
9.39%
611
611,934
888,522
114.70
25.01
Table 4: Chart type distribution and linguistic statisties of the chart pertaining corpus. The charts in the last group (magenta) do not come with an underlying data table. The charts generated by the data augmentation process are shown in blue.
Datasets
#Vocab
Avg. Char
Arg. Token
Avg. Sentence
A.4
Ablation study
Statista
72,725
450.28
106.68
4,46
To further assess the impact of our different pre- training objectives on our model's performance, we conducted ablation studies. Due to computa- tional limitations, we focused on pretraining the model only the lower image size (512x512) and compared it against the corresponding main model (5 1512x112). From Table 8, we observe that remov- ing the Chart Summarization or Open-ended Ques- tion Answering objectives led to slight decrease in performance on ChartQA. We attributed this to the abundance of numerical reasoning examples in pretraining. However, removing the Numerical Reasoning pretaining task led to substantial de- crease in performance ChartQA. highlighting the importance of this task in imbuing numerical abilities into our model. Pretraining the model with- out the Data Table Generation objective resulted in a relatively weak performance in the ChartQA benchmark, underscoring the importance of under- standing underlying data tables of charts in answer- ing reasoning questions.
OWID
58,212
463.48
105.00 99
4,47
OECD
24.752
414.97
95.08
4.78
PloiQA
112,394
666.09
149.84
5,20
Data
162,239
468
113.46
4.49
PowResearch
LineCap
Caption
Table 5: Statistics about the captions of the datasets used in LM pretraining.
charts, the underlying data tables are not provided. However, we have observed that the underlying data values written the visual elements in most of these charts. Hence, decided use an OCR tool to extract the layout-prs texts from the chart images, and then feed into Chat- GPT to generate the summarics as shown in Fig. We realized that ChatGPT is capable of understand- ing a chart from the OCR data.
A.3 Dataset Analysis
The linguistic characteristies of the textual ele- ments vary across different datasets, with charts from PlotQA and PewResearch often having longer text elements (c.g., axis labels, legends, titles), while augmented data and Beagle datasets contain shorter text (Table right). In Table we further provide linguistic statistics for the summaries of the datasets used in the summary generation task at pretraining.
A.5 Human and ChatGPT Evaluation
As discussed in section $5.3, we evaluate the fol- lowing three criteria in the human evaluation study: (1) Informativeness which measures how much in- formation from the chart the summary covers. Ide- ally, an informative summary should contain high- level insights from the chart, such as important patterns, trends, and outliers in data; (2) Factual

---

Experiment
# Epochs/Steps
Learning Rate
Batch Size
GPUs
Saving Mechanism
Pretraining
Firstsstage/ablations
300K steps
le-4
160
4xA 100 82GB
each 50K steps
Second-stage
100K steps
lc-4
80
4xA 82GB
each 50K steps
Finetuning (main 960x960 model)
ChartQA
20 epochs
5e-5
24
each epoch 4xV100 32GB
Chart-to-text Pew
200 epochs
5c-5
48
4xA 1x0100 40GB
each 5 epochs
Chart-to-text Statista
100 epochs
5c-5
48
4xA 100 40GB
cach 5 epochs
OpenCQA
200 epochs
5c-5
24
Vx 100 32GB
each 5 epochs
Table Training details for pretraining and finetuning experiments.
ZeroShot
MatCha
ChartQA
Factually
13.15%
21.97%
3,59%
Model
aug.
human
avg.
Elementallencoed
19.42%
26.06%
29.61%
21.07%
Stttiticaiiillainnn sents
57.41 %
33.42%
34,07%
34.70%
UniChart (512x512)
64.72
Penequuallongnnive seres
6.98%
1.41%
0,315
5.39%
No Chart Summatization
84.96
42.72
63.84
Sontestulli<emaiinqeeeeaa sents
20.56%
No Open-ended Question Answering
85.52
64.24
No Numerical & Visual Reasoning
84.08
35.44
59.76
Table 7: Human evaluation on summaries for 150 ran- dom samples from Chat2text Statista test split.
No Data Table Generation
83.84
42.24
63.04
Table 8: UniChart ablations ChartQA benchmark.
Correctness which considers how accurate the sum- mary is. A factually correct summary should only contain information (e.g. numbers, events, enti- ties) that is true and/or supported by the chart; (3) Semantic Levels defined by (Lundgard and Satya- narayan, 2021) which categorize the content of summaries across four levels: visual encoding (e.g., axis, legends, color), statistiicl//elaaiinaal (c.g., min, max, avg.). perceptuull/cognitive (e.g., describing overall trends, complex patterns, or outliers), and context/domain specific information. Our process for evaluating the informativeness is explained in $5.3. For factual correctness and semantic level measures, the annotator goes through each sen- tence of the summary to determine whether the sentence contains any factual error and what levels semantic content are present for that sentence. Table 7 shows the results our human evaluation study on factual correctness, and different semantic levels.
look challenging for SoTA models. Q3 an exam- ple of an overpopulated chart with so many data elements which confuses the model to generate in- sightful summary. Finally, Q4 shows factual error in generated summary from finetuned UniChart.
A.7 Time and Memory Efficiency
To compare the time efficiency, measure the erage inference time of the models on three bench- marks: ChartQA, Chart-to- Text (Pew), and Chart- to-Text (Statista) using random samples from each benchmark. The experiments were conducted on Google's Colab platform with cpu type. Over- all, UniChart shows much faster inference times compared to MatCha shown in Fig. 4.
A.8 Templates for Numerical and Visual Reasoning Question Generation
Table is the list of the templates used gen- erate numerical and visual reasoning questions.
Fig. 7 shows overview of the paradigm in our ChaiGPT-driven evaluation study. Fig. 8 de- picts the interface we used in our human evaluation study.
A.6 Error Analysis
Fig. 9 shows the models performance on challeng- ing samples. Q1 and Q2 examples are two visual numerical reasoning questions about charts which

---

Golasse.og Kodama Kunio Detalled Statistics
Datlagg Collgge at Hartung Golegel Box doom (821/0015 (okodbal
(a)
(b)
UMoss Dortmouth Westieed State Box Soore 2011:0013-- OStootbal
Dagarts
Roond
Cole Weralaoo
(C)
L'tnae Var Fuebus 500 Coaerra Came feadt snc n0e
6/M0
(c)
Authenti Collection Tech Fleece
How to call Togo from Maloni| NabalCom
2A
(h) Figure 2: Visually diverse charts generated by D3 and Vegalite for WDC corpus (Fig. 2a, Fig. 2b. Fig. 2c, Fig. 2d, Fig. 2g and Fig. 2h from D3-WDC, Fig. 2e and Fig. 2f from Vegalite-WDC. Visual factors like color scheme, width bars, and existence grids and axis labels are different among the samples.
(g)

---

Analyze following bar chart in one paragraph and round the numbers. numbers represent Rating of statistical capacity (0-100).
pepiate born doceery
Chart: Years Botswana Samoa 2010 55.0 2011 30 I 50.0 2012 40 60.0 2013 301 60.0 Summary: The chart shows the rating statistical capacity for Botswana and Samoa from 2010 2013.
In 2010, both countries
statistical capacity rating
30, but 2012, Samoa's rating improved while Botswana's rating only increased to 40. In 2013, both countries had rating 60 for Samoa and for Botswana. It can be concluded
Samoa shown better improvement in its statistical capacity the four-year period.
compared to Botswana
Analyze following bar chart in one paragraph and round the numbers. chart title Percentage of population having access to electricity and the numbers represent Access resorces of population).
InstructGPT
shows percentage population having access to electricity in Kiribati, St. Lucia, and Venezuela from 1990 2012. In 1990, 48.86%, 81.14%, and 98% population
electricity Kiribali, Lucia, and Venezuela respectively, while in 2012, the percentages Increased to 59.35%, 90.88%, and 100% respectively.
Chart: Years Kiribati St. Lucia I Venezuela 1990.0 48.86 $1.14 98.0 2000.01 52.53 85.41 89.0 2010.0 87.87 100.0 2012.0 59.33 90.88
countries experienced significant increase access to electricity over the 22-year period, with Venezuela having the highest rate 100% in 2012.
100.0
Summary:
Figure 3: An example of the performance of InstructGPT in generation summaries for data tables. On the left side, the red text is full example of demonstration and its summary followed by the demonstration for the target chart. The paragraph in green shows the summary generated by the model.
1200
UnKhart
1000
Tine (2cOnd)
400
OariOA
Chart Text (Pow)
Chart Text (Statala)
Average
Figure 4: Average inference time for 10 random samples from three major benchmarks in chart understanding domain for UniChart and MatCha models

---

Permyyerraal Scise Pambiree.o
The bar chart companes the performance of two high school girls frams, Firetooed and Trisity,
The comparss ountier potents ensironment technologics between Canada and Mesicn. Caaada 63 pateets covironmeeat technologies whik Mesico his 26.3.
The har chart shows the reveltss of the Vital Leeds poll in chronological order foe uven teame: Hudderdiebl, Hall Ciry, Iporich, Lecds United Miädichrraph, MK Dons, and Presion. All icams had score of tull is the poll, indicating that they had not bren vatud on yer. This suggests that the from was not yet cansidored to be clear lavorite in the poll, and that the results of the poll may be skewed by the onteome of the nest few gomes.
Both with Fleetwood scoring and Trinity scaring 6. Buth seams had 0.0 in shees gool saves,
had
difference
indicates double number
that Mledes more patcets en cavireument technologics compared Canada, This suggesis Mexice more sctive feld reviromment technoloogis.
and corser kirks.
7.0 kirks, while Trisiry had a0 grat, 38 saves, points, and 3. corner kicks. Overail, the chart showss both trams had similar perfermances the with Floetroodd seoring 1. and Triniry scaring (Jl.
Howeverr Fleetword had 4 shots points, and
line chart disphass number domesstc talicofls and abread of carriers registerd Isrsel
The line chart shows the amount of agricultural raw mattrials, fond, and foel exports of Nerway from 1998 as 1995, Agricultural row materials exports decrensed from 2.18% in 1990 to 1.32% In 1993, Faod exports incressed from 6.94% in 1990 to 8.5% in 1964, Farl exports incressed from 47.78% in 1990 to 51.26% in 1993, Overall, the trend shows decrease in the amount of agricaltural now matorials, food, and fnd rxports of Norway from 1998 to 1995.
intividual players performance
line
1998
Widener
50.100 1995, flight 1999, decreased 56.427 decrensed further 2001,
categoeics
carries
gilted average yards per carry lengest
decreased Bactnating trend the Israel flight the sis-year peried. with decreasing 1998 2003.
Therefore,
carries Lewis Visccal
yards
(64,09)
per imprevdvee sverage yards Ruiz
average 5.42
Mike touchdowas
(1.43)
yards Overall, staaaed that both varying performance pame.
ssuchdowa.
Yander uceernne vle acteonbee
11c
bar chart shows number uns-year-olds did not the aqpiast diphiheeris. pertosiis, and tetanss (DTP3) 2080 the countries: Lebanos,
The chart shows. the roults of the Hungarian Grand Pris mator race is 2020. culegerize by the wisaer of the reet. The winner of the accounted for 17% of the total roslts, fallowed by Danill Ksyat at 33%, and Daniel Risciardo at 50%. This chart indicates that the race was closch contested, with all three drivers finising withia chise starga of each other.
The lise chart displays the number nos-fatal from terrorist attarks periud from 1970 1981. The v-rtis shows the number son-fatal injurirs, while the shows the number of fatalities. The sumber of non-fatal injuries intrensd from 4,23 1970 211,9 1980, before docreasing 42.8 in 1981. Overall, the chart shoms increasing tread is the number nnn-fatal injuries from turrerist setacks during the nine-yaar period, with a slight diceaase in 1978.
nine-year
Iran, Comeros, and Sei highest number
was Sehastian Vistel, who
Laska. Lebanos not receive vacsine year-alds
12,09 and 11,401. Comoros 5,667 and Lanka had 3,291. data vuggests variation the number one-year-olds aod reesive the vaccine aprinst diphtheria, pertussis. and sotanus across these countrios
while Eritres
there
signifieant
did
Figure 5: Examples of summaries generated by PIan-T5A XL model after fine-tuning.

---

From
Gonnestedd
The chart shows The perointage U.S. teans, 17, feel overthaimee Farres presuuee to post content for ther. wone about thiir His dins soxid media. Grrs more Buily than boys to feel ovenehatred by drama. escluded friends, and warse about thair Mi. Around 45% girt
them overwielmmed by axcluded by friundda about their life denglefurit buliite
excluded
worls sepr utivay giread
Chasgit
Bos
doamss foeding nommllbrad while only 32% of boys fisl the samre Similally, 57% of girts report fealing enciuded by their Mands comparee to 24% of boys. Regarding pressuro to post content for like, 22% of boys and 27% of girs feel this pressure Finally, 28% of girls and 18% boys report fealing warse about thair own life due to social media. Overall, the chart sugpests that social modia has greazer negative impect on the mental health on teen gits than teen boys.
une the te en
24+
Prosors sor camse deccell
The
an 2017
Figure 6: An example of the layout-preserved OCR-extracted text for PewReseacch chart image where the underlying data table is not available. The extracted text is then given to ChatGPT to generate a summary. ChatGPT can still extract and comprehend important information and insights from the payout-prs text of the chart image.
You will one summary wiltan a chari Your lask rale the summary on one metrc and proude appropenta hechars.
CHildPT
1. Ruad the summery cauiddly and ensure undensarding of the infommtion presented in the char: data unle.
2. Eiomtte the intomat-onnes the summar by sounnning how much infommation fon tie data
Exotuation Crtarto: Informatoonnes (1-5)
mesearee has
covered.
Informatvv summery
informalieness
Informaiion
informatve
the char the summary covers, lowally an informatow aummary should conten tighioeel naighti from the cheri, such as important pattrns, mreds and outlers in dat3.
and contsining key insight.
4. Doneider whather the summary affectoodyy comsaaa the key information from the chart data tabre and factually accurane spectfic
areas impoxoment, neceesary
Evaluation staps:
Chart Sumpll
Feceback Generated By ChalGPT
Eostuatio::
Feedback The summary only cosers one dats poit from the lable, which thu direct comfiction CDe. II does not provide any
spenting, invesinent. The summary informaive enough It does conta any koy insigts from the chart dara table The summary needs to include more information from sacl
sponding.
more iriormatioe.
Summary Sample
This statistic aconomic contribution and tourism industry to United Kingdom UK in 2017, by secior. The dires contribution to GDP is approximately 93.5 billoo U.S. dollars.
The pipeline designed for the ChatGPT Evaluation Experiment. First, we feed task description followed by our desired criteria into ChaiGPT order to get detailed grading instructions. Then, the chart (underlying data table representation) and sample summary are appended to the prompt which fod again into CharGPT to receive the feedback.
Figure 7:

---

Model Evaluation
Instructions
Progress
Save Progress
Chart
Chart
Answers
Title
#1
Question:
This statistic shows the proportion of the population with access loat busic and limited drinkking water worldwide 2017. region
tion population
worldwide percent./11
This statistic represents the proportion the population least basic and limited drinking services Occania access
region/Di services(1)
Answer
chart shows proportion population with leant banc and limited drinkingg
gisballyin2111
(1)) percentagm
driiking water services globally are follows:
Australia
North America & Europe: 99% Latin America & the Caribbean: 96% Eastern South easiernnais 98%. Waster Asia Northar Africa 92%, Central Asiu Southern 92%, Sub Sahuran Africae 61%
Oceania: 55%(1)
Answer #4
The statistic shows the proportion the population with access to leant basic and limited drinicing water services selected world regions in 2017 .0) That percent of the population in Northern Africa had
(a)
Evaluation Questions
Answer #1
summary?
How many sentences contain ractual emors?
many seatences contain Memennts/enccoe level Information
How many sentances contain Per toggtaall informatio?
How many sentences contain Contotualdooraln specific level infor matiinn
How Informative the summany?
many terteno erro??
sentences Bleeetailenceee
Information?
How many sentences contain Statistical Bebeiiiiesslllleen Information
0.a
0
O
many
0
Contentuatioraat-
information"
Answer #3
How many sontonces contain factrallnoos?
O
(b)
Figure Human evaluation interface.

---

Many think China's global Influence will decline after coronavirus
outhresi
outfresk
U.S.
20%
43%
706
50
3B6
14
China
17
31
noupe Survey adults condacted
3DC
"Aewricam Glas Higher Rasingg South iloea and Germany Than
2020.
Shorse Rxpeederr
Dealing Connemmar
PEW RESGARDH CENTER
Vose day
2-10
day
Lesoullen
Q1: What is the largest value of red bar?
Q2: What's the average of the values the green bars (round decimaly? Answer: 21.6 UniChart 0.4
Answer: Un/Chart
MatCha: 40
MatCha:0.51
Van-dreeu lusir suerudee dar of CIT trmm varc Nermilee saar mere
30C
alownt Gldlk Eraide Canelos
Media Naaa Taunde Warzeu
UniChart ZeroShot: The line chart shows the year-on-year change in air operations due to the outbreak COVID-19 in Poland from March to November 2020, broken down by occupations. The occupations measured in air operations (excluding Fight operations), in wholes (excluding Fight operations). operations) The occupations Fight operations). in wholes (excluding Fight operations). and in wholes (excluding Fight operations). The occupations operations (excluding Fight operations). in wholes (excluding Fight operations).
112
wholes (excluding Fight measured in air operations (excluding
1380 20000
10300
Q4: Summarize the charl.
UniChart Finetuned: This statistic illustrates annual consumption of fluid cow milk worldwide in 2019, by country. In that year. India's fluld cow milk amounted to 77.7 million metric tons, making the world's second largest consumer of milk.
(excluding Fight operations). cocupations measured in air operations (excluding Fight operations), in wholes (excluding Fight operations), wwwpe//wwwiieieeeeeeeeeeeeeeeiie
Figure 9: Some challenging examples from the ChartQA and Chart-to-text benchmarks.

---

Templati-hrred Questions
1 Firs the tuplich the socond group
lepitet
2) Fins hor from the bettmmlighh the uscond group from the hettombight
3) Sccond bar from the bottnnliggt the fisst group from the bonombigght
Second
rightbotton fist group from letttop
5) Togunsotcchhesttt
6) BonommossRRgleau bar
7) Secord hur from the upplett
Second
rightbotton
toprost
100 Lefinaee bottomenne
11] Rightmus trpmus her
12) Righement bottommee bar
Lettnoo <gelos>
14) Righensez cooLos?
15) Secoud from the left <colos> data
16) Second from the right soalors data
Which
repessemeed scolor>?
color <legenca
19) Which one graster, oxd cxts?
20) Divide the sum of lingestand and lowed walurs by en>
21) When did line clegend lamel> 13 yesk?
diffesenee Sum segmtent above <valuc>
and inniimn of <loqued label>?
24: What the sua of top firee values?
25) What the medinimode of <logend label>?
26) What negrive peak of legend 15be
lagest/omallle Which two s-axis libel clegend Isba sums up lowa?
<legend. 1sbel=?
29) What is the sam of the sccond highes and second lowest valuz of clagend label>?
300 Which e-anis label sccond highes for <legend
sbel>?
Whet
mildle values
lebel>?
Whtch two label <logand label>
diffesence evaluc=?
353 Whet is the overage of clequudd label> too va Label to ss laiml 2>?
343 What the avengg the highest and lowest value of <logend label - 119
35) What sumidiifeeence the matimum <lcgand label elgenn label 2>9
of the
label 13-md
minmuu <logend labol
37) Which c-ssis lshel the maximum/minimum difference berwern di clegend 1abel and minimum of <legend label 253
38) Which s-anis label witressed the smallce value of Clegenc
label>?
19) Which lehell contiine bangeatiemallee valuos
all labak??
Sun up melliass
scics
chart
evengge values abeve usll?
421 Whet the sau of the lugest and snalles difference between cleaind
lalee1
15 and clooend label 2>2
4.3 What the matmmm/mmmmmm difference between <legend label 1n and <Logend
label 2>1
44) What ratio the birest
emallens segment?
karles//nnallest segmenns
46) What dhfferrnce betwern the Infimue and rightmost hans?
47) What the sum of the hass in the sccond group from the left?
48) What the of the him in the fins grop from the right"
What diffesence between the rightnost soulor
lefloot burs?
and leftnsost suolus bex?
311 What the oeergge of coolos o has values?
521 How mare coolorr bass are luggr than <b>>? ?
What
the ban the second
from right?
How mane buss leftnest group have
over
55) Whet does the ccol orz repeesent?
56) What is the modlon wilue of the coollors handinne?
57) What the ovenge the cacloz 13 sums and coulos 2> suat?
What swrnge the diffesenes between Ycoles 1> and cuolos
median
median?
66) What the ratio between the lettmost and rightmest bar is the fust group from the left?
G1) Whatt is the maximma value the scolosy bankllne?
What munimum
barshine?
What
hanoline?
64) What the difference between the matimum values of the two leftmost bet groups!
65) Sum of the find con for 12 and last scolor 25 hee/line points
66) Difference betwern the two lowest coolor> bes
Adi langest snallest
lineftrr valsscand divila
What value of cuolock linefbuss ix axia Laiee>>?
699 Sunu Average of ccolos 15 and coolor 2> valces in en axis labels?
200 San of hightstt points in sueles 15 and Scolox 25 lineshars
Whtch color has highestsmallestt values?
How mony volues equal
linefort
751 Sum two righmoss values of coolor> gaap.
74) Proditct of ww emiilie valuos in the graph
75) Sum of lowest and molian valuza <ool gughbbars
the evenge the rightmost three goints coolaro line?
line
781 How many cooler? data points are dhove evalous?
790 What's the catio of the largsst and the thirdheoond-aag
ccolozo bar?
of
ccolor 1> and ceulor
bar
largest value of <aolox 3> bar! bus?
ccolor
greater
821 is the media of all the ccoloz 1- bars greater than the lagges value of scolor bar?
83) What's i the product of ccolers bons Indio and lipan?
84) la the sum of the ran middll bars pretee then the sum of top and bottom bas?
the axie -o-ocooooo bar? has prester then uotal all keolor bes!
la
czo. ar
87) Take the sum of the two smallest caulas 15 bus and smallest saolos 2> bas, deduct the smaller value from the larger value. what's the realt??
85) What the samievengg of two inallsddagge caulars bas?
What
cooles segment repeesented by ecolors?
scolos
Table 9: Numerical & Visual reasoning templates.


Reviews:
[{"review_id": "dkeuP98Gz0", "reviewer": "Reviewer_bdEC", "summary": "", "questions": "- Can you provide ablation experiments that would allow us to understand the effect of the pretraining corpus vs. the proposed UniChart model architecture? Which contribution is more important for better performance, and by how much?\n- What are the parameter counts of each of the models in Table 2? It would be great if the parameters and pretraining data used could be listed here for ease of comparison.\n- Did you perform de-duplication of the evaluation datasets used in Table 2 against the pretraining corpus? This is important to ensure that none of the charts used in validation leaked into the pretraining dataset.\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to concerns in the Reasons To Reject and the Reviewer Questions\n1.  MatCha/Pix2Struct ablation studies (Reason 1 and Question 1) \n\n    *  **Response:** We chose Donut architecture as the backbone model since it was much more efficient in terms of computational complexity during both training and inference time (as described in lines 654-668), especially for the high-resolution chart images. Our pretraining computational resources were limited (4xA100 80GB) compared to the computational resources needed for Pix2Struct/MatCha. It is also worth noting that the original MatCha model from Google was pretrained using 64 GCP-TPUv3 which is infeasible in an academic setup. Moreover, based on the results reported in the Pix2struct paper, fine-tuned Donut exhibits lower performance compared to Pix2struct on ChartQA benchmark, which suggests the effect of our pretraining corpus  in improving the results rather than the pretrained backbone model.\n\n2. ChatGPT evaluation (Reason 2)\n\n    * **Response:** We opted for the rating strategy (Table 3), mainly because it was less expensive and had less latency compared to pairwise comparisons (more API calls). However, to address the reviewer’s comment, we did an additional analysis using pairwise comparisons executed by ChatGPT on the same 150 random samples used in Table 3 (see below). The findings from this analysis align with our initial conclusions regarding the performance of various models. The numbers below represent the proportion of instances where ChatGPT favored one model over another within the pairwise comparison framework.\n\n\n        * ZeroShot vs Finetuned: (ZeroShot wins: 83.8%, Finetuned  wins: 7.7%, Tie: 7.7%)\n\n        * ZeroShot vs MatCha: (ZeroShot wins: 89.5%, MatCha wins: 6.9%, 3.4% Tie)\n\n        * Finetuned vs MatCha: (Finetuned wins: 48.5%, MatCha wins: 35%, 16.4% Tie)\n\n        * Finetuned vs Gold: (Finetuned wins: 28.3%, Gold wins: 53.9%, 17.7 Tie)\n\n        * Gold vs MatCha: (Gold wins: 62.3%, MatCha wins: 23.9%, 15.7% Tie)\n\n        * Gold vs ZeroShot: (Gold wins: 27.4%, ZeroShot wins: 62.6%, 9.8% Tie)\n\n    * Based on the reviewer's suggestion, we will also de-emphasize ChatGPT evaluation and move it into the appendix. \n\n3. Parameters Count for the models in Table 2 in the paper (Question 2)\n\n   * **Response:** We will elaborate the number of parameters and pretraining data for each model in Table 2 in the paper.\n4. De-duplication of the evaluation datasets (Question 3)\n    * **Response:**  Yes, we excluded validation and test sets from all the sources that were shared among downstream tasks like Statista and Pew. We will clarify this further in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper prepares and releases a new large pretraining corpus for chart data. This data is derived from real world sources, as well as generated from real-world data tables. It will be a valuable dataset for the community.\n- The paper proposes a new model and pretrains it on the chart dataset, showing that it outperforms prior work on several tasks. Evaluations are comprehensive, spanning automated metrics (BLEU, RA), human evaluations, and LLM evaluations.\n- The error analysis in the paper highlights important drawbacks of the existing model, and suggests ways to improve them in future work.\n- The evaluation and analysis provided in the main paper and the appendix is very comprehensive.\n", "reasons_to_reject": "- There is an important ablation missing in this paper to compare architecture against pretraining data. The paper proposes both a new dataset, as well as a new model architecture. It’s unclear whether the improved results are due to a better model architecture, or cleaner pretraining data (I suspect the latter). How do prior models do when they are pretrained on the same corpus? In particular, I would like to see results on MatCha or Pix2Struct if it was pretrained (or finetuned) on the same dataset as UniChart.\n- The ChatGPT evaluation is a bit questionable, given the non-determinism and opaqueness of ChatGPT, and its limitations as a model for providing real (or integer) valued ratings (the paper seems to get ChatGPT to generate ratings from 1-5). One of the cited papers, Luo et al. (2023) also highlights the limitations of such an approach. A better evaluation might be to give ChatGPT inputs from two different models, and ask it to rank them instead. This type of evaluation is more robust than generating score outputs. However, it’s good that the results generally correlate with human scores, so this is not a big issue. I would encourage the authors to de-emphasize the ChatGPT evaluations (maybe even move it to the appendix), given that we don’t know the model details, whether the model has changed since the evaluations were run, and other implementation details.\n", "questions_to_authors": "- Can you provide ablation experiments that would allow us to understand the effect of the pretraining corpus vs. the proposed UniChart model architecture? Which contribution is more important for better performance, and by how much?\n- What are the parameter counts of each of the models in Table 2? It would be great if the parameters and pretraining data used could be listed here for ease of comparison.\n- Did you perform de-duplication of the evaluation datasets used in Table 2 against the pretraining corpus? This is important to ensure that none of the charts used in validation leaked into the pretraining dataset.\n\n", "paper_topic_and_main_contributions": "This paper proposes a new chart-specific pretraining dataset, as well as a new pretrained model for chart comprehension and reasoning. The dataset is derived from several sources: they extract the actual svg charts from various web sources, augment the corpus by generating new charts from data tables with the D3 framework, and obtain image-only charts which they furnish with textual content by applying OCR. They train a new model, UniChart, on this corpus, and show that it outperforms existing models on several evaluation datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No major missing references that I am aware of.", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is generally well written and easy to follow. The font size of Table 1 could be increased as it is difficult to read when printed."}, "extra_text": {}}, {"review_id": "H0r89X5n9k", "reviewer": "Reviewer_8Lrx", "summary": "", "questions": "1.\tThe authors have said that text-davinci-003 can generate “coherent and relevant text”. However, after given the generated text by text-davinci-003, LLM can also do all the tasks (like question answer, summarization and so on). What is the advantage to use UniChart compared to directly using LLM like Flan-T5 XL and other LLMs? Especially, can the UniChart perform better than “text-davinci-003” (by using text-davinci-003 to generate text, and then answer questions and summarize over these generated texts). \n\n2.\tThere are also many data-to-text methods. If the Chat can be converted to a data table (by OCR and other operations), what is the advantage of using UniChart compared to converting the Chat to the data table, and then using the data-to-text method to conduct these tasks? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to concerns in the Reasons To Reject and the Reviewer Questions\n\n1. Real-world data in Data Augmentation (Reason 1)\n\n    * **Response:** Unlike corpus from DVQA/FigureQA which randomly generates the data table values, we only used real-world data tables during training. Our data augmentation pipeline is based on the data tables we extracted from WDC corpus which is a source of real world data tables available on the internet.\n \n2. Ablations experiments on ChartQA (Reason 2)\n\n    * **Response:** As we discussed in the paper (lines 551-556), automatic evaluation metrics like BLEU score for text generation benchmarks like Chart-to-Text and OpenCQA are not very informative. In contrast, the results of ChartQA are more interpretable as they are based on simple relaxed accuracy measures. Therefore we decided to do ablation studies only for ChartQA which can demonstrate the performance of different setups clearly.\n\n3. UniChart vs LLMs and data-to-text models (Reason 3, Question 1, Question 2)\n\n    * **Response:** First, addressing questions regarding a chart requires more information from the chart than just extracted OCR text. For example, in the ChartQA benchmark, a common type of user queries are those that refer to visual elements of a chart (e.g., what is the least difference between light blue bar and dark blue bar?) which cannot be answered using OCR text only. It is already confirmed in the ChartQA paper that models that rely on the data tables only (e.g., TaPas and T5) exhibit lower performance than their multimodal counterparts (e.g., VisionTapas and VL-T5) which also take the chart image into consideration. Moreover, UniChart is an end-to-end reproducible, transparent, as well as memory and time-efficient solution for chart comprehension compared to models like text-davinci-003 which are not publicly available and have larger parameter counts.\n\n## Reproducability \n   * **Response:** Due to the anonymity requirement, we removed our github repo link from the paper (lines 125-126). After the anonymity period is over, we will provide the github repo link that contains all the data and checkpoints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1．\tThe paper highlights the limitations of existing methods in chart-based data analysis tasks. These methods often rely on pretraining from language or vision-language tasks, but overlook the explicit modeling of chart structures.\n\n2．\tThe paper employs an extensive dataset for pretraining, with the authors dedicating significant efforts to the collection and processing of the dataset.\n", "reasons_to_reject": "1. There are certain descriptions that lack precision. For instance, in lines 226-229, there is a statement that reads: \"Our goal was to train the model based on real-world data, thus, we did not consider the ones that are generated from synthetic data.\" However, in the subsequent \"Data Augmentation\" section, the authors proceed to construct charts based on data tables, which seems contradictory.  Therefore, the intent that the authors wanted to convey in lines 226-229 should be expressed in a clearer and more accurate manner, so as to avoid inconsistency with the subsequent content.\n\n2. Ablation experiments were only done on the ChartQA dataset, which I think is somewhat inadequate.\n\n3. I have concerns about the application of UniChart, comparing the UniChart to LLMs and the data-to-text method. Please see my question as below.\n", "questions_to_authors": "1.\tThe authors have said that text-davinci-003 can generate “coherent and relevant text”. However, after given the generated text by text-davinci-003, LLM can also do all the tasks (like question answer, summarization and so on). What is the advantage to use UniChart compared to directly using LLM like Flan-T5 XL and other LLMs? Especially, can the UniChart perform better than “text-davinci-003” (by using text-davinci-003 to generate text, and then answer questions and summarize over these generated texts). \n\n2.\tThere are also many data-to-text methods. If the Chat can be converted to a data table (by OCR and other operations), what is the advantage of using UniChart compared to converting the Chat to the data table, and then using the data-to-text method to conduct these tasks? \n", "paper_topic_and_main_contributions": "This paper aims to address the problem that current approaches to chart analysis ignore explicit modeling of chart structure, and to this end, proposes a generalized visual language pretraining model for chart comprehension and reasoning, UniChart ,which utilizes a chart-based text decoder to encode text, data, and visual elements. The authors propose pre-training tasks for charts, including low-level element extraction and high-level comprehension. Pre-training UniChart on a large scale and then fine-tuning it achieved state-of-art performance in all four tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No", "Typos_Grammar_Style_and_Presentation_Improvements": "No"}, "extra_text": {}}, {"review_id": "H7nC6iZfXw", "reviewer": "Reviewer_hLdJ", "summary": "", "questions": "1. In Table 2, you've compared UniChart with ChatGPT. Could you please clarify how this comparison was conducted? Specifically, I'm interested in knowing whether images were used as input for ChatGPT. Further, I would like to know whether this comparison was fair and reasonable given the fact that ChatGPT is primarily a text-based model. \n\n2. In the same table, it would be beneficial to include the number of parameters for each of the compared methods. This would provide a more detailed overview of the computational complexity involved with each model.\n\n3. For the statistical analysis, could you elaborate on the significance and interpretation of the p-values you've reported in your results?\n\n4. In regard to Table 1, I would like to understand whether all the data mentioned was used for pre-training. If there were any data filtering or preprocessing steps involved, please provide some information about them. Additionally, it would be helpful to know where this collected dataset can be accessed.\n\n5. Lastly, details about the model training are not clearly outlined in the paper. Were the weights of the Chart image encoder and text decoder pre-trained on other datasets, or were they trained from scratch? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to concerns in the Reasons To Reject and the Reviewer Questions\n\n1. ChatGPT (Reason 1 and Question 1)\n    * **Response:** We did not compare UniChart results to ChatGPT. In fact, there is even no mentioning of ChatGPT in Table 2. We used ChatGPT only in evaluating the summaries generated by UniChart and other baselines. To do this, we fed the underlying data table of the chart in text format beside the generated summary to ChatGPT and asked to rate the summary. The detailed pipeline can be found in Figure 7 in the paper. \n\n2. Number of parameters (Reason 2 and Question 2)\n    * **Response:** We will elaborate on the number of parameters and pretraining data for each model in Table 2 in the paper.\n\n3. Statistical Interpretation in Table 3 (Reason 3 and Question 3)\n    * **Response:** We did a statistical test (p-value) for ratings from humans and ChatGPT, with the null hypothesis that the ratings are two independent samples. The p-values in each row in Table 3 demonstrate that it is very infrequent that two rating samples are independent based on the observed ratings. We will elaborate more on this point in the final version of the paper. \n\n4. Dataset Preprocessing (Reason 4 and Question 4)\n    * **Response:** The numbers in Table 1 represent the information after preprocessing the data. During the data preprocessing, we excluded validation and test sets from all the sources that were shared among downstream tasks like Statista and Pew.  Our model was pretrained on all the examples mentioned in Table 1. The dataset will be published in the project repository after the anonymity period.\n\n5. Model Training Details (Reason 5 and Question 5)\n    * **Response:** As mentioned in the A.4 section in Appendix (lines 1289-1291), we initialized the weights from the base Donut weights. Given the importance of this information, we will move this information to the main text from Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors have constructed a large-scale pre-training dataset for charts covering a wide range of themes and visual styles. This is a significant contribution as such a diverse and large dataset can potentially facilitate more research in this area.\n\n2. The model demonstrated state-of-the-art performance on four downstream tasks, showcasing its effectiveness. Moreover, it showed strong generalization capabilities on unseen chart data, which suggests it has been well-trained and has broad applicability.", "reasons_to_reject": "1. Lack of Model Comparison Details: The authors compare their model, UniChart, with ChatGPT in Table 2 without providing sufficient details about how the comparison was conducted. Given that ChatGPT is not designed for visual input, it raises concerns about the fairness and validity of this comparison.\n\n2. Incomplete Presentation of Models: The authors did not include important information, such as the number of parameters for each model, in their comparison. This omission limits the reader's understanding of the complexity and computational demands of the proposed model compared to others.\n\n3. Ambiguity in Statistical Interpretation: The authors used p-values in their analysis without properly explaining their significance or interpretation, causing confusion for readers unfamiliar with statistical testing.\n\n4. Insufficient Information on Data Preprocessing: The paper lacks a clear description of the data preprocessing steps and whether all data in Table 1 was used for pre-training. This omission hinders the reproducibility of the results.\n\n5. Unclear Training Details: The authors did not specify whether the Chart image encoder and text decoder were trained from scratch or used pre-trained weights from other datasets. This lack of information further limits the replicability of the research and the full understanding of the model's capabilities.", "questions_to_authors": "1. In Table 2, you've compared UniChart with ChatGPT. Could you please clarify how this comparison was conducted? Specifically, I'm interested in knowing whether images were used as input for ChatGPT. Further, I would like to know whether this comparison was fair and reasonable given the fact that ChatGPT is primarily a text-based model. \n\n2. In the same table, it would be beneficial to include the number of parameters for each of the compared methods. This would provide a more detailed overview of the computational complexity involved with each model.\n\n3. For the statistical analysis, could you elaborate on the significance and interpretation of the p-values you've reported in your results?\n\n4. In regard to Table 1, I would like to understand whether all the data mentioned was used for pre-training. If there were any data filtering or preprocessing steps involved, please provide some information about them. Additionally, it would be helpful to know where this collected dataset can be accessed.\n\n5. Lastly, details about the model training are not clearly outlined in the paper. Were the weights of the Chart image encoder and text decoder pre-trained on other datasets, or were they trained from scratch? ", "paper_topic_and_main_contributions": "The paper introduces UniChart, a pre-trained model developed specifically for understanding and reasoning based on charts. This is an innovative area of study that addresses an important need in the AI field, as visual representations like charts are a fundamental way humans communicate information. \n\nThe main contributions of this paper are:\n\n1. The development of UniChart, a pre-trained model designed for chart comprehension and reasoning. The model encodes related text, data, and visual elements of charts, and utilizes a text decoder for text generation based on the chart.\n\n2. The creation of a large-scale pre-training dataset for charts. This dataset is versatile and extensive, covering a variety of topics and visual styles, and providing a rich resource for training models.\n\n3. Through pre-training UniChart on this extensive chart corpus, and then fine-tuning it, the authors demonstrate that the model can achieve state-of-the-art performance on four downstream tasks. Notably, UniChart also demonstrates strong generalization capabilities, outperforming methods that lack chart-specific targets and rely on limited chart resources.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "ChartReader: A Unified Framework for Chart Derendering and Comprehension without Heuristic Rules. ICCV, 2023 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers felt generally positively about both the soundness and excitement of the paper, describing the proposed pre-training dataset as "real world... valuable", "extensive", and "diverse" with the evaluation of the new proposed model as "comprehensive", showing its effectiveness on several downstream datasets, and appreciated the analysis-based suggestions for future work on the dataset and task. Some smaller reviewer concerns about presentation/clarity of details, and ablations, were adequately addressed in the author response.
Decision: Accept-Main