ChartAssistant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (2024)

Abstract: Charts play a vital role in data visualization, understanding data patterns, and informed decision-making. However, their unique combination of graphical elements (e.g., bars, lines) and textual components (e.g., labels, legends) poses challenges for general-purpose multimodal models. While vision-language models trained on chart data excel in comprehension, they struggle with generalization. To address these challenges, we propose ChartAssistant, a chart-based vision-language model for universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT, a comprehensive dataset covering diverse chartrelated tasks with basic (e.g. bars and pies) and specialized (e.g. radars, and bubbles) chart types. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text, followed by multitask instruction-following fine-tuning. This approach enables ChartAssistant to achieve competitive performance across various chart tasks. Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and ChartLlama methods, especially outperforming them on real-world chart data with zero-shot setting. The code and data are available at https://github.com/OpenGVLab/ChartAst .

heading: 1 Introduction
text: People around the world generate a multitude of charts daily, including data visualizations for business reports, market analysis, scientific experiments, and data-driven presentations (Horn, 1998; Hoque et al., 2017, 2022). Charts are an effective tool for understanding data patterns, such as the distributional properties depicted in histograms
† Corresponding Authors: shaowenqi@pjlab.org.cn; pluo@cs.hku.edu This work was done when Fanqing Meng and Quanfeng Lu were interning at Shanghai AI Laboratory.
and growth trends illustrated in line graphs. Developing chart learning methods enables the design of machine analysts with enhanced capabilities to solve various chart-related downstream tasks such as chart question answering (QA) (Masry et al., 2022; Kantharaj et al., 2022a; Methani et al., 2020) and chart summarization (Hsu et al., 2021; Rahman et al., 2022).
However, chart comprehension is challenging due to the intricate visual marks (e.g. lines, bars and symbols), implicit numerical information, and complex spatial relationships between elements (e.g. axes and labels). Interpreting charts requires specialized knowledge, spatial reasoning, and numerical understanding. The advanced general-purpose multimodal models (Zhang et al., 2023b; Li et al., 2023a; Zhang et al., 2023a) such as LLaVA (Liu et al., 2023b), trained on natural images, struggle with chart-related tasks due to the specific complex-
7775
ities and relationships unique to charts. Although recent multimodal literate models (Lv et al., 2023; Lee et al., 2023) have achieved impressive results in processing various document-level tasks, they still face difficulties in accurately answering chartrelated questions.
In pursuit of universal chart reasoning and comprehension, prior works propose pre-training vision-language models on chart-related tasks as shown in Fig.1(a). For example, both MatCha (Liu et al., 2022b) and UniChart (Masry et al., 2023) perform multitask instructional tuning on chart data. Although these methods exhibit good performance on several chart-related tasks, they require taskspecific fine-tuning. Moreover, the existing training data (Methani et al., 2020; Masry et al., 2022) is deficient in image-text annotations aimed at improving the model’s comprehension of visual elements and mathematical reasoning, as well as annotated data from the specialized chart types such as box-plots. Due to the above factors, existing chartbased models have poor generalization on various downstream tasks as illustrated in Fig.1(a).
To address these challenges, we propose ChartAssistant, a new multimodal model for universal chart comprehension and reasoning. To improve generalization, ChartAssistant is trained on a large-scale chart-specific instruction-tuning benchmark dubbed ChartSFT. The training process involves a two-stage pre-training pipeline which employs chart-to-table pre-training to align the chart and its structured text and then perform joint tuning on multiple chart-related tasks as shown in Fig.1(b). As a result, our ChartAssistant can achieve good results on various chart-related tasks with a single model. We implement ChartAssistant with two variants, i.e. ChartAst-D and ChartAstS. ChartAst-D is built upon Donut (Kim et al., 2021), a lightweight (260M parameters) but powerful vision-language model for visual document understanding. While ChartAst-S is built upon SPHINX (Lin et al., 2023), a large (13B parameters) vision-language model for universal multimodal comprehension. Inherited from SPHINX, our ChartAst-S obtains enhanced chart representation by dynamic resolution processing and mixed visual encoders. Therefore, ChartAst-S offers increased robustness and usability for chart understanding, demonstrating strong performance in various chart-related tasks.
Specifically, we first construct ChartSFT by collecting instruction-following data from various
chart-related tasks. To address the limitations of existing chart-based benchmarks (Methani et al., 2020; Masry et al., 2022; Kantharaj et al., 2022a), we introduce several modifications to improve the quality of data annotation: 1) instruction-following data involving various topics for chart-to-table translation is added, which we find helps align the chart and the associated structured text; 2) the chain-of-thought annotations for chart numerical QA task are generated to improve mathematical reasoning abilities (Wei et al., 2022); 3) the task of chart referring question answering is created to enhance the understanding of visual elements and their relationships (Chen et al., 2023; Yang et al., 2023); 4) chart with specialized types such as radar and box plot are included to improve the generalization. Overall, ChartSFT encompasses a larger corpus of instruction-following data, incorporates a wider range of chart-related tasks and types, and features more comprehensive data annotations compared to previous benchmarks (Masry et al., 2022; Methani et al., 2020; Kantharaj et al., 2022a).
Before conducting multitask instruction tuning, as done in existing research (Masry et al., 2023; Liu et al., 2022b), we start with pre-training ChartAssistant on the chart-to-table translation task as shown in Fig.1(b). This task involves parsing a chart and generating a Markdown table. It shares similarities with dense captioning for natural images, allowing the model to interpret the elements and relationships within the chart. Similar to the role of image captioning in training multimodal models (Liu et al., 2023b; Shao et al., 2023; Xu et al., 2023), chart-to-table translation facilitates alignment between the chart and its structured text. Following pre-training, we proceed with multitask instruction tuning using ChartSFT. This two-stage training approach enables ChartAssistant (a single model) to achieve strong performance across a range of chart-related tasks.
The contributions of this paper can be summarized as follows. 1) We present ChartAssistant, a vision-language model for chart comprehension and reasoning. ChartAssistant is versatile enough to solve various chart-related tasks across a wide range of chart types. 2) We build a chart-specific visual instruction-following benchmark dubbed ChartSFT. ChartSFT surpasses existing chart-based benchmarks with its larger instruction-following data corpus, a broader range of tasks and chart types, and more comprehensive data annotations. 3) Extensive experimental results on various down-
stream tasks demonstrate that ChartAssistant surpasses the previous SoTA method UniChart (Masry et al., 2023) by 50.0%, 28.1% performance gain on numerical QA and ChartQA, respectively. Notably, ChartAssistant continues to significantly outperform existing chart-specific models in the zeroshot setting, with 29.5% performance gain on RealCQA (Ahmed et al., 2023) compared with Unichart and 23.6% performance gain on ChartLLM (Ko et al., 2023) compared with ChartLlama (Han et al., 2023).

heading: 2 Related Work

heading: 2.1 Multimodal Foundation Model
text: Multimodal foundation models (Li et al., 2023a; Zhu et al., 2023) mainly focus on natural images, which have shown remarkable progress, advancing in areas like image captioning (Vinyals et al., 2015) and visual question answering (Vinyals et al., 2015; Johnson et al., 2017). SPHINX (Lin et al., 2023) leverages LLM and multiple visual encoders to achieve advanced performance on multiple multimodal tasks. Among these, visual document understanding is a topic of both industrial importance and research challenge. Donut (Kim et al., 2021) proposed an OCR-free Transformer trained in endto-end manner,which is a powerful document understanding model. Nougat (Blecher et al., 2023) is fine-tuned on Donut and useful for academic documents understanding. However, extracting information from real-world images like charts and plots presents unique challenges as compared to natural images or documents. Furthermore, the complexity of queries increases, often involving sophisticated mathematical calculations. As a result, contemporary document models and multimodal foundation models often fall short when tasked with handling chart-related tasks, demonstrating a significant decline in performance (Liu et al., 2022b).

heading: 2.2 Chart-specific Vision-Language Model
text: Some methods modify vision-language models for chart-related tasks (Han et al., 2023; Liu et al., 2023a) or develop plugin for LLM to understand the chart (Xia et al., 2023). MatCha (Liu et al., 2022b) extends Pix2Struct (Lee et al., 2023) by integrating mathematical reasoning and chart data extraction tasks, excelling at chart question answering and chart summarization. Unichart (Masry et al., 2023) and ChartLlama (Han et al., 2023) undergoes multitask instruction tuning on many chart-related
tasks, establishing itself as the most versatile and effective chart vision-language model currently available. However, these methods have poor generalization. Furthermore, they struggle with mathematical computations in charts and perform poorly on uncommon chart types such as radars and bubbles. Therefore, we propose ChartSFT, the most extensive dataset to date, supporting a wide variety of chart tasks and types. We develop ChartAssistant using ChartSFT with a two-stage training strategy, capable of handling diverse chart-related tasks.

heading: 3 ChartSFT
text: We construct a large-scale chart-specific instruction-tuning benchmark called ChartSFT by collecting data from various tasks. The composition of ChartSFT is shown in Table 7, as extensively described below. Our ChartSFT consists of 39M pieces of chart-text annotated data, 4.75 and 5.62 times larger than MatCha (Liu et al., 2022b) and UniChart (Masry et al., 2023), respectively, as illustrated in Fig.3. ChartSFT contains charts with both base and specialized types, as presented in Sec. 3.1 and Sec. 3.2, respectively.
Overall, our ChartSFT encompasses nine types of charts by collecting data from various sources as shown in table 12. First, most charts with base types including bar, line, dot-line, and pie are collected from several existing datasets (Masry et al., 2022; Methani et al., 2020; Kantharaj et al., 2022a; Rahman et al., 2022; Li and Tajbakhsh, 2023; Tang et al., 2023; Kantharaj et al., 2022b). Second, we also generate some charts with base types from arXiv tables (arX) and data augmentation techniques (e.g. various APIs and figure parameters). In particular, we use ChatGPT to suggest the proper chart type given each table data from arXiv. Third, we synthesize table data which is appropriate for depicting charts with specialized types.

heading: 3.1 Chart with Base Types
text: We collect instruction-following data with base chart types (i.e. bars, lines, dot-lines, and pies) from 5 chart-rated tasks, including chart-to-table translation, chart numerical QA, chart referring QA, chart open-ended QA, and chart summarization as shown in Fig.2. Instead of directly utilizing existing chart-based benchmarks, we introduce several modifications to improve the data annotation quality. For each task, we present the details of
data collection as follows.

heading: 3.1.1 Chart-to-Table Translation
text: The task of chart-to-table translation aims at parsing a chart into its underlying data table in text form. Pre-training with chart-to-table translation enables our ChartAssistant to comprehend the chart’s elements and their relationships, facilitating alignment of the chart and its underlying structured text.
Data Collection. We collect 17141 and 224386 pieces of chart-text data from ChartQA and PlotQA for chart-to-table translation. However, these benchmarks vary little in chart styles and involve limited topics. We propose two strategies to address the issue. i) More Chart Styles. We re-plot the chart with diverse visualization tools for tables in ChartQA and PlotQA. Specifically, we utilize 5 APIs in Python, including ggplot, plotly, matplotlib, seaborn, and pyecharts, along with over 20 variations in parameters color, size, font type, background, and more. After style augmentation, 220050 pieces of chart-text data are created for chart-to-table translation from PlotQA, respectively. ii) Table from arXiv Papers. We collect more real table data to increase the topic diversity. To this end, we crawl 1301932 papers involving various topics such as computer science, biology, finance, and more from arXiv platform (arX). For each paper, we extract the table from the source LaTeX code where table data can be localized in the table environment. We employ ChatGPT (Ouyang
et al., 2022) to transform the latex table into the markdown table. We also make the chart in a specific base type (e.g. pies) by following ChatGPT’s suggestion. We find that ChatGPT works well to generate text in the target format and give appropriate advice for chart types. There are 132719 pieces of chart-text data obtained from the arXiv.

heading: 3.1.2 Chart Numerical Question Answering
text: Chart numerical QA targets at responding to the request about mathematical reasoning given a chart. It requires an accurate understanding of the chart, as well as reasoning and math calculation abilities.
Data Collection. The data for numerical QA mainly comes from the PlotQA benchmark. However, PlotQA generates numerical QA data from 40 templates with limited types of questions and direct final answers, resulting in poor generalization and math reasoning. with our proposed two strategies to improve the data quality below, more than 24M QA pairs are collected. i) More Templates. We create 101 templates to generate numerical QA questions automatically involving various types of questions with complex calculations. Here is one template for analyzing the correlation between two items: ‘Across all <plural form of X label>, are the <Y label> values of <legend label1> and <legend label2> negatively correlated?’ The comparison between templates in our ChartAssistant and PlotQA is provided in Table 1 where we can see that our improved templates encompass larger
token lengths and more complex calculations. We present all templates in the Appendix A. ii) Chainof-Though (COT) Annotations. Instead of utilizing the final answer as the response annotation, we generate COT annotation for the final answer, which has been proven to improve the model’s mathematical reasoning ability (Wei et al., 2022). We first define a set of available functions to segment the problem’s solution into smaller steps, each encompassing function calls and parameters. These steps are then organized into a JSON-formatted text. As shown in Fig.2, the maximum extraction problem is decomposed into a step of data retrieval and a step of maximum calculation. When computing the answers, the backend executes the calculations by following the ordered function calls within the text. This approach not only enhances reasoning ability but also mitigates calculation errors.

heading: 3.1.3 Chart Referring Question Answering
text: We create a new task for chart named referring question answering, considering that users may utilize a set of marks to denote some pieces to their interest in the chart as shown in Fig.2. Note that referring question answering with a bounding box has been explored in general-purpose multimodal models such as GPT4ROI (Zhang et al., 2023c) and Shikra (Chen et al., 2023) where the referential QA has been shown to benefit comprehending spatial relationships. The task of referring QA is expected to enhance the understanding of visual elements and their relationship in the chart.
Data Collection. We extend a part of COT annotations for numerical QA in Sec.3.1.2 to the task of Referring QA. Three steps are conducted to produce referring QA pairs with diverse patterns. i) The color, size, and width are randomly selected to make the mark. ii) We use several marks such as an arrow and a bounding box to refer to an item in the chart. iii) Multiple marks can be depicted in the same chart to describe the relationships between
elements. Overall, we collect 5899842 pieces of data for the chart referring QA.

heading: 3.1.4 Chart Open-ended QA
text: Chart open-ended QA (OpenQA) deals with openended questions regarding charts as illustrated in Fig.2. It requires both low-level Chart comprehension and high-level reasoning abilities.
Data Collection. We collect data from existing benchmarks, such as plotQA (Methani et al., 2020), ChartQA (Masry et al., 2022), OpenCQA (Kantharaj et al., 2022a) and ScigraphQA (Li and Tajbakhsh, 2023). We further introduce our collected table data from arXiv in Sec.3.1.1 for this task. i) Open-ended QA data by ChatGPT. Other than tabular data crawled in Sec.3.1.1, we extract corresponding captions, and the first paragraph describing the table from the source code of the paper. By utilizing ChatGPT, we generate 3 open-ended QA pairs for each table by feeding the table and the descriptive information.
By putting the above benchmarks together, our ChartSFT covers diverse topics for Open-ended QA. In total, there are 7075243 pieces of data for this task.

heading: 3.1.5 Chart Summarization
text: Chart Summarization is a vital task aimed at generating concise and informative summaries for various types of charts, which has been studied extensively (Herdade et al., 2019; Tang et al., 2023; Kantharaj et al., 2022b).
Data Collection. We collected a substantial amount of existing open-source datasets (Tang et al., 2023; Kantharaj et al., 2022b; Rahman et al., 2022; Kantharaj et al., 2022a), but the scale is still not sufficient. Therefore, we further incorporate
a large-scale chart summarization dataset generated through Knowledge Distillation by Unichart (Masry et al., 2023) into our training process. There are 1006738 pieces of data for the chart summarization task.

heading: 3.2 Charts with Specialized Types
text: Previous chart-based models have exhibited poor performance when dealing with specialized chart types, such as radar, area, histogram, bubble, and box-plot. To enhance the model’s generalization capabilities, we have trained our ChartAssistant on these charts with specialized types. To overcome the challenge of obtaining large-scale real-world chart data, we have employed synthetic data generation techniques. For more detailed information, please refer to Appendix A Through this approach, we can obtain a substantial and diverse collection of complex charts across these specialized types.

heading: 4 Our ChartAssistant

heading: 4.1 Architecture
text: The key to completing the tasks related to charts lies in accurately understanding the content of the charts. As shown in Fig. 4, we implement ChartAssistant with two variants, i.e. ChartAst-D and ChartAst-S, which have 260M and 13B parameters in total. In addition, their input image resolutions are 224 × 224 and 448 × 448, respectively. Both ChartAst-D and ChartAst-S perform well in many chart-related tasks. But ChartAst-D has a smaller size and ChartAst-S enjoys better generalization.
ChartAst-D is a vision-language model for chart understanding built upon Donut (Kim et al., 2022). It consists of a visual encoder Swin-Base (Liu et al., 2021) and a textual BART decoder (Lewis et al., 2019). For an input image XV , the visual encoder employs fixed-sized non-overlapping windows to divide the image and performs self-attention layers to consolidate information across these windows,
which transforms the image into a set of tokens ZV = { zi | zi ∈ Rd, 1 ≤ i ≤ n } , where n is encoded token length and d is the token size. By taking ZV as key and value and tokens of text instruction Xq as the query, the BART decoder generates the corresponding response Yq = (yi) m i=1, and m is the length of responses. ChartAst-S is a large vision-language model for chart understanding built upon SPHINX (Lin et al., 2023). For high-resolution images, it preserves the original information through sampling and partitioning methods, ensuring greater fidelity to the image content. Moreover, SPHINX leverages the abundant prior knowledge of LLM (Touvron et al., 2023) to handle various tasks such as visual question answering and image summarization. Specifically, for an input image XV . ChartAst-S incorporates multiple visual encoders to extract more informative visual features ZV , such as DINOv2 (Oquab et al., 2023), CLIP (Radford et al., 2021), and ConvNeXt (Woo et al., 2023). Unlike ChartAst-D where visual tokens are involved in a language decoder with a cross-attention module, ChartAst-S directly appends visual tokens to the text tokens Xq. The merged tokens are then fed into the LLM to generate the response. Thanks to the intricate design of the visual encoder and the powerful reasoning ability of LLM, ChartAst-D generalizes well in various real-world chart-related applications.

heading: 4.2 Training
text: In our ChartSFT, we have a corresponding instruction Xq and response Yq for each image XV . We input these image-text pairs into the model. The objective is to minimize the cross-entropy loss of predicting the next token. To improve the generalization in various downstream tasks, we adopt a two-stage training pipeline to train our ChartAst-D and ChartAst-S below.
Stage I: Pretraining on Chart-to-table Translation. Charts are special images that visualize the data and underlying relationships between elements in the chart. Understanding the numerical values and their meanings is a prerequisite for completing downstream tasks related to charts. Given a chart Xc2tV , this stage is to convert the chart into a text-form table Y c2tq under the instruction Xc2tq . Here the superscript c2t indicates the instruction-following data comes from the task of chart-to-table translation. Our training loss function for Stage I is given by LStage1 = −∑mi=1 logPθ(Y c2tq,i |Xc2tV , Xc2tq , Y c2tq,<i) where Y c2tq,<i are all the response tokens before the current prediction token Y c2tq,i . θ are the learnable weights initialized from the pre-trained weights of the Donut model (Kim et al., 2021). By the pre-training, we align the chart with its structured text-form table, enabling the model to comprehend elements in charts and their relationships. We show that this strategy better serves the multitask instruction tuning in Sec.5.4.
Stage II: Multitask Instruction Tuning. In this stage, we put all the instruction-following data together from five tasks in our ChartSFT. We employ a single model to solve all the tasks. Our training loss function for Stage II is given by LStage2 = −∑k∈Ω ∑m i=1 logPθ(Y k q,i|XkV , Xkq , Y kq,<i), where Ω is the set of instruction-following data from all tasks in ChartSFT and θ are the learnable weights initialized from the checkpoint in the Stage I. During training, we sample the data from each task with certain proportions as provided in our experimental setup in Appendix B. By multitask instructional tuning, our ChartAssistant exhibits strong performance on all the tasks.

heading: 5 Experiment
text: we present our experimental setup in Appendix B, where we indicate the training details. After that, we provide an overview of the selected baselines and evaluation details in Sec.5.1 and demonstrate the superior effectiveness of our method through extensive experiments in Sec.5.2 .

heading: 5.1 Baselines and Evaluation
text: Evaluation. We assess the performance of ChartAssistant across various tasks and datasets. Following the evaluation of Unichart (Masry et al., 2023), we utilize Chart-to-text (Kantharaj et al., 2022b) for evaluating chart summarization task,
OpenCQA (Kantharaj et al., 2022a) and ChartQA (Masry et al., 2022) for open-ended question answering task. To evaluate numerical question answering and referring question answering, we sample test sets from the datasets constructed in Sec.3.1.2 and Sec.3.1.3 called MathQA and ReferQA. Lastly, we conduct separate evaluations on base type and specialized type charts to highlight the superior performance of our method more explicitly. We put a detailed description of the dataset and more experiments in Appendix B.
Baselines. We choose SPHINX (Lin et al., 2023), Monkey (Li et al., 2023b), Blip2-flant5xl (Li et al., 2023a), Qwen-VL (Bai et al., 2023), ChartLlama (Han et al., 2023), Unichart (Masry et al., 2023), MatCha (Liu et al., 2022b), Pix2Struct (Lee et al., 2023), T5 (Raffel et al., 2020) and ChartT5 (Zhou et al., 2023) as baselines. We provide a detailed description in Appendix B.

heading: 5.2 Main Results
text: Charts with base types. In table 2, we present a comprehensive summary of ChartAssistant’s performance on charts with base types across chartrelated tasks. It demonstrates that ChartAssistant consistently outperforms the baseline across all tasks. In particular, we surpass the current leading methods ChartLlama by 17% and 2.5% on ChartQA-human and ChartQA-augment, respectively. Besides, Most existing models struggle with numerical question answering, while the COT answer significantly enhances performance for our models, demonstrating a substantial 16.1% improvement over MatCha. Notably, existing models cannot handle the chart referring question answering task effectively. Overall, our model is the top performer across all chart-related tasks. It is important to note that both the performance of Unichart and MatCha’s are obtained after task-specific fine-
tuning with the training set of the test dataset, whereas ChartAssistant’s results are obtained using a single model after a two-stage training.

heading: 5.3 Zero-shot Study
text: To validate the generalization of ChartAssistant, we test it on samples not included in the training set. To this end, We sample 200 examples from StructChart (Xia et al., 2023) and RealCQA (Ahmed et al., 2023), including two types: mathematical computation and numerical extraction, and collect all 48 publicly available examples from ChartLLM (Ko et al., 2023) for tasks like chartto-table translation, chart-based question answering, and summarization. For evaluation, RealCQA uses accuracy within a 5% error margin, ChartLLM employs GPT-4 scoring used in ChartLlama (Han et al., 2023), while StructChart is evaluated using RMSF1 metrics. As shown in table 3, we find ChartLlama performs poorly in precise numerical question answering but excels in summarization tasks. We attribute this to the robust language capabilities of LLM. But ChartAssistant surpasses existing models in tasks such as precise numerical question answering in OCR and summarization, which involves generating long texts. Furthermore, we observe that if the model’s decoder is not powerful enough, errors are more likely to occur in the zero-shot setting when tasked with generating long text outputs, such as in summarization or providing answers in COT format. The use of Large Language Models (LLMs) can significantly alleviate this issue. Overall, our ChartAst-S exhibits the best zero-shot performance across all tasks.

heading: 5.4 Ablation Study
text: We thoroughly analyze the key aspects of our approach. We first consider the significance of alignment pre-training and the referring question answer-
ing task. Furthermore, We put more experiments in Appendix C, including the key component of ChartSFT. We adopt ChartAst-D to illustrate the superiority of our designed ChartSFT, as well as to emphasize the importance of the training strategy.
The impact of alignment pretraining. We initially validate the importance of alignment pretraining. We ensure that the "Ours w/o align" version of the model is trained for the same number of iterations as the full ChartAssistant model. Table 4 shows that using only multitask instruction tuning falls considerably behind two-stage training strategies. Exact numerical recognition greatly influences mathematical calculation accuracy, leading to a 9.8% and 3.2% performance drop for MathQA and ChartQA-human tasks. We think alignment pre-training, which allows the model to learn charttable correlations, helps the model better adapt during multitask instruction tuning than handling these processes separately (Liu et al., 2023b).
The impact of referring question answering task. In our experiments, we have observed that integrating referring question answering into multitask instruction tuning training can enhance the model’s performance in other tasks. As shown in table 4, incorporating the referring question answering task leads to improvements across almost all tasks, particularly in tasks requiring mathematical reasoning. For instance, the average performance in ChartQA improves by 3.1%, and in MathQA, it improves by 11.9%. We believe that this task strengthens the model’s ability to understand the visual elements and their relationship in the chart, which contributing to overall performance enhancement (Zhang et al., 2023c; Chen et al., 2023).
Error case analysis. Although our model currently achieves the most competitive performance on various chart-related tasks, there is still significant room for improvement. For example, ChartAst-S achieves only 65.9% accuracy on the human split of ChartQA. As an example, we conduct an error analysis on this segment. Within ChartQA-human, there are a total of 1,250 QA
pairs. The questions mainly focus on extracting elements and solving mathematical questions and also include some questions about basic chart attributes. For ChartAst-S, there are a total of 426 errors.
As shown in table 5, we categorize the errors into i) inability to generate CoT; ii) generated CoT but with computation errors; iii) errors in extracting numbers; and iv) others. The results are reported in Table D where we see that ChartAst-S mainly occur in mathematical calculations. Error types i) and iii) account for less than 8% of the total errors. Among mathematical questions, 58.5% of errors result from wrong COTs. Although the remaining 41.5% of answers contain the correct COT steps, the calculation error occurs because the extracted elements are wrong. Hence, our future work would focus on enhancing the model’s ability to extract multiple elements based on the question.
Further, we analyze the errors related to generating the wrong CoT or failing to generate CoT. Approximately 44% of these errors occur because the question incorporates visual information from the chart, such as size, colour, or position (e.g., "What is the value of the longest blue bar?"). Previous works like TinyLVLM-eHub (Shao et al., 2023) also found that multimodal models are deficient in identifying visual commonsense such as shape and colour. It implies that the ability to recognize visual commonsense should be improved.

heading: 6 Conclusion
text: Our work is aimed at developing a generalized multimodal model for chart-related tasks. We propose ChartSFT, a comprehensive and expansive dataset with the most diverse range of supported chart tasks and types. In conjunction, we suggest ChartAssistant, a multimodal model trained using a two-stage strategy over ChartSFT, which can achieve state-ofthe-art results across multiple chart-related downstream tasks. Through detailed experiments, we further demonstrate the superiority of it.

heading: 7 Limitations
text: Due to limitations in the training data and the large vision-language model employed, the current ver-
sion of ChartAssistant performs significantly better on English than on those in other languages. To overcome this issue, we aim to enhance our model by incorporating multilingual training data and expanding the range of chart types supported.

heading: A ChartSFT
text: A.1 Chart Types
Our ChartBench encompasses nine types of charts by collecting data from various sources as shown in table 6. First, most charts with base types including bar, line, dot-line, and pie are collected from several existing datasets (Masry et al., 2022; Methani et al., 2020; Kantharaj et al., 2022a; Rahman et al., 2022; Li and Tajbakhsh, 2023; Tang et al., 2023; Kantharaj et al., 2022b). Second, we also generate some charts with base types from arxiv tables and data augmentation techniques (e.g. various APIs and figure parameters). In particular, we use ChatGPT to suggest the proper chart type given each table data from arxiv. Third, we synthesize table data which is appropriate for depicting charts with specialized types.
A.2 Details of Chart Data Generation in ChartSFT
We illustrate the pipeline of data generation in Fig. 5. In a concrete manner, the chart data are generated in the following stages:
Stage 1: Table generation: Taking into account the diversity of tabular data, we have predefined over 20 types of probability density distributions, including normal distribution, uniform distribution, beta distribution, Laplace distribution, and more. For each sample, we randomly choose one type of probability density distribution and utilize it to generate values. For different types of charts, we impose further constraints on these values based on their characteristics. (e.g., value range, ratio of positive and negative values, range interval). For radar, bubble and area charts, We directly utilize randomly generated values as the tabular data. For histogram and box plot, we generate an array of extensive values using this distribution and calculate the statistical metrics of this array to serve as the tabular data (e.g., frequencies corresponding to histograms, upper whiskers corresponding to box plots). And then we use the generated data to prompt ChatGPT for creating titles, legends, and labels that align with the numerical characteristics.
Stage 2: Chart generation: To ensure the diversity of the generated charts, we utilize multiple plot APIs, such as matplotlib, plotly, pyecharts, ggplot, seaborn, altair, and more, to plot a variety of styles of the chart. For each chart, we randomly select the following parameters: line (style, thickness), font (style, size, bold, italic), colors, markers,
the position of the elements (title, labels, legends), the size of the charts and so on. Besides our own synthetic tabular data, we also use the table from PlotQA (Methani et al., 2020), ChartQA (Masry et al., 2022), ChartSumm (Rahman et al., 2022) and Chart-To-Text (Kantharaj et al., 2022b) to plot the charts for area and radar charts.
Stage 3: Instruction Data generation: For the chart summarization and open-ended QA tasks, we instruct ChatGPT to build datasets by supplying both the table and the corresponding types of charts. For numerical QA and referring QA tasks, we adhere to the approach of the chart with base types by crafting a series of mathematical question templates tailored to the distinct characteristics of various chart types. Subsequently, we manually generate answers with COT annotations.
We adopted a flexible approach by combining ChatGPT with human intervention, which included the utilization of predefined distributions and custom coding of plot API, among other techniques. Through this three-stage chart data generation process, we ensured the diversity and complexity of the table, chart, and instruction data, respectively. As a result, we were able to generate a substantial volume of diversified high-quality chart data.
A.3 Numerical QA Templates We present all the Numerical QA templates in this section. We systematically record both the number of steps in the COT annotation and the number of unique functions used to obtain for each template. Fig 12 shows 101 general templates designed for charts with different types. However, not all of these general templates are applicable to all types of charts. Hence, we’ve customized templates to match the unique characteristics of several specific chart types, such as box plots, bubbles, histograms, and pies, as demonstrated in fig . 17
A.4 Details of Referring QA in ChartSFT In this section, We introduce the details of the generation pipeline of referring QA in our ChartSFT.
Chart Generation. We generate charts with the referring box in two ways. 1) For base types of charts, we utilize the bounding box annotations from plotQA to add referring markers onto their original images. 2) For specialized types of charts, we directly generate charts with integrated referring markers leveraging certain Python API(e.g., matplotlib) functionalities. Fig.11 shows different types of charts with different referring markers.
QA Generation. Following the pipeline used in generating numerical QA templates, we extend its application to the referring QA task. As outlined in fig. 18, we define a total of 114 templates, encompassing questions related to label recognition and mathematical calculations. Note that the x_tick of line and area charts is continuous, therefore, we tailor these templates to accommodate such scenarios.

heading: B Experiments
text: B.1 Experimental Setups
We begin by conducting alignment pre-training, utilizing the chart-to-table translation task for 65k steps. Following that, we engage in multitask instruction tuning. We employ the Adam optimizer (Kingma and Ba, 2014) with a scheduled learning rate, where the initial rate is set to 5e-5 for
ChartAst-D and 2e-6 for ChartAst-S. The input resolution is established at 448×448, while the maximum length in the decoder is defined as 1536 for ChartAst-D and 2048 for ChartAst-S. After training for four epochs for ChartAst-D and only one epoch for ChartAst-S, we perform testing on multiple downstream tasks. During inference, each task receives an image and a textual instruction as input, and the model generates a textual answer. All training processes are carried out on 16xA100 80GB GPUs. ChartAst-S outperforms ChartAst-D and has stronger robustness. This is partly due to the special high-resolution image handling method employed by ChartAst-S, which retains more detailed chart information. Additionally, ChartAst-S incorporates richer pre-training knowledge and the larger model possesses greater robustness.
Evaluation. We assess the performance of ChartAssistant across various tasks and datasets. Following the evaluation of Unichart (Masry et al., 2023), we utilize the test set of Chart-to-text (Kantharaj et al., 2022b) for evaluating chart summarization task, and test sets of OpenCQA (Kantharaj et al., 2022a) and ChartQA (Masry et al., 2022) for open-ended question answering task. The ChartQA dataset consists of two subsets: augmented and human. The augmented set comprises machinegenerated summaries with a predominantly extractive nature, while the human set contains manually crafted summaries that require more advanced reasoning. The Chart-to-Text task encompasses two sets named "Pew" and "Statista" indicating the origin of the image examples. In the Pew set, summaries are automatically extracted from areas surrounding the images, while in the Statista set, summaries are authored by human annotators. We use ChartQA and PlotQA to evaluate chart-to-table translation tasks due to their various chart styles. To evaluate numerical question answering and referring question answering, we sample test sets from the datasets constructed by ourselves called MathQA and ReferQA. When evaluating the effectiveness, we test three times and take the average.
Metrics. For evaluating ChartQA, MathQA, and ReferQA, we adopt the approach used in previous studies (Liu et al., 2022b; Masry et al., 2023), which considers relaxed correctness (allowing for an exact match with tolerance for a 5% numerical error). As for Chart-to-Text and OpenCQA, we employ BLEU as the evaluation metric following previous works (Liu et al., 2022b; Masry et al., 2023). For chart-to-table translation, we use RMSF1 from
DePlot (Liu et al., 2022a). Baselines. We choose SPHINX (Lin et al., 2023), Blip2-flant5-xl (Li et al., 2023a), Qwen-VL (Bai et al., 2023), ChartLlama (Han et al., 2023), Unichart (Masry et al., 2023), MatCha (Liu et al., 2022b), Pix2Struct (Lee et al., 2023), T5 (Raffel et al., 2020) and Chart-T5 (Zhou et al., 2023) as baselines. ChartLlama and Unichart are the current state-of-the-art models that handles the maximum number of chart tasks and delivers the best overall performance. Besides, Unichart also considers the open-ended QA task. MatCha outperforms previous models in mathematical calculations. Pix2Struct and Donut stands out as an excellent document understanding model. We finetune these document models on the train set of the respective evaluation datasets and present the results. T5 is a text-to-text model and needs OCRbased system to extract the data table from the chart image, Chart-T5 is a model modified from T5 for chart-related tasks. We use the results from Unichart (Masry et al., 2023) for them. SPHINX (Lin et al., 2023), Blip2(Li et al., 2023a) and QwenVL (Bai et al., 2023) are all commonly used large vision-language models at present. We observe that these models underperform in processing Chart tasks. Finally, ChartLlama, utilizing LLaVA for training on Chart data, demonstrates superior performance in Chart tasks. Therefore, we only compare with ChartLlama.
B.2 More experiments
Specialized type charts. Following the similar training strategy shown in Fig.2, we fine-tune ChartAssistant on chart data of specialized types. As depicted in table 9, compared to the current chartspecific vision-language models, none of them can generalize effectively to specialized types of charts due to lack of these training data. ChartAssistant demonstrates an absolute advantage in all five tasks related to specialized types of charts compared to them.
Chart-to-Table translation on PlotQA A significant portion of the ChartQA (Masry et al., 2022) dataset labels corresponding numerical data on the charts, but there also exists a considerable amount
of charts where the numbers are not visualized. Consequently, we utilize the PlotQA (Methani et al., 2020) dataset to conduct additional chartto-table translation experiments. As table 8 shows, the results indicate that compared to the ChartQA dataset, the ChartAssistant demonstrates a more significant advantage when implemented on the PlotQA dataset.

heading: C Ablation Study
text: We thoroughly analyze the key aspects of our approach.
The impact of arXiv data. we conduct experiments by excluding the arXiv data at two distinct stages: the alignment pre-training (stage 1), and the multitask instruction tuning (stage 2). As shown in table 10, it demonstrates that the arXiv dataset significantly assists the model in aligning charts with tables, thereby improving the performance across various tasks. We believe this is due to the fact that in comparison to existing chart-to-table translation datasets, the arXiv dataset boasts more diversity in terms of style and context; Besides, the open-ended question-answering task contributed by the arXiv dataset is proved to be pivotal for the multitask instruction tuning. We note that the removal of this leads to a drop in the performance of all tasks, most notably math QA and the referring QA. The possible reason for this is because the context and diverse meanings of the arXiv dataset contribute to higher quality question and answering pairs. Therefore, it better promotes multitask tuning.
COT answer vs. Direct answer for numerical question answering. In Fig.6, we compare using COT answer with direct answer in the same training pipeline for the chart numerical question answering task. Using COT answers instead of direct answers increases the accuracy from 51.9% to 72.1%, with improvements across all chart types, especially in dot-line and line charts, where accu-
Table 10: A comparison of the results of ChartAssistant without arXiv dataset on five tasks with base type charts, which indicates that the arXiv dataset significantly improve the performance of the alignment pre-training and mulittask instruction tuning.
ChartQA Chart-to-Text Chart-to-Table
Model aug. human Pew Statista ChartQA OpenCQA MathQA ReferQA stage1 w/o arXiv 89.9 43.7 13.8 39.1 91.1 14.5 64.1 61.1 stage2 w/o arXiv 89.7 42.6 12.6 37.5 91.3 13.2 56.7 56.4 Ours-D 91.3 45.3 14.0 40.2 92.0 14.9 72.1 64.2
Figure 6: A comparison of the results of using COT answer and direct answer on numerical question answering task, which indicates that using COT answer significantly enhances the model’s capability in handling chart numerical question answering tasks with all types.
racy has increased by 22% and 26.6% respectively. This improvement indicates the effectiveness of COT answers in elevating the overall accuracy and performance across various chart types, which reflects that using COT answers teaches the model the reasoning steps and offloads the calculations to the backend system, thus boosting the model’s mathematical computation ability.
Compared with Unichart after task-specific fine-tuning(except for Chart-to-Text). We employ the same training strategy and train with the identical model to highlight the effectiveness gains from our data. Following Unichart’s lead in multitask instruction tuning, as table 11 shows, we finetune the model on various test datasets (apart from Chart-to-Text, it utilizes fine-tuning during testing), resulting in improvements across different tasks surpassing those of Unichart. It is noteworthy that both Unichart and ChartAst-D are trained using Donut, emphasizing the superiority of ChartSFT.
The impact of each multitask instruction tuning component. We evaluated the impact of each segment in our multitask instruction tuning by excluding one task at a time during training and not-
ing effects on ChartQA performance. As table 12 shows, any omission led to a performance drop. In particular, chart summarization’s contribution is smallest, possibly because ChartQA centers on data extraction and numerical question answering and not overall chart understanding. Furthermore, a significant performance decline when the numerical question answering task is excluded underlines its critical importance for the model.
Key components of ChartSFT analysis. For reasoning tasks involving specific numerical values, such as ChartQA, as shown in table 12, the math question-answering task benefits greatly from this, especially, as illustrated in fig .6, training in COT-format can significantly enhance the accuracy of mathematical computation problems. For tasks involving the output of long texts, such as openCQA, as demonstrated by table 4 and table 10, we find that incorporating a question-answering dataset composed of arXiv data can to some extent improve the performance of these tasks. We believe this is due to the broad scope, diversity, and specificity of the arXiv data. Moreover, compared to SciGraphQA (Li and Tajbakhsh, 2023), the arXiv data we provide has precise numerical values, results in higher quality question generation. Lastly, thanks to the robust language capabilities of GPT-3.5, it is capable of generating high-quality, comprehensive question-answering datasets.
The impact of generating equivalent math questions. Considering that generating questions purely through templates can be rather rigid in the math question answering task, we attempt to provide both the template questions and table information to ChatGPT simultaneously, asking it to gener-
ate more significant equivalent questions based on the meaning of the tables. In particular, "What is the difference between the highest and the lowest Amount of Least developed countries ?" can be converted to "What is the range of the Amount for Least developed countries ?". We divide these new question-answer pairs into training and test sets, then compare the performance on the test set when training with and without this additional data.
As fig. 7 demonstrates, we find that including the newly generated equivalent questions in the training can enhance the performance of all types compared to the original approach. In detail, the overall accuracy changes from 71.8% to 76.2%.

heading: D Some demos from Out of Distribution
text: To demonstrate the model’s generalization capability, we randomly take screenshots of several charts, as shown in Fig .9 and Fig .10 . We find that the model possesses generalization ability on out-ofdistribution samples. Additionally, as shown in fig. 8, we visualize some demos comparing the performance of zero-shot scenarios with baseline methods. We observe that in summarization tasks, UniChart and MatCha tend to produce repetitions or hallucinations, whereas ChartLlama and ChartAssistant exhibit relatively stronger capabilities in handling summarization tasks. However, ChartLlama commits some factual errors; in question answering, thanks to the incorporation of COT-format QA training data, ChartAssistant effectively addresses QA tasks requiring mathematical reasoning. Lastly, in chart-to-table translation, UniChart and MatCha accurately model the table structure.
Although ChartLlama can model the table structure accurately, the values are completely incorrect. Only ChartAssistant successfully constructs the table of the chart accurately.
Table 13. Referring QA Templates in ChartSFT.
NO. Template COT Steps Func. Num. General
1 What does the <box0> represent? - - 2 What does the <color0> box represent? - -
3 What does the <arrow0> represent? - -
4 What does the <color0> arrow represent? - - 5 What is the label of <box0> ? - - 6 What is the label of <color0> box? - - 7 What is the label of <arrow0> ? - - 8 What is the label of <color0> arrow? - - 9 Across all <plural form of X label> , what is the maximum <Y label> of the legend
represented by the <arrow0> ? 2 2
10 Across all <plural form of X label> , what is the maximum <Y label> of the legend represented by the <color0> arrow?
2 2
11 Across all <plural form of X label> , what is the maximum <Y label> of the legend represented by the <box0> ?
2 2
12 Across all <plural form of X label> , what is the maximum <Y label> of of the legend represented by the <color0> box?
2 2
13 Across all <plural form of X label> , what is the maximum <Y label> of the <color0> arrows?
2 2
14 Across all <plural form of X label> , what is the maximum <Y label> of the <color0> boxes?
2 2
15 Across all <plural form of X label> , what is the minimum <Y label> of the legend represented by the <arrow0> ?
2 2
40 If total <Y label> is <N> , how many <Y label> would be in <ithx tick> ? 5 4
41 If <ithx tick> is removed, what would be the new percentage of <Y label> of/in <jthx
tick> ? 6 4
42 What is the proportion of <Y label> of/in <ithx tick> ? 4 3 43 What is the total proportion of <Y label> of/in <ithx tick> and <jthx tick> ? 6 4
Table 13. Referring QA Templates in ChartSFT.
NO. Template COT Steps Func. Num. General
1 t does the <box0> represent? - 2 What does the <col 0> box represent? - - 3 What does the <arrow0> represent? - - 4 What does the <col 0> arrow r present? - - 5 What is he label of <b x ? - - 6 What is the label of <color0> box? - - 7 What is he label of <arrow0> ? - - 8 What is the label o <color0> arrow? - - 9 Ac o s all <plural form of X label> , what is the maximum <Y label> of the legend
represented by the <ar ow0> ? 2 2
10 A ross all <plural form of X label> , what is the maximum <Y label> of the legend represented by the <color0> arrow?
2 2
11 A ross all <plural form of X label> , what is the maximum <Y label> of the legend represented by the <box0> ?
2 2
12 Across all <plural form of X label> , what is the maximum <Y label> of of the legend represented by the <color0> box?
2 2
13 Across all <plural form of X label> , what is the maximum <Y label> of the <color0> arrows?
2 2
14 Across all <plural form of X label> , what is the maximum <Y label> of the <color0> boxes?
2 2
15 Across all <plural form of X label> , what is the minimum <Y label> of the legend represented by the <arrow0> ?
2 2
Figure 18: Referring QA Templates in ChartBench.

heading: Future Work
text: Hence, our future work would focus on enhancing the model’s ability to extract multiple elements based on the question. Further, we analyze the errors related to generating the wrong CoT or failing to generate CoT. Approximately 44% of these errors occur because the question incorporates visual information from the chart, such as size, colour, or position (e. g. , "What is the value of the longest blue bar?"). Previous works like TinyLVLM-eHub (Shao et al. , 2023) also found that multimodal models are deficient in identifying visual commonsense such as shape and colour. It implies that the ability to recognize visual commonsense should be improved