ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning (2024)

Abstract: Charts provide visual representations of data and are widely used for analyzing information, addressing queries, and conveying insights to others. Various chart-related downstream tasks have emerged recently, such as questionanswering and summarization. A common strategy to solve these tasks is to fine-tune various models originally trained on vision tasks language. However, such task-specific models are not capable of solving a wide range of chartrelated tasks, constraining their real-world applicability. To overcome these challenges, we introduce ChartInstruct: a novel chart-specific vision-language Instruction-following dataset comprising 191K instructions generated with 71K charts. We then present two distinct systems for instruction tuning on such datasets: (1) an end-to-end model that connects a vision encoder for chart understanding with a LLM; and (2) a pipeline model that employs a two-step approach to extract chart data tables and input them into the LLM. In experiments on four downstream tasks, we first show the effectiveness of our model–achieving a new set of state-of-the-art results. Further evaluation shows that our instruction-tuning approach supports a wide array of real-world chart comprehension and reasoning scenarios, thereby expanding the scope and applicability of our models to new kinds of tasks.

heading: 1 Introduction
text: Information visualizations, such as bar charts and line charts, play a pivotal role in data analysis, offering critical insights and aiding in informed decision-making (Hoque et al., 2022). However, discerning key patterns and trends from these visualizations and addressing complex queries can pose significant challenges. Recent research has introduced various tasks to assist users in chart analysis, including chart question answering (Masry
∗Equal contribution.
et al., 2022a; Kantharaj et al., 2022b), summarizing insights from visualizations (Obeid and Hoque, 2020a; Shankar et al., 2022), reasoning over chart images, fact-checking (Akhtar et al., 2023a), and automated visual data storytelling (Shi et al., 2020).
Early work attempts to tackle these tasks by finetuning models originally trained on language and vision tasks (Raffel et al., 2020; Masry et al., 2022a; Lee et al., 2022; Kantharaj et al., 2022c). However, such models may not be optimal for chart-specific tasks as they overlook explicit modeling of chart structures such as relationships between chart elements like bars, legends, and axes. Recent models such as UniChart (Masry et al., 2023), Chart-T5 (Zhou et al., 2023), and MatCha (Liu et al., 2022b) are specifically designed for charts by considering visual and mathematical reasoning over chart elements and values. However, they often consider charts from a limited range of sources and focus on a narrow set of tasks, constraining their real-world applicability. Indeed, for real-world widespread adoption, we cannot presume how and on what tasks these models will be used.
A promising solution to this challenge is instruction tuning, as demonstrated by language models like InstructGPT (Ouyang et al., 2022a), FLANT5 (Chung et al., 2022), Alpaca (Alpaca, 2023), (Chiang et al., 2023) and LLaMA-chat (Touvron et al., 2023). They show that training LLMs on instruction-following datasets significantly enhances their alignment with user intent across various tasks, including tasks that are unseen during training. Recent advances in vision-language tasks (Li et al., 2023a; Dai et al., 2023) have adopted similar methodologies, fine-tuning vision-language models (VLMs) with visual instructions to better match user intentions and improve efficacy. However, to our knowledge, instruction tuning for chart comprehension and reasoning remains underexplored. Existing methods (Liu et al., 2023a; Han et al., 2023), which are concurrent to our work,
10387
lack variety and scope in instruction-tuning tasks, limiting their effectiveness in real-world chart understanding scenarios.
In this paper, we introduce Chart Instruction Tuning (ChartInstruct), to pave the way towards building general-purpose chart comprehension and reasoning assistant based on VLMs. To this end, we have developed a new chart instruction-tuning dataset featuring real-world charts collected from 157 online platforms, covering wide and diverse visual styles. Leveraging advanced LLMs such as GPT-3.5 (OpenAI-Blog, 2022) , GPT-4 (OpenAI, 2023), and Gemini (Team et al., 2023)), we generate 191K instructions covering a broad array of tasks reflecting real-world applications (Figure 1).
As charts are unique and pose challenges distinct from general multi-modal data, a structured approach is crucial for enabling VLMs to effectively leverage the instruction dataset, optimizing their performance in chart analysis tasks. We introduce two innovative VLM designs in this regard. The first system modifies the LLaVA architecture (Li et al., 2023a), substituting its CLIP vision encoder with the UniChart (Masry et al., 2023) vision encoder pre-trained specifically on chart images. For language modeling in this design, we experiment with two different models: Llama2 (7B) – a decoder-only model (Touvron et al., 2023)
and Flan-T5-XL (3B) – an encoder-decoder model (Chung et al., 2022). Our second design adopts a two-step pipeline approach: first extract the underlying data table from the chart image, then provide it as input to the LLM. This range of models provides a spectrum of efficient solutions, making our systems adaptable to various real-world scenarios and computational demands.
Our comprehensive evaluation across four benchmarks: ChartQA (Masry et al., 2022b), Chart2Text (Obeid and Hoque, 2020b), OpenCQA (Kantharaj et al., 2022b), ChartFC (Akhtar et al., 2023a) demonstrates our models’ state-of-the-art performance in chart understanding and reasoning tasks. Human evaluation further suggests the effectiveness of our instruction-tuning approach in supporting a wide array of real-world chart comprehension and reasoning scenarios, broadening its adaptability to numerous new tasks.
Our main contributions include: (i) A new instruction-following corpus with real-world charts and a wide range of tasks by utilizing LLMs, (ii) two distinct systems specifically tailored for chart understanding tasks; (iii) extensive evaluations that demonstrate the state-of-the-art performance of ChartInstruct across existing chart-related benchmark tasks while also expanding its applicability to new tasks. We have made our code and chart
corpus publicly available at https://github.com/visnlp/ChartInstruct.

heading: 2 Related Work

heading: 2.1 Chart Modeling
text: Chart understanding methods fall into two main categories: those directly fine-tuned from language or vision-language models (Masry et al., 2022b; Masry and Hoque, 2021; Lee et al., 2022) and those specifically crafted for chart-specific tasks (Masry et al., 2023; Liu et al., 2022b; Zhou et al., 2023). Models in the former category often exhibit limited performance due to a lack of chartspecific pretraining. In contrast, models in the latter group are pretrained on primitive chart-specific tasks like question answering and summarization which constrains their applicability across diverse real-world chart scenarios. Recent works (Wang et al., 2023a; Do et al., 2023; Huang et al., 2023) have also employed LLMs such as GPT-3, Llama (Touvron et al., 2023), and GPT-4 for chart-related tasks. They utilize pipelines that extract data values from chart images using specialized models like UniChart (Masry et al., 2023) and Deplot (Liu et al., 2022a), which are then used by the LLMs in different downstream tasks like question answering and summarization. However, these methods either rely on proprietary models (Brown et al., 2020; OpenAI, 2023) or public models without chart-specific training, limiting their effectiveness and generalization.

heading: 2.2 Visual Instruction Tuning
text: Instruction tuning in LLMs has shown benefits in aligning models with human intent and enhancing task generalization (Chung et al., 2022; Ouyang et al., 2022b; Wang et al., 2023b; Alpaca, 2023; Chiang et al., 2023). These techniques have also been extended to the vision-language space (Li et al., 2023a; Zhu et al., 2023; Ye et al., 2023; Li et al., 2023b; Dai et al., 2023). However, visual instruction tuning approaches in the chart domain are rare. Although a few studies have implemented multimodal instruction tuning (Liu et al., 2023a; Han et al., 2023), they depend on CLIP (Radford et al., 2021) for vision encoding, designed for natural images rather than charts. Moreover, their training often relies on synthetic charts or a narrow selection of real-world charts, focusing on a narrow set of instruction-tuning tasks. In contrast, our work presents two systems explicitly designed
for the chart domain, trained on a diverse array of chart images and covering a wide range of realworld chart applications.

heading: 2.3 Chart Domain Downstream Tasks
text: Interest in chart-related tasks is rising, focusing on understanding and generating information from charts. Chart question answering (CQA) addresses queries about charts with some datasets (Methani et al., 2020) and (Masry et al., 2022a) focusing on visual and arithmetic reasoning, while others focus on open-ended explanatory question answering (OpenCQA) (Kantharaj et al., 2022b). Additionally, Chart-to-Text involves creating natural language summaries from charts (Shankar et al., 2022), and Chart-to-Table focuses on converting charts into data tables (Choi et al., 2019; Masry et al., 2023). Automated Fact-Checking (AFC) for images, including charts, aims to check claims against data (Akhtar et al., 2023a,b). In this paper, we evaluate our models on these different downstream tasks and also generate new kind of chart reasoning tasks through the instruction data generation process.

heading: 3 Chart Instruction Data Generation
text: We build an instruction tuning dataset for enhancing VLMs’ capabilities in tackling diverse understainding and generation tasks related to chart analysis. In this section, we describe the chart corpus collection followed by the instruction tuning data generation process. Figure 2 provides an overview of the instruction tuning process.

heading: 3.1 Chart Corpora Collection
text: Our goal is to build a diverse chart dataset using real-world data to enhance our model’s generalizability. In order to build that we collect chart images from two main sources: existing public datasets and web-crawled charts. We chose UniChart (Masry et al., 2023) from existing datasets, as it provides one of the largest and most diverse chart pretraining corpora, containing 611K charts with metadata like data tables, titles, and captions (refer to Masry et al. (2023) for details). However, these charts come from a few specific online sources such as Pew (Pew, -), Statista (statista, -), OECD (OCED, -), and OWID (OWID, -), limiting the variety of visual styles and data domains covered.
To address this limitation, we contribute with a new corpus, WebCharts, which contains 41K di-
verse chart images. We started with a seed list of web domains containing charts (Hoque and Agrawala, 2019) and then use the top image search results from these domains using queries such as "chart images," "graphs," and "visual data". We then develop a binary VIT classifier (Dosovitskiy et al., 2021) to distinguish chart images from nonchart images in our search results, followed by manual removal of any remaining non-chart images to refine the dataset. However, these charts lack the underlying data tables which are critical for instruction generation on various chart data analysis tasks. Therefore, we automatically extract the data tables and chart titles using Gemini Pro Vision (Team et al., 2023). The choice of Gemini was influenced by the cost and the unlimited API rate features. More details about the web charts collection process are provided in Appendix A.1.

heading: 3.2 Instruction Data Generation
text: To enhance LLMs’ performance in chart-related tasks via instruction tuning, we develop our chart instruction Dataset. This dataset has 190,774 instructions corresponding to 70,882 charts, covering various aspects of chart comprehension and reasoning (see examples in Figure 1). Below, we describe the process of generating instruction data.
(i) Tasks Selection: To cover diverse aspects of chart reasoning and comprehension, we identify a set of tasks that are similar to some existing downstream tasks such as chart summarization and question answering (QA) but also included other tasks such as code generation and Chain of thought reasoning. Additionally, we prompted LLMs to propose novel tasks to enrich the dataset. Below we briefly explain these tasks. • Summarization and QA The summarization task aims to generate a chart caption that captures the key insights such as trends and patterns from a given chart (Kantharaj et al., 2022c). We also include Open-ended QA (Kantharaj et al., 2022a) in which the model generates an explanatory answer to the given question about a chart. • Fact Checking task (Akhtar et al., 2023a) is included to improve our model’s ability to reduce errors and interpret chart data accurately. It takes a claim as input and then generates the verdict (‘refute’ or ‘accept’) along with an explanation. • Chain-of-thought (CoT) Reasoning aims to enhance the model’s ability to perform complex mathematical and visual reasoning. However, opensource models (e.g., Touvron et al. (2023), Jiang et al. (2023)) often incur errors in numerical computations (Islam et al., 2024a). To address this, we devised two types of questions: (1) Variable Dependent questions, which use tools to compute statistics, inspired by ToolFormer (Schick et al., 2023), and (2) Variable Independent Questions, focusing on retrieval, comparison, and basic math analysis. • Code Generation task is included to generate executable Python scripts to answer user queries, drawing inspiration from the success of this approach demonstrated by PAL (Gao et al., 2023). • Novel Tasks play a crucial role in enhancing the diversity of the instruction set. We tasked an LLM to propose different possible chart-related tasks. To prevent overlap with existing tasks like chart summarization and QA, we instructed the LLM not to replicate these tasks. The generated instructions involve tasks that may require new forms of reasoning and analysis (e.g., future value predictions). (ii) Prompt Design: To create the instructions
for different tasks, we first design a set of prompt templates, where each template contains: (1) task description, (2) the input chart data table, along with metadata such as the chart title, (3) output constraints (if any), (4) output format. An example prompt is shown in Figure 5.
(ii) Input and Output Generation: After designing prompts, we generate instructions, creating input-output instances for each prompt template. We concatenate each chart’s underlying data table and title with one of the prompts designed for the expected task by utilizing OpenAI’s GPT3.5 Turbo and GPT4 (see Table 4). The choice between these APIs for different tasks is dictated by the task complexity, with GPT-4 being employed for complex reasoning tasks, and GPT3.5 for tasks with moderate complexity. Moreover, in order to reduce the generation cost, and increase the variety of generated samples in each call, in our prompts, we ask the LLMs to come up with multiple samples for each chart. The input prompt for each task is provided through Tables 8-14 in Appendix A.10.

heading: 3.3 Dataset Analysis
text: We present key statistics and analyze the diversity and quality of the instruction dataset.
Statistics: Our Chart corpus (WebCharts) is highly diverse, encompassing a variety of bar and line charts, pie and donut charts, and even unconventional chart types not prevalent in existing chart corpora (see Figure 8). The generated instructions set is dominated by Chart summarization and open-ended QA to improve the chart comprehen-
sion ability but also augmented with reasoning tasks and creative new tasks generated by LLM (see Table 1). We placed particular emphasis on WebCharts dataset due to its diversity, constituting 67.5% (157,190 samples) of our dataset.
Diversity: To investigate the diversity of generated instructions, we employed the Berkeley Neural Parser (Berkeley, 2024) to identify the verb closest to the root along with its first direct noun object in each instruction. The analysis reveals a broad spectrum of comprehension and reasoning tasks expressed in the instructions (Figure 3). We further analyzed the newly proposed tasks generated by GPT-4 by clustering the instructions using the Kmeans algorithm. From Table 6 in Appendix A.4, we observe that Pattern and Outlier Detection is the most common type of task, followed by various statistical analyses. Notably, the dataset includes interesting tasks not typically captured by existing downstream tasks, such as identifying correlations, predicting values and trends, and distribution analysis. Overall, it suggests that the generated instructions set is indeed diverse and creative. We also visualize diversity in the length of the instructions’ inputs, and instance outputs in Figure 9 and 10.
Quality: We asked an expert annotator to evaluate the quality of the generated data on a random set of 100 instructions. We find that in general, the instructions describe a valid task (87%) and the input matches the task description (86%) among generated instructions. In 61% and 8% cases, outputs for the generated inputs were fully and partially correct respectively. We list a number of correct and incorrect examples in Figure 11. We note that even when the outputs may be incorrect (e.g., contain factual errors), the corresponding task instructions can provide informative training signal as found by others (e.g., Honovich et al. (2022)).

heading: 4 Modeling
text: In this section, we describe our two architectures for chart instruction tuning.

heading: 4.1 End-to-End System
text: Our end-to-end system utilizes the LLaVA (Liu et al., 2023b) architecture, which incorporates CLIP (Radford et al., 2021) for visual encoding, an LLM for language generation, and an adapter module for transforming the encoded visual features to the LLM’s input embedding space (Figure 4). LLaVA is originally designed for natural image un-
derstanding. We make the following modifications to adapt it for chart understanding. First, we substitute the CLIP vision encoder with the UniChart vision encoder (Masry et al., 2023), which is pretrained and optimized for chart image understanding. For the LLM, we investigate two model types: the decoder-only architecture of Llama2 (Han et al., 2023) and encoder-decoder structure of Flan-T5 (Chung et al., 2022). In the Llama2 setup, projected visual features are injected directly into the language decoder, whereas in the Flan-T5 model, these features, along with the instructions, are first processed by the language encoder before the decoder generates a text. We experimented with both the 7B variant of Llama2 and the 3B variant of Flan-T5 to explore different model sizes suitable for various applications and understand their learning differences (Parvez et al., 2023).
In this end-to-end design, before fine-tuning the model on instructional data, we first fine-tune only the adaptor module keeping the vision encoder and LLM frozen. This critical alignment stage is necessary to align the visual features from the UniChart vision encoder with the input embedding space of the LLM, and enables the LLM to accurately interpret chart images. We focus on two specific tasks for this phase: generating data tables from charts and summarizing chart contents. After alignment, we finetune the model on instruction tuning data, keeping the vision encoder frozen while training the weights of both the adaptor and LLM.

heading: 4.2 Pipeline System
text: In this approach, a data table generation module first converts the chart image into a textual data table representation.This generated data table is then combined with the input instruction and fed into an LLM. We utilize UniChart (Masry et al., 2023), which has been shown to be able to generate highquality data tables from chart images, ensuring the textual representation closely mirrors the original chart’s information. For the LLM, we conduct experiments with both Llama2, and Flan-T5 models similar to our end-to-end approach. Unlike the endto-end system, this setup skips the alignment step since the visual features are not directly fed into the LLM. Hence, we directly finetune the models on the instruction data (see Figure 6).

heading: 5 Experiments and Results
text: We evaluate the usefulness of ChartInstruct and demonstrate that our models built upon ChartInstruct achieve excellence in chart understanding and generation tasks. In addition to the existing downstream tasks, they also posit superior capabilities in new tasks. Below, we first discuss the setups, then experiments on downstream benchmarks and new chart tasks, and finally present an error analysis and challenges. Our evaluation complements the trivial quantitative approach based on automated metrics with detailed human evaluation on both seen and new tasks in multiple aspects, reflecting the true understanding of effectiveness of ChartInstruct . To ensure the reproducability of our research, we present the hyperparameters of our instruction tuning and downstream tasks experiments in Table 5. All experiments were conduced on a 4-A100 GPUs (80GB) machine.

heading: 5.1 Experimental Setup
text: Downstream Tasks: To assess the generalizability of our models across a spectrum of practical chart applications, we evaluate them on four established downstream tasks in the literature: (i) ChartQA (Masry et al., 2022b) – a factoid chart question answering dataset, (ii) OpenCQA (Kantharaj et al., 2022b) – an open-ended chart question answering dataset, (iii) Chart2Text (Shankar et al., 2022) – a chart captioning dataset collected from two sources: Statista (statista, -) and Pew Research Center (Pew, -), and (iv) ChartFC (Akhtar et al., 2023a) – a chart fact checking dataset. Furthermore, we conduct a human evaluation to explore their adaptability in real-world scenarios beyond these benchmarks. Baselines: We compare ChartInstruct against seven open-source baselines: (1) T5 (Raffel et al., 2020), a unified seq2seq Transformer model; (2) VL-T5 (Cho et al., 2021), a T5-based model
for Vision-Language (VL) tasks; (3) VisionTapas (Masry et al., 2022a), an extension of TaPas (Herzig et al., 2020) for chart question answering; (4) ChartBERT (Akhtar et al., 2023b), a BERTbased model utilizing textual and visual information of charts for fact verification; (5) Pix2Struct (Lee et al., 2022), a pretrained image-to-text model; (6) MatCha (Liu et al., 2022b), an adaptation of Pix2Struct for charts pretrained on math reasoning; and (7) UniChart (Masry et al., 2023), achieving SoTA on Chart-to-Text, ChartQA, and OpenCQA. Another open source model is MMC (Liu et al., 2023a), but we exclude it from our evaluation results due to its poor performance. In addition to that, we also compare ChartInstruct against two closed-source LLMs: Gemini Pro (Team et al., 2023) and GPT4-V (OpenAI, 2023). In particular, we report the accuracy on ChartQA from the technical reports while the remaining datasets’ metrics were obtained from (Islam et al., 2024b).
Evaluation Metrics We use Relaxed Accuracy (RA) for ChartQA (following Methani et al. (2020)), Accuracy for ChartFC (Akhtar et al., 2023b) and BLEU for text-generation tasks (Chartto-Text and OpenCQA) (Post, 2018). However, BLEU focuses mainly on n-gram matching, overlooking factors like informativeness and factual
correctness (Goyal et al., 2022). To address this, we conduct human evaluations to compare these aspects (see Section §5.3).

heading: 5.2 Results and Findings
text: We present the experimental results on downstream tasks in Table 2 and compare with existing baselines. ChartInstruct models (both end-to-end and pipeline) outperforms previous open-source stateof-the-art models, UniChart on all ChartQA and Chart-to-Text datasets. In particular, the Flan-T5XL version excels on the ChartQA including the challenging human-written question set (Masry et al., 2022a), which suggests that the model learned more complex mathematical and visual reasoning through the relevant instruction tuning tasks such as CoT reasoning, and coding abilities. ChartInstruct also achieved a higher BLUE score compared to UniChart on OpenCQA benchmark, which demonstrates our model’s capability to generate explanatory answers for questions about charts. Finally, ChartInstruct surpasses ChartBERT by a wide margin (8.85%) on the recently released fact-checking task. Overall, these results establish ChartInstruct as the SoTA open-source model for chart comprehension and reasoning tasks. Compared to closed-source models, ChartInstruct
models outperforms both Gemini Pro (Team et al., 2023) and GPT4-V (OpenAI, 2023) on ChartFC, OpenCQA, and Chart-to-Text Statisa, while our ChartInstruct Flan-T5 achieves comparable performance on ChartQA.
Our observations reveal that the end-to-end system for ChartInstruct-LLama2 generally surpasses the corresponding LLama2 pipeline system across all benchmarks. This performance disparity is likely due to the fact that the data table alone does not capture all the nuanced information present in the charts, thus becoming a limiting factor in the pipeline system’s effectiveness. Similarly, the end-to-end system of ChartInstruct-Flan-T5-XL performs better than the pipeline system on both OpenCQA and Chart-to-Text benchmarks. One notable exception is the reasoning-intensive tasks like ChartFC and ChartQA on which the pipeline FlanT5-XL system exhibits better performance. Furthermore, we notice that both ChartInstruct-Flan-T5XL and ChartInstruct-Llama2 achieve comparable performance, even tho the former has 4B fewer parameters. This efficiency makes ChartInstruct-FlanT5-XL more suitable for real-world applications with computational constraints.
To further assess the impact of our different instruction tuning tasks on our model’s performance, we conducted ablation studies on the ChartQA dataset using our best performing model, ChartInstruct-Flan-T5-XL (Pipeline System). Our ablation studies reveal that excluding tasks like Chart Summarization or Open-ended Question Answering results in a minor decline in performance (Table 7). This performance dip becomes significantly pronounced upon the removal of the reasoning tasks (CoT and Coding), emphasizing their pivotal role in enhancing the model’s reasoning capabilities. More details about the experiments can be found in Appendix A.6.

heading: 5.3 Human Evaluation on Chart Tasks
text: Reference-based evaluation metrics like BLEUscore may not align with human-perceived text quality attributes (Liu et al., 2023c; Smith et al., 2016). To ensure accurate evaluation of our approach, we conducted a human experiment, assessing the generated responses from UniChart and our ChartInstruct-Llama2 model across three metrics: (a) Informativeness, (b) Relevance, and (c) Factual Correctness.
For the study, we chose 150 samples that are un-
seen by both UniChart and ChartInstruct-Llama2. Half of them are randomly from the ChartQA test set, while the other half is from a small set of webcrawled charts not used in the instruction generation pipeline. These samples contain queries from Open-ended QA, Chart Summarization, and novel instruction samples that involve a diverse set of tasks for evaluation. In terms of task distribution, 75 (50%) of the study samples belonged to novel tasks, while the other half comprised Chart-to-Text and OpenCQA tasks (40 and 35 samples). We use UniChart and ChartInstruct-Llama2 to generate responses for these samples. We asked 2 different annotators to rate the sample’s responses based on the mentioned factors from 1-5, having 100 samples in common to measure their agreement level1 toward the responses. We presented the responses randomly to prevent any biases toward models.
From Table 3, we observe that ChartInstructLlama2 significantly outperforms UniChart across all three measures of human evaluation, especially in relevance (4.06 vs. 2.74). Upon manual examination, we observed that UniChart often provides a general summary of the chart without addressing specific task instructions (sometimes repeating the same tokens), particularly evident in novel and OpenCQA task samples. In contrast, ChartInstructLlama2 consistently offers relevant answers for these cases (see an example in Figure 7). Overall, these findings affirm that our instruction-tuning approach enhances the model’s ability to adhere to task instructions, thereby expanding its capacity to address a wide array of new real-world chartrelated scenarios beyond the capabilities of the state-of-the-art pre-trained model for the chart domain.

heading: 5.4 Error Analysis and Challenges
text: We reviewed our model’s results across various samples to highlight the challenging aspects encountered.
1We found Cohen’s Kappa of 0.447 as the agreement level.
Value Estimation and Comparison Charts with crowded or minimal details pose challenges in pairing visual elements (e.g., bars) with their associated values, estimating data values, and making comparisons based on visual attributes (compare based on height). For instance, errors occurred in Q1 and Q2 of Figure 12, where the correct value associated with specified items was not identified. Factual Errors Although our models have shown improved text generation quality and better utilization of available information, they still produce statements unsupported by the chart or factually incorrect. In Q3 of Figure 12, the model produces coherent text but also introduces factual errors. Numerical Reasoning Despite advancements, LLMs sometimes struggle with dependable mathematical operations (Gao et al., 2022; Liu et al., 2022b). While achieving state-of-the-art performance in ChartQA (Masry et al., 2022a) and attempting to teach the model to use external tools, LLMs still exhibit inconsistencies in calculations. Q4 in Figure 12 illustrates the unreliability of LLMs in some numerical reasoning tasks.

heading: 6 Conclusion
text: We present ChartInstruct, an automatically generated dataset of chart-related instructions and two instruction systems designed for a broad range of chart-related tasks. To the best of our knowledge, this is the first instruction tuned dataset that not only includes pre-defined tasks but also many new types of tasks automatically distilled by LLMs. Our model sets the state-of-the-art performance on four different downstream tasks on various automatic measures while the human evaluation further confirms the effectiveness of our approach on many new kinds of tasks. We believe that our models and instruction-tuning dataset will be valuable resources for future research and encourage further exploration into the unique problem domain of chart understanding and reasoning.

heading: Limitations
text: First, while our research covers key tasks such as Chart Summarization, Chart Question-Answering, Open-ended Chart Question-Answering, and Chart Fact Checking, it does not cover other tasks, e.g., Chart-to-table. Second, while our manual inspection of instruction-tuning dataset suggests that the novel tasks distilled by LLM are generally valid and answerable, occasionally the outputs are in-
correct which may influence the instruction-tuning process. Third, although our instruction tuning approach significantly enhances the model’s ability to follow instructions compared to its counterpart, it does not entirely prevent the model from deviating from instructions. Fourth, despite the state-of-theart performance on the numerical reasoning task, ChartQA, our model still struggle with complex numerical questions. Finally, the model may produce factually incorrect statements in the text generation tasks.

heading: Ethics Statement
text: During the dataset collection process, we were mindful of several ethical considerations. The first three sources of our chart corpus ( Statista2, OWID3, OECD4) grant publication rights for academic use of their content. Moreover, the PlotQA dataset (Methani et al., 2020) is a publicly available dataset published under MIT license5. We plan to release the chart images collected from these resources along with their metadata. For the WebCharts corpus, we plan to release only the URLs from which the chart images were collected following relevant large-scale vision-language datasets (e.g., LAION6). Furthermore, we release our models for only academic research purposes.
In our commitment to exclude harmful content from our chart images, we employed Google search for sourcing the chart images, leveraging its strict policies against harmful content7. Moreover, all sourced chart images underwent an initial automatic filtering process using a chart classifier, followed by a manual review phase. Additionally, the WebCharts images were processed through the Gemini API, which is designed to block unsafe content8, thereby providing an additional layer of assurance regarding the appropriateness of the content included in our dataset.
Given the generative nature of our models, there is a potential risk of misuse where users can generate harmful or factually incorrect outputs, poten-
2https://www.statista.com/getting-started/publishingstatista-content-terms-of-use-and-publication-rights
3https://ourworldindata.org/faqs#can-i-use-orreproduce-your-data
4https://www.oecd.org/termsandconditions/ 5https://github.com/NiteshMethani/PlotQA 6https://laion.ai/ 7https://blog.google/products/search/when-and-why-we-
remove-content-google-search-results/ 8https://ai.google.dev/docs/safety_setting_gemini
tially leading to the spread of misinformation. We urge users to exercise responsibility and caution, restricting their use of our models to academic and research purposes only.
The human evaluation was performed by the authors and their collaborators who were involved with this research. As the focus of the research was solely on assessing models’ capabilities, effectiveness, and limitations in several chart understanding tasks, the human evaluation performed by the authors does not add any ethical issues or unwanted biases. We present our instructions to the human evaluators as well as a sample in Figures 14 and 15, respectively. Moreover, there were no paid participants involved in the study. Finally, no information has been used that can directly relate to the identification of any person while evaluating the responses from the models.

heading: Acknowledgement
text: We would like to thank the anonymous reviewers for their helpful feedback. This research was supported by the Natural Sciences Engineering Research Council (NSERC) of Canada and Canada Foundation for Innovation (CFI).

heading: Appendices

heading: A Chart Instruction Data Generation

heading: A.1 Chart Corpora Collection
text: WebCharts Collection: For collecting charts, we conducted image queries on Google and collected chart images from 157 unique source domains. The list of keywords we use are: "chart images," "charts ","graphs," "visual data", and "data visualization". In each query, we included the web domain (e.g., "site: nytimes.com charts") and retrieved the top image search results. We developed a binary VIT classifier (Dosovitskiy et al., 2021) to distinguish chart images from non-chart images in our search results. For training, we manually labeled 1,200 images and split them into 8:1:1 ratios for train, validation, and test sets. The classifier achieved 91% accuracy on the test set. Using this classifier, we filtered out non-chart images from the WebCharts dataset, followed by manual removal of any remaining non-chart images to finalize the dataset. However, these charts lacked the underlying data tables necessary for instruction generation. Therefore, we automatically extracted the data tables and chart titles using Gemini Pro Vision (Team et al., 2023).

heading: A.2 LLMs used for Instruction Generation
text: Table 4 shows the models used to generate the data for different tasks.
A.3 Input Prompts for Instruction Generation Figure 5 shows an example prompt to LLM and the corresponding output for a fact-checking task.
A.4 Instruction Dataset Analysis WebCharts corpus: Figure 8 shows the chart type statistics for WebCharts corpus. Since we do not have access to the chart types in this corpus, we manually tagged random 200 images from it to estimate the chart type distribution.

heading: A.5 Modeling
text: Figure 6 shows the pipeline system architecture we use for our models.

heading: A.6 Ablation Studies
text: To understand the impact of the different instruction tuning tasks on the performance of our model, we conducted ablation studies on the ChartQA dataset using our top performing model, ChartInstruct-Flan-T5-XL (Pipeline System). These ablation experiments involved the removal of one task at a time, except for reasoning tasks, which were grouped and removed together. Due to computational constraints, we finetuned our model on the instruction tuning data for only 1 epoch only, as opposed to the 3 epochs used in the primary experiment. All other hyperparameters remained consistent with those detailed in our main experiments, as outlined in Table 5, including the fine-tuning experiments on the downstream tasks.
As depicted in Table 7, removing tasks like Chart
Summarization and Open-ended Question Answering had a negligible effect on the performance on ChartQA. However, a more significant performance decline was observed upon the exclusion of the fact-checking task, which is important for enhancing the model’s data retrieving and reasoning capabilities. This decline was further amplified when reasoning-associated tasks (CoT and Coding) were removed, underscoring their critical role in improving the numerical reasoning capabilities of our model.

heading: A.7 Human Evaluation Study
text: Figure 7 Shows a comparison between ChartInstruct-Llama2 and UniChart.

heading: A.8 Error Analysis
text: Figure 12 show a few samples which our model faced a challenge generating factual and accurate responses for. In samples Q1 and Q2, our model fails to find the right value for the expected target either by estimation, matching or comparison to other visual elements. In Q3, Although It generates a cohesive summary, it produces some statements that are not true. Q4 shows a numerical error that ChartInstruct-Llama2 didn’t perform the subtraction operation correctly.

heading: A.9 Sample Outputs from ChartInstruct
text: In Figure13, we provide some sample outputs on various tasks.

heading: A.10 Generated Data Samples
text: The input prompt for each specific task is provided through Tables 8-14.

heading: Prompt:

heading: Fact Checking Prompt for GPT-3.5

heading: Prompt:

heading: COT Prompt for GPT-4

heading: Prompt:

heading: OCQA Prompt for GPT-3.5

heading: Prompt:

heading: COT Prompt for GPT-4-preview

heading: Future Work
text: We believe that our models and instruction-tuning dataset will be valuable resources for future research and encourage further exploration into the unique problem domain of chart understanding and reasoning