Venue: NAACL 2024

# MMC: Advancing Multimodal Chart Understanding with Large-scale

Instruction Tuning

Fuxiao Liu\({}^{1,2*}\), Xiaoyang Wang\({}^{2}\), Wenlin Yao\({}^{2}\), Jianshu Chen\({}^{2}\),

**Kaiqiang Song\({}^{2}\), Sangwoo Cho\({}^{2}\), Yaser Yacoob\({}^{1}\), Dong Yu\({}^{2}\) \({}^{1}\)University of Maryland, College Park \({}^{2}\)Tencent AI Lab, Bellevue, USA {fl3es, yaser}@umd.edu, {shawnxywang, wenlinyao, jianshuchen, riversong, swcho, dyu}@global.tencent.com**

This work is done during internship at Tencent AI Lab.

###### Abstract

With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction (**MMC-Instruction**) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop MultiModal Chart Assistant (**MMCA**), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark (**MMC-Benchmark**), a comprehensive human-annotated benchmark with nine distinct tasks evaluating reasoning capabilities over charts. Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts. Code and data are available at [https://github.com/FuxiaoLiu/WMC](https://github.com/FuxiaoLiu/WMC).

+
Footnote †: This work is done during internship at Tencent AI Lab.

## 1 Introduction

Large Language models (LLMs) such as GPT-3, PaLM, ChatGPT, Bard, and LLaMA Brown et al. (2020); Chowdhery et al. (2022); OpenAI, Manyika (2023); Touvron et al. (2023) have undergone rapid development, demonstrating significant capabilities in performing a wide range of tasks effectively. To enable LLMs with vision ability, open-source large multimodal models (LMMs) such as MiniGPT-4 Zhu et al. (2023), LLaVA Liu et al. (2023), mPLUG-Owl Ye et al. (2023), Multimodal-GPT Gong et al. (2023), and LRV Liu et al. (2023b) have been developed, incorporating advanced image understanding capabilities into LLMs to interpret and analyze visual inputs. While successful in the general domains, such open-source LMMs are less effective for chart images because chart understanding differs tremendously from natural scene image understanding. In contrast with natural scene images, which primarily contain objects and reflect their spatial relationships, chart images contain unique abstract elements, including trend lines and color-coded legends that convey specific data-related information.

Current open-source LMMs are limited in their ability to accurately interpret complex chart contents, as they often lack domain-specific training essential for tasks such as differentiating between various types of graphs, interpreting axis labels and data points, and extracting meaningful patterns and trends. Integrating advanced chart understanding capabilities could further refine the LMMs' ability to analyze contextually and reason about the information presented in charts, thereby broadening their applicability in fields like data analytics, academic research, and business intelligence.

In this paper, we introduce MultiModal Chart Instruction (**MMC-Instruction**), a 600k chart understanding dataset consisting of both chart-text alignment data and chart instruction-tuning data. _MMC-Instruction_ is not only much larger but also more diverse compared to existing public datasets Kahou et al. (2017); Masry et al. (2022); Methani et al. (2020); Kafle et al. (2018). Unlike previous work with templated-based questions, _MMC-Instruction_ is constructed by prompting GPT-4 OpenAI, 2023a to generate instructions with diverse language styles and tasks (Tab. 1). Furthermore, our _MMC-Instruction_ considers a variety of chart types, including but not limited to histograms, scatter plots, area charts, and more complex graphical representations. By performing unified instruction tuning

upon current LMMs with _MMC-Instruction_, we further propose a modularized LMM, namely Multimodal Chart Assistant (**MMCA**), jointly finetuned on a wide range of visually situated language understanding tasks. _MMCA_ achieves state-of-the-art performance on current chart question-answer benchmarks compared with existing open-source LMMs.

To accurately assess the capabilities of current Large Multimodal Models (LMMs) for chart understanding, we introduce a novel comprehensive evaluation tool: the MultiModal Chart Benchmark (**MMC-Benchmark**). First, _MMC-Benchmark_ is the first human-annotated benchmark in line with human cognition to evaluate LMM's ability to comprehend visual charts. Second, it contains a wide range of tasks, including _chart information extraction_, _chart reasoning_, _contextual chart understanding_, _chart topic classification_, _stock chart analysis_, _multiple chart understanding_, _chart type classification_, _chart-to-datatable_ and _chart-to-json_. Third, _MMC-Benchmark_ offers two quantitative evaluation methods, including free-format Generation Ability Evaluation using GPT-4 and multiple-choice QA format Chart Understanding Ability Evaluation without the requirement of GPT-4. Our evaluation highlights the limitations of existing open-source LMMs. In addition, we further broaden our analysis through experiments with GPT-4V [19, 20]; Liu et al. [23], which are also used as input for the multi-domain dataset. We then present an overview of our results on the three datasets.

\begin{table}
\begin{tabular}{l c l c c c c} \hline \hline
* & Unbounded & 2 & ✗X \\ \hline
**MMC-Instruction** & 600k & Free-form & Open Ended/MOA & 23.7 & Unbounded & 9 & Human Check \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison between _MMC-Instruction_ with existing **chart** question-answering datasets. _MQA_ means multiple-choice question answering. _MMC-Instruction_ is **larger** and **more diverse**. “Ans.” stands for “Answer”.

Figure 1: Diagram of our human-annotated _MMC_, consisting of **nine** distinct tasks, various topics (_business, health, biology, engineering, etc_), various chart types (_bar, histograms, line, scatter, heatmap, etc_), free-form questions and open-ended answers. More examples are shown in the Appendix.

 

et al., 2023), the latest multimodal version of GPT-4. Our experiments indicate that _MMC-Benchmark_ also poses significant challenges to GPT-4V, especially in _Chart to Datatable_ and _Chart to Json_ tasks. It indicates the importance of _MMC-Instruction_ corpus and _MMC-Benchmark_ in advancing multimodal understanding.

Our main contributions are as follows:

* **MMC-Instruction** dataset. We present a novel large-scale instruction-tuning dataset for chart understanding. It includes diverse topics, language styles, chart types, and open-ended answers in line with human cognition.
* **MMC-Benchmark**. We present a manually annotated benchmark specifically designed to assess the capability of LMMs in chart understanding across nine distinct sub-tasks to ensure a comprehensive evaluation.
* **MMCA** model. We propose an instruction-tuned LMM model that outperforms existing open-source state-of-the-art LMMs for chart understanding on both existing chart understanding benchmarks and our benchmark.

## 2 Related Work

**Multimodal Large Language Model.** Recently, Large Language Models (LLMs) have shown strong performances in zero-shot tasks across multiple domains. Recent studies explore using LLMs for multi-modal task completion. One direction Wu et al. (2023); Yang et al. (2023);a uses ChatGPT as the intermediary to choose the best tools or experts for visual interpretation according to user's inquiries. Another direction is end-to-end training Zhu et al. (2023); Liu et al. (2023); Ye et al. (2023); Yin et al. (2023); Wu et al. (2023); Zhang et al. (2023); Cao et al. (2023); Zhai et al. (2023) utilizing LLMs and visual encoders to create integrated models for multimodal tasks with inter-connected parameters to relate them. These existing approaches perform well on general visual and language tasks like image captioning and visual question answering with strong language skills. However, when it comes to chart understanding, they often fall short due to a lack of specific training to bridge the chart information with the textual content. Our work enhances chart understanding by introducing a novel chart visual instruction-tuning corpus and chart understanding model.

**Chart Text Understanding.** Another line of research Kantharaj et al. (2022); Masry et al. (2023); Lee et al. (2023) is to train a high-resolution image encoder on a large image-text pair corpus to learn text recognition during the pretraining stage. However, these models rely on specific finetuning on different downstream datasets and cannot achieve open-domain multi-task understanding like LLMs or LMMs do. Earlier datasets such as Kahou et al. (2017); Chaudhry et al. (2020); Methani et al. (2020); Masry et al. (2023); Liu et al. (2020, 2023) primarily rely on synthetic data, with template-generated questions and answers selected from a fixed vocabulary. More recently, ChartQA Masry et al. (2022) utilizes real-world, web-crawled charts to develop its visual question-answering datasets, supplemented by human annotators. However, it mainly focuses on compositional and visual questions. Li and Tajbakhsh (2023) uses Palm-2 to generate question-answering data for academic charts. However, the answers generated by Palm-2 contain hallucinations. Comparatively, the advantages of our dataset come from its larger size, more diverse topics, richer language styles, and good quality.

## 3 MMC-Instruction

### Chart-Text Alignment Data

To build a large training corpus for chart-text alignment with a diverse range of styles and topics, we aim to collect chart and text data from online sources. We first collect the _Scientific ChartCaption_ corpus with both chart and text crawled from arXiv. In addition, we filter several existing public datasets that are suitable for chart-text alignment. The collected charts can be categorized into multiple topics, including (_computer science, business, health, biology, agriculture, etc._), and a variety of chart types, including but not limited to (_histograms, scatter plots, area charts, and heatmap)_. More statistic is shown in Tab. 1 and Tab. 2.

_Scientific Chart-Caption_ data collected by us. We first download the academic articles (2010-2020) through an official dump from the arXiv website. It is licensed under CC-0, which grants re-make and republish rights. Unlike Hsu et al. (2021) using PDFs, we utilize the source files containing the original LaTeX and figure files. In order to improve the dataset quality, we removed the source files without LaTeX or figure files and the source files that are hard to parse. We only keep the chart figures with rich text information by deleting the pairs whose caption length is less than 25 tokens. Finally, we collect 210k ch 

_Leveraging Existing Datasets._ For chart-text alignment training with diverse chart caption data, we further include the following five public chart datasets for which the underlying data tables are available: (i) Statista [14], (ii) PlotQA [20], (iii) VisText [35], (iv) ChartInfo [9], (v) Unichart [6]. We randomly picked approximately 190k image-text pairs from these public datasets to increase the diversity.

### Chart Instruction-Tuning Data

This section introduces the construction of our instruction tuning data with 200k instances. To align the model to follow a variety of instructions, we construct diverse instruction-tuning instances about the provided chart images by prompting the language-only GPT-4 [7]. Specifically, given a chart description, we design instructions in a prompt that asks GPT-4 to generate questions and answers in a style as if it could see the image (even though it only has access to the text). The prompt examples for GPT-4 are shown in Fig. 23, 24, 25, 26. Our instruction-tuning format is: "Human: {question} AI: {answer}". _MMC-Instruction_ includes the following tasks: _chart information extraction_, _chart reasoning_, _scientific chart understanding_, _chart-to-datatable_, and _chart-to-json_.

_Chart Information Extraction_ requires the model to extract from the input chart detailed information such as title, coordinate value, scope, etc. To achieve this goal, we collect the generated L1 captions from [35], whose content enumerates aspects of the chart's construction. Then, we ask GPT-4 to generate question-answer pairs about the detailed construction information about the chart given descriptions (Fig. 23). Additionally, we require the generated answers to be less than 20 words to address hallucination.

_Chart Reasoning_ requires the model to analyze and identify data patterns, relationships, and anomalies of the input chart. To achieve this goal, we collect the generated L2/L3 captions from [35], which summarize the statistics and synthesize the cognitive phenomena of the chart. Then, we ask GPT-4 to generate question-answer pairs that require analysis skills in Fig. 24.

_Scientific Chart Understanding_ is a challenging task that needs scientific background knowledge. To create instruction-tuning data, we combine the abstract, title, and image captions of arXiv papers to construct the comprehensive textual context. Sometimes, the image caption is too short for GPT-4 to generate meaningful questions and answers regarding the image. To provide more context regarding the image, we also created a prompt that included paragraphs mentioning the figure in the paper. From our observation, we find a portion

\begin{table}
\begin{tabular}{l c l r r} \hline \hline
**Benchmark** & **Size** & **Images** & **Source** & **Answer** \\ \hline VQA & \textgreater{}1M & General & Annotated & Open \\ GQA & \textgreater{}1M & General & Synthesized & Open \\ MME & 1.5k & General & Annotated & Y/N \\ Lynx-Bench & 0.5k & Video & Annotated & Open \\ MMBench & 3k & General & Repurposed & MOA \\ MM-Vet & 0.2k & General & Repurposed & MQa \\ MathVista & 1.4k & Math & Synthesized & MQa \\ \hline
**MMC-Benchmark** & 2k & ChartPlot & Internet, Annotated & Open/MQA \\ \hline \hline \end{tabular} 
\begin{tabular}{l c} \hline \hline Statistic & Num \\ _ MMC-Instruction_ & 600k \\ _– Scientific Chart-Caption_ & 210k \\ _– Filtered Existing Datasets_ & 190k \\ _– GPT-Generated Instructions_ & 200k \\ _— MAC-Benchmark_ & 2k \\ _ – Unique number of images_ & 1,063 \\ _– Multiple-choice questions_ & 1,275 \\ _– Free-form questions_ & 851 \\ _– Average question length_ & 15.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison between _MMC-Benchmark_ with existing vision-language benchmarks. _MQA_ means multiple-choice question answering. _Repurposed_ means the benchmark is a compilation of prior datasets. _Y/N_ means yes/no questions. _MMC-Benchmark_ is the only existing benchmark with high-quality images for chart understanding.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline
**Tasks** & **Image Source** & **Question Source** & **Question Type** & **Number** & Human Check \\ \hline Chart Information Extraction & Statista.com & GPT-4 & Free-form/MQA & 330 & ✔ \\ Chart Reasoning & Statista.com & GPT-4 & Free-form/MQA & 256 & ✔ \\ Contextual Chart Understanding & arxiv & GPT-4, human & Free-form/MQA & 56 & ✔ \\ Multiple Chart Understanding & arxiv & GPT-4, human & Free-form/MQA & 52 & ✔ \\ Chart Type Classification & Web Crawl & Groundtruth label & Free-form/MQA & 360 & ✔ \\ Chart Topic Classification & Web Crawl & Groundtruth label & Free-form/MQA & 536 & ✔ \\ Chart To DataTable & VisText & Source Article & Free-form/MQA & 400 & ✔ \\ Chart To Json & VisText & GPT-4 & Free-form/MQA & 96 & ✔ \\ Stock Chart Analysis & Google Bard & Source Article & Free-form/MQA & 40 & ✔ \\ \hline \hline \end{tabular}
\end{table}
Table 3: Compositions of _MMC-Benchmark_. The distributions of topics and types are shown in Fig. 6 and Fig. 7.

 

of the questions are not graph-related but a follow-up on the textual context in previous answers. We use heuristic rules to delete the non-chart-related questions. The prompt is shown in Fig. 25.

_Chart-to-DataTable_ and _Chart-to-Json_ are the tasks of transforming the visual information represented in the chart into the structured data format of a table or a JSON. This process typically requires interpreting the graphical elements of the chart, such as bars, lines, or pie segments, quantifying their values, and then organizing these values into a tabular format that accurately reflects the original chart. As shown in Fig. 26, we transform the groundtruth data table from Tang et al. (2023) to create the JSON format into our _MMC-Instruction_.

_Further Quality Control._ We first remove instances with answers longer than 20 words. We remove the instances mentioning unneeded content like "given caption" and "existing descriptions". As for the _Chart-to-Json_ task, we remove the instances without mentioning _"title"_ as the key. To examine the quality of our dataset, we randomly sample 500 instances and ask expert annotators to determine whether the output answers from GPT-4 are correct or not, with regard to the instruction and the image content. We find that 91% of the instructions are appropriate for the image inputs. Furthermore, 85% of outputs are acceptable responses to the instructions. Though some responses may contain errors, most generations conform to the correct structure, serving as applicable instruction-tuning guidelines.

## 4 MMC-Benchmark

The recent progress of LMMs has enabled the open-ended zero-shot completion of user-oriented vision-language tasks such as open-ended chart understanding. As a result, a comprehensive evaluation benchmark is necessary to evaluate the performances of different LMMs on these tasks and provide quantitative guidance for future research and development. However, for chart understanding, existing benchmarks often fall short of evaluating open-ended questions and unbounded chart types. Our dataset, _MMC-Benchmark_, is therefore motivated to bridge this gap, offering three unique characteristics for chart understanding:

(i) _MMC-Benchmark_ is the first benchmark with human annotations to evaluate LMM's ability to comprehend visual charts.

(ii) _MMC-Benchmark_ is more diverse with various sources and nine different tasks, including _chart information extraction_, _chart reasoning_, _contextual chart understanding_, _multiple chart understanding_, _chart type classification_, _chart topic classification_, _chart-to-datatable_, _chart-to-json_, and _stock chart analysis_, with examples shown in Fig. 1.

(iii) _MMC-Benchmark_ provides two evaluation methods for convenient quantitative analysis, including free-format Generation Ability Evaluation using GPT-4 and multiple-choice QA format Chart Understanding Ability Evaluation without the requirement of GPT-4. The statistic of _MMC-Benchmark_ is shown in Tab. 2 and Tab. 3.

### Data Annotation and Quality Control

For _chart information extraction_ and _chart reasoning_ tasks, the images are samples from Masry et al. (2022), but the instruction-answer pairs are all manually constructed by us rather than from existing public annotations. For _contextual chart understanding_ and _multiple chart understanding_, we collect the source images from scientific charts of arXiv that are not presented in our training sets. Contextual chart understanding requires the models to read the context information to answer the questions of the charts. We utilize the abstract of the scientific paper as the context information. We manually design the questions for the multiple chart understanding, which evaluates the model's complex reasoning ability to compare between multiple charts. The _chart type classification_ task contains seven types: line, bar, pie, scatter, heatmap, histogram, and Radar. The images of line, bar, and pie chart are from Methani et al. (2020) while others are collected by us from Google Bard. The _chart topic classification_ task includes health, business, science, travel, biology, engineering, and sports, whose images are crawled from Google. As for the _chart-to-datatable_ and _chart-to-json_ tasks, we use the images and data tables from Wu et al. (2023). The json data is generated by prompting GPT-4 with the datatable as the input. Finally, for _stock chart analysis_, we collect the chart images of stock from Google Bard without including corresponding captions due to hallucination concerns. Instead, we look through the source article and manually construct the questions about the stock trend, predictions, and corresponding background knowledge. We adhere to copyright and license regulations, avoiding data from sites prohibiting copy and redistribution. More examples are shown in Fig. 1, 9, 10, 11, 12, 13, 14, 15, 16. The topic 

and type distributions are shown in Fig. 7 and 6.

### Evaluation Protocols

In order to evaluate LMMs' generation ability and chart understanding ability, the instructions in _MMC-Benchmark_ consist of two parts.

**Generation Ability Evaluation** utilizes GPT-4 "gpt-4-32k-0314" to assess the accuracy of prediction given question and reference answers using prompts shown in Fig. 8. We randomly select 300 samples from our testing set and manually evaluate the model predictions. We find GPT-4 assisted evaluation can achieve 0.90 agreement (Cohen's kappa agreement) with human evaluation.

**Understanding Ability Evaluation (MQA)**, which aims to let the model select the correct answer from multiple-choice questions (MQA) given the chart. For each image, we manually design choices for each question. _Understanding Ability Evaluation_ does not require the utilization of GPT-4. We adopt micro-averaged accuracy as the evaluation metric in Yu et al. (2023) with the help of systematic, rule-based evaluation pipelines.

## 5 MultiModal Chart Assistant (MMCA)

**Architecture.** Our model _MMCA_ (Fig. 2) is built on mPLUG-Owl Ye et al. (2023)) that guides LLMs to follow multimodal instructions. In order to improve the existing LLMs to perform better on chart understanding tasks, we further fine-tune mPLUG-Owl 7B Ye et al. (2023)) on our proposed _MMC-Instruction_ corpus consisting of _Chart-Text Alignment Data_ and _Chart Instruction-Tuning Data_. mPLUG-Owl contains a pre-trained visual foundation model (CLIP vision encoder), a visual abstractor, and a language foundation model (Vicuna). The visual foundation model is responsible for extracting visual features from the input images, and the visual abstractor distills these features using a set of learnable tokens. The resulting visual features are combined with the word embeddings of the input sentence and fed into the language model to generate the response. We incorporate a two-stage training paradigm.

**Stage-1: Chart Text Alignment.** In this stage, we freeze the language decoder and train the visual parts with our _Chart-Text Alignment Data_ for one epoch. This stage enables the mapping of visual features of charts to LLM's word embedding space.

**Stage-2: Chart Instruction Tuning.** In the second stage, we freeze the visual abstractor, visual encoder, and language decoder and use the low-rank adaptation approach (LoRA) Ye et al. (2023) to fine-tune the language model. Specifically, we train the language model with LoRA on our _Chart Instruction-Tuning Data_ for three epochs. This stage enables LLM's instruction following capabilities for chart understanding.

## 6 Experiments

### Experimental Setup

Our model training and inference are conducted with Tesla V100 GPUs. The evaluation is conducted under a zero-shot setting. More implementation details are discussed in the Appendix.

### Baselines

We compare _MMCA_ with existing models in three setups: (a) Open-source LMMs including MiniGPT-v2-7B Chen et al. (2023), mPLUGWl-7B Ye et al. (2023), LRV-Instruction-7B Liu et al. (2023), LLaVA1.5-7B Liu et al. (2023), and Multimodal-GPT-9B Gong et al. (2023). (b) GPT-4V OpenAI (2023) by OpenAI. (c) NonLLMs based models including Pix2Struct Lee et al. (2023) and Donut Kim et al. (2022).

### Experiment Results

#### 6.3.1 Evaluation Results on _Mmc-Benchmark_

As indicated in Tab. 4, Tab. 5 and Tab. 8, _MMCA_ achieves better performance in all nine tasks in comparison with the existing open-source models. The improvement of _MMCA_ demonstrates the effectiveness of our _MMC-Instruction_ data in enabling the LMM to complete chart understanding tasks. In addition, we find that current LMMs are better at understanding cross-modality relationships in the image but weaker at comprehending text layout information. This can be attributed to their lack of text recognition, scientific knowledge, and math reasoning abilities. Though fine

Figure 2: The overall architecture of MMCA, which is continuously trained in two stages.

 

tuned with instruction-tuning data from text-rich images, LLAVA1.5 and mPLUG-Owl do not perform well, indicating that strong text recognition abilities in images do not guarantee high performance on _MMC-Benchmark_, which requires comprehensive visual perception and chart reasoning capability. Additionally, current LMMs perform badly on the _chart-to-datatable_ and _chart-to-json_ tasks. We speculate it is because these two tasks require strong OCR ability to output all the data values in the chart correctly. If one value is missing, the prediction will be regarded as incorrect. Besides, we also find that the overall performance of _Multiple Chart Understanding_ is lower than _Contextual Chart Understanding_. This phenomenon may be attributed to the lack of training data with multiple images as input.

#### 6.3.2 Results on Public Benchmarks

We compare our _MMCA_ with the state-of-the-art methods on existing public benchmarks including ChartQA Masry et al. (2022), DocVQA Mathew et al. (2021), and TextVQA Yang et al. (2021). As shown in Tab. 6, our _MMCA_ outperforms existing LMMs, including MiniGPT4 and LRV-Instruction, on the three public benchmarks. Moreover, without including any finetuning on the corresponding datasets, our proposed _MMCA_ model outperforms both Pix2Struct and Donut that are finetuned on these datasets. Such results validate that with the help of LMMs, chart understanding performance can be significantly improved.

#### 6.3.3 Comparison with GPT-4V(ision)

We further evaluate and benchmark GPT-4V OpnAI (2023b) via the "gpt-4-vision-preview" model on our proposed _MMC-Benchmark_. The quantitative results are shown in Tab. 4 and Tab. 5. Remarkably, GPT-4V surpasses all open-source LMMs by

\begin{table}
\begin{tabular}{l c c c} \hline \hline
* \\ \hline MiniGP-v2 & 49.5 & 61.3 & 50.7 \\ LLaVA1.5 & 52.5 & 66.5 & 58.2 \\ Mplug-Owl & 51.6 & 62.2 & 54.3 \\ \hline
**MMCA (Ours)** & **57.4** & **72.5** & **59.6** \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 6: Comparison with OCR-free methods and LMMs on existing public benchmarks.

\begin{table}
\begin{tabular}{l|c c c c c|c} \hline \hline
**Free-form Evaluation** & **LLAVA1.5** & **MiniGPT-v2** & **mPLUG-Owl** & **LRV-Instruct** & **MMCA (Ours)** & **GPT-4V** \\ \hline Chart Information Extraction & 0.32 & 0.29 & 0.27 & 0.24 & **0.35** & **0.63** \\ Chart Reasoning & 0.30 & 0.23 & 0.22 & 0.19 & **0.30** & **0.57** \\ Contextual Chart Understanding & 0.33 & 0.29 & 0.28 & 0.23 & **0.33** & **0.55** \\ Multiple Chart Understanding & 0.27 & 0.20 & 0.23 & 0.21 & **0.29** & **0.39** \\ Chart Type Classification & 0.30 & 0.27 & 0.25 & 0.22 & **0.31** & **0.79** \\ Chart Topic Classification & 0.31 & 0.23 & 0.24 & 0.21 & **0.32** & **0.82** \\ Stock Chart Analysis & 0.27 & 0.28 & 0.25 & 0.23 & **0.32** & **0.70** \\ Chart to Datatable & 0.00 & 0.00 & 0.05 & 0.00 & **0.08** & **0.05** \\ Chart to Json & 0.01 & 0.00 & 0.00 & 0.00 & **0.05** & **0.04** \\ \hline Overall & 0.24 & 0.21 & 0.20 & 0.17 & **0.26** & **0.51** \\ \hline \hline  & & & & & & \\ \end{tabular}
\end{table}
Table 4: _MMC-Benchmark_ evaluation results of LLaVA1.5, MiniGPT-v2, mPLUG-Owl, LRC-Instruct, MMCA, and the recent GPT-4V regarding the Generation Ability Evaluation. Given the reference response, we apply _GPT-4_ to determine the correctness/incorrectness (as in Fig. 8) of the response for each test sample. The ratio of correct responses out of responses for all test samples in each task is used for evaluation. Tab. 9 shows the sizes of models.

\begin{table}
\begin{tabular}{l|c c c c|c|c} \hline \hline
**MOA Evaluation** & **LLAVA1.5** & **MiniGPT-v2** & **mPLUG-Owl** & **LRV-Instruct** & **MMCA (Ours)** & **GPT-4V** \\ \hline Chart Information Extraction & 0.47 & 0.43 & 0.45 & 0.45 & **0.49** & **0.76** \\ Chart Reasoning & 0.45 & 0.39 & 0.41 & 0.41 & **0.47** & **0.74** \\ Contextual Chart Understanding & 0.49 & 0.51 & 0.50 & 0.42 & **0.55** & **0.79** \\ Multiple Chart Understanding & 0.42 & 0.41 & 0.43 & 0.45 & **0.47** & **0.65** \\ Chart Type Classification & 0.55 & 0.52 & 0.55 & 0.50 & **0.59** & **0.85** \\ Chart Topic Classification & 0.59 & 0.56 & 0.54 & 0.51 & **0.64** & **0.87** \\ Stock Chart Analysis & 0.52 & 0.49 & 0.45 & 0.45 & **0.57** & **0.81** \\ Chart to Datatable & 0.57 & 0.46 & 0.44 & 0.35 & **0.64** & **0.71** \\ Chart to Json & 0.51 & 0.44 & 0.41 & 0.39 & **0.59** & **0.69** \\ \hline Overall & 0.51 & 0.47 & 0.45 & 0.43 & **0.56** & **0.76** \\ \hline \hline  & & & & & & \\ \end{tabular}
\end{table}
Table 5: _MMC-Benchmark_ evaluation results of ILaVA1.5, MiniGPT-v2, mPLUG-Owl, LRC-Instruct, MMCA and the recnet GPT-4V regarding the Understanding Ability Evaluation via _Multichoice QA_ (MQA) task. We calculate the accuracy of the model predictions in the MMA setting. There is no need to call _GPT-4_ for this evaluation.

 

substantial margins on seven out of the nine tasks of _MMC-Benchmark_. Such results prove GPT-4V's strong skills in text recognition, scientific knowledge, and math reasoning. _The only two tasks GPT-4V performs poorly are Chart to Datatable and Chart to Json_. As shown in Fig.20, GPT-4V misrecognizes the data value from the charts. GPT-4V also predicts incorrectly on the _Multiple Charts Understanding_ tasks such as Fig. 18, 19.

We examine 100 randomly sampled error instances from GPT-4V's predictions. The instances are analyzed by expert annotators who identify the root causes. The distribution of errors is in Fig. 4. **Language Bias (35%)**: As indicated in Fig. 3 (right), the strong language prior or parametric memory misleads GPT-4V to answer _"China appears to be the third largest country by land area in the world"_, which conflicts with the information mentioned in the chart _"USA appears to be the third largest country by land area"_. **Perception Error (39%)**: As in Fig. 3 (left) and Fig. 18, the perception error occurs when GPT-4V fails to interpret the chart Liu et al. (2023a). The remaining errors include **Reasoning Error (15%)** in Fig. 19 and **Lack of Knowledge (11%)** in Fig. 21. These errors are attributed to various factors such as complex text interpretation, lack of domain-specific knowledge, or failure to extract answers from long context. More cases are shown in Fig. 20, and 22.

#### 6.3.4 Error Analysis of Open-Source Models Not Following Instructions

Even with a very concise instruction design, there are LMMs that do not follow the user's instructions. For example, in Fig. 27b, when asked _"Please identify the proportion of Americans who favor the coal mining."_, PixSstruct and MiniGPT-v2 answer _"Yes"_ and _"Most Americans favor exporting or expanding solar and wind powers."_, respectively. In our opinion, a good chart understanding model should be able to follow instructions. However, to the best of our knowledge, most of the existing LLM-based or LMM-based models, except for GPT-4V, are not able to follow human instructions well. More examples are shown in Fig. 27a, 27c, and 28.

_Vision Encoder is Weak_. Existing LMMs typically use CLIP as the vision encoder and do not update its parameters during training. However, as CLIP is trained to align visual embeddings with short captions, its capability of modeling the spatial interactions of chart elements like trend lines and color-coded legends is limited. The potential method is to add segmentation Kirillov et al. (2023) and project the segments into the LLM token embedding space. Instead, in our proposed _MMTCA_ approach, we finetune LMMs on our _MMC-Instruction_ data by updating the vision parts during training and improving the integration of visual elements into the LLM input domain. The result improvements prove the effectiveness of _MMC-Instruction_ and the training strategy in _MMTCA_. Fig. 5 shows the distributions of failure causes.

## 7 Conclusion

This paper aims to tackle the challenge of chart understanding with Large Multimodal Models (LMMs). Firstly, we present a large-scale chart

Figure 3: RED highlights incorrect answers while BLUE highlights the correct ones. (Left): the failure of GPT-4V and LLAVA1.5 can be attributed to _vision perception error_. (Right): the failure can be attributed to _language bias_.

 

instruction-tuning dataset _MMC-Instruction_, including diverse topics, language styles, chart types, and open-ended answers in line with human cognition. Secondly, we introduce a human-annotated benchmark called _MMC-Benchmark_ to evaluate LLMs' abilities for chart understanding quantitatively. Finally, we propose an instruction-tuned LMM called _MMCA_ that outperforms existing open-source SoTA methods.

## 8 Limitations

Our study innovatively utilizes a large multimodal model with 7 billion parameters, showcasing substantial capabilities within the constraints of our current computational resources. While we recognize that employing even larger models, such as the 13 billion parameter variants, could further enhance our findings, lacking access to high-end computing resources like A100 limits our current scope. This presents an exciting avenue for future research, where we aim to expand our model's complexity and depth as more advanced computational means become available.

## 9 Ethical Considerations

**Copyright and Licensing**: Strict adherence to copyright and licensing regulations is mandatory. Data from sources that prohibit copying or redistribution will be explicitly avoided. **Data Privacy**: Compliance with privacy laws and ethical standards in data handling is paramount. The annotators should avoid collecting questions that contain any private information.

## References

* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _Advances in neural information processing systems_, 33:1877-1901.
* Cao et al. (2023) Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. 2023. Towards generic anomaly detection and understanding: Large-scale visual-linguistic model (gpt-4v) takes the lead. _arXiv preprint arXiv:2311.02782_.
* Chaudhry et al. (2020) Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. Leaf-qa: Locate, encode & attend for figure question answering. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pages 3512-3521.
* Chen et al. (2023) Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yungyang Xiong, and Mohamed Elhoseiny. 2023a. Minipet-v2: large language model as a unified interface for vision-language multi-task learning. _arXiv preprint arXiv:2310.09478_.
* Chen et al. (2023b) Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023b. Shikra: Unleashing multimodal llm's referential dialogue magic. _arXiv preprint arXiv:2306.15195_.
* Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. _arXiv preprint arXiv:2204.02311_.
* Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. _arXiv preprint arXiv:2305.06500_.
* Gong et al. (2023) Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. 2023. Multimodal-gpt: A vision and language model for dialogue with humans. _arXiv preprint arXiv:2305.04790_.
* Hsu et al. (2021) Ting-Yao Hsu, C Lee Giles, and Ting-Hao'Kenneth' Huang. 2021. Scicap: Generating captions for scientific figures. _arXiv preprint arXiv:2110.11624_.
* Kafle et al. (2018) Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 5648-5656.
* Kahou et al. (2017) Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. _arXiv preprint arXiv:1710.07300_.
* Kantharaj et al. (2022) Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thaskkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. _arXiv preprint arXiv:2203.06486_.
* Kim et al. (2022) Gewook Kim, Teakgyu Hong, Moonbin Yin, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. 2022. Ocr-free document understanding transformer. In _European Conference on Computer Vision_, pages 498-517. Springer.
* Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023. Segment anything. _arXiv preprint arXiv:2304.02643_ 

Jay Lal, Aditya Mitkari, Mahesh Bhosale, and David Doermann. 2023. Lineformer: Line chart data extraction using instance segmentation. In _International Conference on Document Analysis and Recognition_, pages 387-400. Springer.
* Lee et al. (2023) Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In _International Conference on Machine Learning_, pages 18893-18912. PMLR.
* Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstraping language-image pretraining with frozen image encoders and large language models. _arXiv preprint arXiv:2301.12597_.
* Li and Tabakhsh (2023) Shengzhi Li and Nima Tabakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. _arXiv preprint arXiv:2308.03349_.
* Liu et al. (2023a) Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2023a. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. _arXiv preprint arXiv:2310.14566_.
* Liu et al. (2023b) Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacobo, and Lijuan Wang. 2023b. Aligning large multi-modal model with robust instruction tuning. _arXiv preprint arXiv:2306.14565_.
* Liu et al. (2023c) Fukxiao Liu, Hao Tan, and Chris Tensmeyer. 2023c. Documentclip: Linking figures and main body text in reflowed documents. _arXiv preprint arXiv:2306.06306_.
* Liu et al. (2020) Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2020. Visualnews : Benchmark and challenges in entity-aware image captioning.
* Liu et al. (2023d) Haotiian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023d. Improved baselines with visual instruction tuning. _arXiv preprint arXiv:2310.03744_.
* Liu et al. (2023e) Haotiian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023e. Visual instruction tuning. _arXiv preprint arXiv:2304.08485_.
* Manyika (2023) James Manyika. 2023. An overview of bard: an early experiment with generative ai. _AI. Google Static Documents_.
* Masry et al. (2023) Ahmed Masry, Parsa Kavehizadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. _arXiv preprint arXiv:2305.14761_.
* Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. _arXiv preprint arXiv:2203.10244_.
* Mathew et al. (2021) Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_, pages 2200-2209.
* Methani et al. (2020) Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_, pages 1527-1536.
* OpenAI (2022) OpenAI. 2022. Introducing chatgpt.
* OpenAI (2023a) OpenAI. 2023a. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* OpenAI (2023b) OpenAI. 2023b. Gpt-4v(sion) system card.
* Tang et al. (2023) Benny J Tang, Angie Boggust, and Arvind Satyanarayan. 2023. Vistext: A benchmark for semantically rich chart captioning. _arXiv preprint arXiv:2307.05356_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wu et al. (2023a) Chenfei Wu, Shengnming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023a. Visual chatgpt: Talking, drawing and editing with visual foundation models. _arXiv preprint arXiv:2303.04671_.
* Wu et al. (2023b) Shengguong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023b. Next-gpt: Any-to-any multimodal llm. _arXiv preprint arXiv:2309.05519_.
* Yang et al. (2023a) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023a. Gpt4tools: Teaching large language model to use tools via self-instruction. _arXiv preprint arXiv:2305.18752_.
* Yang et al. (2023b) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023b. The dawn of lmms: Preliminary explorations with gpt-4v (vision). _arXiv preprint arXiv:2309.17421_, 9.
* Yang et al. (2023c) Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023c. Mmreact: Prompting chatgpt for multimodal reasoning and action. _arXiv preprint arXiv:2303.11381_.
* Yang et al. (2021) Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. Tap: Text-aware pretraining for text-vqa and text-caption. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 8751-8761 

* Ye et al. (2023) Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. 2023. mplug-owl: Modularization empowers large language models with multimodality. _arXiv preprint arXiv:2304.14178_.
* Yin et al. (2023) Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. _arXiv preprint arXiv:2306.13549_.
* Yu et al. (2023) Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. _arXiv preprint arXiv:2308.02490_.
* Zhai et al. (2023) Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. 2023. Investigating the catastrophic forgetting in multimodal large language models. _arXiv preprint arXiv:2309.10313_.
* Zhang et al. (2023) Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. 2023. Transfer visual prompt generator across llms. _arXiv preprint arXiv:2305.01278_.
* Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. _arXiv preprint arXiv:2304.10592_.

 

Appendix

### MMC-Benchmark

In this section, we discuss more about our _MMC-Benchmark_.

**Generation Ability Evaluation** utilizes GPT-4 to assess the accuracy of the model prediction given the question and reference answers in Fig. 8. Then we ask GPT-4 to assess the prediction accuracy.

**Distriutions of Plot Types and Topics.** Fig. 7 and Fig. 6 present the distributions of chart topic and plot types in _MMC-Benchmark_. Fig. 9, Fig. 10, Fig. 11, Fig. 12, Fig. 13, Fig. 14, Fig. 15 and Fig. 16 show the data examples of different tasks in our _MMC-Benchmark_.

### Experiment

#### a.2.1 More Experiments Results

We further compare _MMCA_ with Donut Kim et al. (2022), BLIP-2 Li et al. (2023), InstructBLIP Dai et al. (2023) and Shikra Chen et al. (2023b). From Tab. 8, we observe that non-LLM based models like Donut work well on the _Chart Information Extraction_ and _Chart Reasoning_ tasks. However, the performance drops a lot when facing other tasks, including _Multiple Chart Understanding_, _Chart Type Classification_, and _Chart to Json_. There could be two reasons. First, the language decoder of non-LLM can not understand the questions correctly. Second, Donut's training set is not diverse enough to cover various topics and plot types. It demonstrates the value of our _MMC-Instruction_.

#### a.2.2 Implementation Details

Our _MMCA_ model is trained with 8 Nvidia Tesla V100 GPUs. Based on the second-stage checkpoint of mPLUG-Owl, we conduct Chart Text Alignment training for one epoch with a batch size of 8. We use the same data augmentation strategy as in BLIP-2 Li et al. (2023), including random resized cropping and horizontal flipping with a probability of 0.5. The number of learnable queries is set to 64. We use the AdamW optimizer. The cosine learning rate decay scheduler is used with a peak learning rate of \(1e^{-4}\) and 1,000 warmup steps. For the learning rate of the vision encoder, we employ layer-wise learning rate decay with a factor of 0.9 to retain the low-level visual representation. For Chart Instruction Turning, we train the language model for three epochs with a learning rate of \(2e^{-5}\) and a batch size of 8.

#### a.2.3 Multiple-Choice Questions Evaluation

For multiple-choice questions, we design systematic, rule-based evaluation pipelines. Specifically, we construct robust regular expressions and develop response-processing workflows to mitigate the potential influence of any intermediate generations (e.g., reasoning steps, calculations) in the long response. These are employed to extract key phrases, such as numbers and conclusion phrases, from the long responses for accurate answer matching. If there is no valid answer in the model's response, we perform random selection as a remedy for multiple-choice questions or consider the response incorrect for open questions.

#### a.2.4 Error Analysis of GPT-4V(ision)

We examine 100 randomly sampled error instances from GPT-4V's predictions. The instances are analyzed by expert annotators who identify the root causes. The distribution of errors is in Fig. 4.

**Language Bias (35%).**_Language Bias_ refers to perceptions formed without relevant visual input. As indicated in Fig. 3 (right), the strong language prior or parametric memory misleads GPT-4V to answer _"China appears to be the third largest country by land area in the world"_, which conflicts with the information mentioned in the chart _"USA appears to be the third largest country by land area"_.

**Perception Error (39%).**_Perception Error_ denotes the misinterpretation of accurate visual information. As depicted in Fig. 3 (left), the perception error occurs when GPT-4V fails to detect the trend in the chart (Fig. 18).

**Other Errors.** The remaining errors include **Reasoning Error (15%)** in Fig. 19 and **Lack of Knowledge (11%)** in Fig. 21. These errors are attributed to various factors, such as complex text interpretation challenges, lack of domain-specific knowledge, or failure to extract precise answers from long context. More cases are shown in Fig. 20 and Fig. 22.

#### a.2.5 Error Analysis of Open-Source Models

We examine 100 randomly sampled error instances from open-source models. The instances are analyzed by expert annotators who identify the root causes. The distribution of errors is in Fig. 5. Different from GPT-4V, one key issue of the open-source model is _Not Following Instructions (27%)_. Even with a very concise instruction design, there are LMMs that do not follow the user's instructions. For example, in Fig. 27b, when asked _"Please iden 

tify the proportion of Americans who favor the coal mining."_, PixsStruct and MiniGPT-v2 answer _"Yes"_ and _"Most Americans favor exporting or expanding solar and wind powers."_, respectively. In our opinion, a good chart understanding model should be able to follow instructions. However, to the best of our knowledge, most of the existing LLM-based or LMM-based models, except for GPT-4V, are not able to follow human instructions well. More examples are shown in Fig. 1(a), 1(c), and 9(d).

Another key issue is _Vision Encoder is Weak (29.6%)_. Existing LMMs typically use CLIP as the vision encoder and do not update its parameters during training. However, as CLIP is trained to align visual embeddings with short captions, its capability of modeling the spatial interactions of chart elements like trend lines and color-coded legends is limited. One potential method is to add segmentation Kirillov et al. (2023) and project the segments into the LLM token embedding space. Instead, in our proposed _MMCA_ approach, we fine-tune LMMs on our _MMC-Instruction_ data by updating the vision parts during training and improving the integration of visual elements into the LLM input domain. As shown in Tab. 7, the model without fine-tuning the vision encoder under-performs our proposed MMCA model. It indicates that fine-tuning the vision encoder part of the model is necessary. The improvements in our experiments also demonstrate the effectiveness of our proposed _MMC-Instruction_ dataset and the training strategy in _MMCA_. Please refer to Fig. 1(a), Fig. 1(b), Fig. 1(c), and Fig. 9 for more examples.

#### a.2.6 More Discussions

_Chart-to-DataTable and Chart-to-Json are extremely Difficult._ As shown in Tab. 4, all current LMMs, including _GPT-4V_, perform badly on these two tasks. It is probably due to the fact that these two tasks require strong OCR skills to output all the data values in the chart correctly. If one value is missing, the prediction will be regarded as incorrect. Compared to the baselines in Fig. 1(a), our _MMCA_ model is able to produce more accurate responses in correct output formats.

_MMC-Benchmark is more Challenging than Previous Benchmarks._ From Tab. 5, we find that the overall scores for existing models on _MMC-Benchmark_ are lower than those on the current benchmarks like ChartQA. Such results are expected since the questions in _MMC-Benchmark_ are more diverse, and the answers are open-ended. Additionally, _MMC-Benchmark_ contains more topics that require both a comprehensive understanding of charts and proficient language skills.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & ChartQA & DocVQA & TextVQA \\ \hline
**MMCA (Ours)** & **57.4** & **72.5** & **59.6** \\ -w/o FT Vision Encoder & 54.2 & 67.8 & 57.2 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Ablation experiments without fine-tuning vision encoder in MMCA.

Figure 4: Error distribution of GPT-4V over 100 randomly sampled error instances.

Figure 5: Error distribution of open-source models over 100 randomly sampled error instances. _Not-F Instruction_ means _’Not Following Instructions”_.

Figure 6: Distributions of chart types in _MMC-Benchmark_.

 



\begin{table}
\begin{tabular}{l|c|c c c|c} \hline
**MQA Evaluation** & **Donut** & **Shikra** & **BLIP2** & **InstructBLIP** & **MMCA (Ours)** \\ \hline Chart Information Extraction & 0.46 & 0.38 & 0.36 & 0.41 & **0.49** \\ Chart Reasoning & 0.42 & 0.39 & 0.38 & 0.40 & **0.47** \\ Contextual Chart Understanding & 0.37 & 0.43 & 0.42 & 0.45 & **0.55** \\ Multiple Chart Understanding & 0.38 & 0.41 & 0.40 & 0.42 & **0.47** \\ Chart Type Classification & 0.42 & 0.48 & 0.50 & 0.52 & **0.59** \\ Chart Topic Classification & 0.45 & 0.56 & 0.51 & 0.55 & **0.64** \\ Stock Chart Analysis & 0.41 & 0.47 & 0.44 & 0.48 & **0.57** \\ Chart to Datatable & 0.32 & 0.39 & 0.40 & 0.41 & **0.64** \\ Chart to Json & 0.38 & 0.41 & 0.39 & 0.48 & **0.59** \\ \hline Overall & 0.51 & 0.47 & 0.42 & 0.45 & **0.56** \\ \hline \end{tabular}
\end{table}
Table 8: _MMC-Benchmark evaluation results on Donut, Shikra, BLIP-2, InstructBLIP, and our MMCA regarding the Understanding Ability Evaluation via Multichoice QA (MQA) task. We calculate the accuracy of the model predictions in the MQA setting. There is no need to call GPT-4 for this evaluation._

Figure 7: Distributions of chart topic in MMC-Benchmark.

\begin{table}
\begin{tabular}{l|c c} \hline \hline
**Method** & **Vision Encoder** & **Language Model** \\ \hline Donut & ViT-g (1.3B) & Bert (0.34B) \\ Pix2Struct & ViT-g (1.3B) & BART (1.3B) \\ BLIP-2 & ViT-g (1.3B) & Vicuna (7B) \\ MiniGPIT-v2 & ViT-g (1.3B) & Vicuna (7B) \\ LLaVA1.5 & ViT-L (0.3B) & Vicuna (7B) \\ mPLUG-Owl & ViT-L (0.3B) & LLaMA (7B) \\ InstructBLIP & ViT-g (1.3B) & Vicuna (7B) \\ LRV-Instruction & ViT-L (0.3B) & Vicuna (7B) \\ \hline
**MMCA (Ours)** & ViT-L (0.3B) & Vicuna (7B) \\ \hline \hline \end{tabular}
\end{table}
Table 9: The backbones for the vision encoder and language model of the baselines and our MMCA model.



* [18] A. B. Krizhevsky, S. Glorassi, and M. Welling. Learning from a large dataset to be used in the training of deep neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 257-260, June 2014.
* [19] R. C. Barnett, J. L. Friedman, D. E. Hinton, and P. H. Zisserman. The output of a generative adversarial network. _arXiv preprint arXiv:1302.1198_, 2013.
* [20] T. Chen, Y. Li, X. Zhang, and Q. Weinberger. Deep residual learning for image recognition. In _Advances in Neural Information Processing Systems (NIPS'09)_, volume 1, pages 1-10, 2009.

[MISSING_PAGE_POST]

 

Figure 12: Examples of the _Chart Topic Classification_ task.

Figure 13: Examples of the _Arxiv Chart Understanding_ task.

 



## Appendix A

Figure 16: Examples of the _Chart Reasoning_ task.

Figure 14: Examples of the _Stock Charts Analysis_ task.

Figure 15: Examples of the _Chart Reasoning_ task.





**Question:**

The stock price is trading within a narrow range, which is a neutral signal. Is it true or false?

_GPT-4V Output:_

In technical analysis, trading within a narrow range indicates that the market is consolidating, and it doesn't give a clear signal about a bullish (positive) or bearts (negative) trend until a breakout occurs above the resistance or below the support level. Therefore, saying that the stock price is trading within a narrow range is a neutral signal is true. It suggests that investors are waiting for more definitive information that could push the stock out of the range in either direction.

Figure 17: GPT-4V works well on _Stock Chart Analysis_ task.

Figure 18: Failure cases of GPT-4V on _Multiple Chart Understanding_ task. RED means the incorrect answers from GPT-4V. **Error category**: _Vision Perception Error_.



Figure 19: Failure cases of GPT-4V on _Multiple Chart Understanding_ task. RED means the incorrect answers from GPT-4V. GREEN denotes the section of the question where the order of choices is switched. **Error category**: _Reasoning Error_.

Figure 20: Failure cases of GPT-4V on _Chart to Json_ task. RED means the incorrect answers from GPT-4V. **Error category**: _Vision Perception Error_.

 



**Question: A 38-year-old man comes to the physician because of recurrent palpitations over the past 2 months. He says that he was diagnosed with a minor heart defect during childhood but does not remember the details and has not had regular medical follow-up examinations since. Physical examination shows a heart murmur. Cardiac catheterization is performed, and hemodynamic results show <image 1>.. Which of the following is the most likely diagnosis? Option: (A) Mktal stenosis (B) Mktral regurgitation. (C) Aortic value reurgurgitation (D) Aortic stenosis

**GPT-4V Output: Based on the provided hemodynamic results: The oxygen saturation of 75% in the right atrium, right ventricle, and pulmonary trunk, followed by a jump to 99% in the left atrium, left ventricle, and ascending aorta, suggests the presence of a left-to-right shunt, where experienced blood is mixing with deoxygenated blood. The pressure readings are also indicative. The normal pressure in the left atrium is usually around 10 mm Hg, while the pressure in the left ventricle during systote (Contraction) can go up to 120 mm Hg or more and during diastole (relaxation) corps significantly. The given pressure for the left ventricles is 165/45 mm Hg, which is significantly high during systolic. Meanwhile, the pressure in the ascending aorta is 155/70 mm Hg; which is closer to normal systolic pressure but has an elevated diastolic pressure. Thus, the most likely diagnosis based on the predicted data is: (B) Mittal regurgitation.**

Figure 21: Failure cases of GPT-4V on _Chart to Json_ task. RED means the incorrect answers from GPT-4V. **Error category: _Lack of Knowledge_**.**

Figure 22: Failure cases of GPT-4V on _Chart to Json_ task. RED means the incorrect answers from GPT-4V. **Error category: _Vision Perception Error_** and _Language Bias Error_**.**



**Prompt: Here is the description of a chart "_Here is a area chart is labeled Cumulative number of patients diagnosed with coronavirus (COVID-19) in Japan as of December 4, 2020, by place of infection. On the x-axis, Month is measured with a categorical scale starting with April and ending with October. There is a linear scale with a minimum of 0 and a maximum of 150,000 along the y-axis, labeled Patients within Japan_ *

Please generate 3 different questions and answers pairs about title, x-axis, y-axis, data range or data pattern of the chart. The answers should come from the descriptions above. Each Answer must be less than 20 words. The output format should be as follows:

question1=> answer1 => question2=> answer2 => question3=> answer3=> question1=> answer1 => question2=> answer2 => question3=> answer3=> ```

**GPT4 OUTPUT Example: Question1=> What does the area chart represent? answer1 => Cumulative COVID-19 cases in Japan by place of infection from April to October 2020. question2=> What does the x-axis represent? answer2 => Months from April to October 2020 question3=> When did the greatest increase in COVID-19 cases in Japan occur? answer3= Between November and December 2020.**

Figure 23: An example prompt for text-only GPT-4 we use to generate instruction and answers for _Chart Information Extraction_ task. The sentence in BLUE is the captions of the chart.





**Prompt:**

Here is the description of a chart "_Displays vary widely m brightness, hue, white balance, contrast and many other parameters that will influence the appearance of light. To affirm this hypothesis, an SLR camera with fixed parameters observes 3 displays and models the CDTF for each one. See Samsung in Fig. 4(a), LG in Fig. 4(b), and iMac 4(c). Although each display is tuned to the same parameters, including contrast and RGB values, each display produces a unique CDTF. Variance of Light Output among Displays. An SLR camera captured a range of grayscale [0,255] intensity values produced by 3 different LCDs. These 3 CDTF curves highlight the dramatic difference in the light emmitance function for different displays, particularly the LG._"

Please generate questions and answers pairs about title, data pattern and purpose of the chart. The answers should come from the descriptions above. The output format should be as follows:

question=> answer >>

Figure 26: An example prompt for text-only GPT-4 we use to generate instruction and answers for _Chart-to-Json_ task. The sentence in BLUE is the captions of the chart.

Figure 25: An example prompt for text-only GPT-4 we use to generate instruction and answers for _Multiple Chart Understanding_ task. The sentence in BLUE is the captions of the chart.



Figure 27: Result examples of our model and other open-source models for three types tasks in _MMC-Benchmark_. RED means incorrect answers, and BLUE means correct answers.

 



## Appendix A

Figure 28: Examples of _Scientific Chart Understanding_ task results from our model and other open-source models in _MMC-Benchmark_. RED means incorrect answers, and BLUE means correct answers.