On Pre-training of Multimodal Language Models Customized for Chart Understanding

## Abstract

Abstract Recent studies customizing Multimodal Large Language Models (MLLMs) for domain-specific tasks have yielded promising results, especially in the field of scientific chart comprehension.
These studies generally utilize visual instruction tuning with specialized datasets to enhance question and answer (QA) accuracy within the chart domain.
However, they often neglect the fundamental discrepancy between natural image-caption pre-training data and digital chart image-QA data, particularly in the models’ capacity to extract underlying numeric values from charts.
This paper tackles this oversight by exploring the training processes necessary to improve MLLMs’ comprehension of charts.
We present three key findings: (1) Incorporating raw data values in alignment pre-training markedly improves comprehension of chart data.
(2) Replacing images with their textual representation randomly during end-to-end fine-tuning transfer the language reasoning capability to chart interpretation skills.
(3) Requiring the model to first extract the underlying chart data and then answer the question in the fine-tuning can further improve the accuracy.
Consequently, we introduce ChopinLLM , an MLLM tailored for in-depth chart comprehension. ChopinLLM effectively interprets various types of charts, including unannotated ones, while maintaining robust reasoning abilities.
Furthermore, we establish a new benchmark to evaluate MLLMs’ understanding of different chart types across various comprehension levels.
Experimental results show that ChopinLLM exhibits strong performance in understanding both annotated and unannotated charts across a wide range of types.

### 1 Introduction

Figure: Figure 1: The underlying data values can be inferred regardless of whether the chart is annotated. However, existing MLLMs rely on annotations and struggle with unannotated charts. In contrast, our model bridges this fundamental discrepancy between natural image-caption pre-training data and digital chart image-QA data, enabling it to extract values regardless of whether the chart is annotated.
Refer to caption: /html/2407.14506/assets/x1.png

In today’s data-driven world, visualizations like bar and pie charts are crucial for deciphering complex datasets.
However, the increasing diversity and complexity of these charts highlights the need for advanced tools to enhance human capabilities in data analysis.
Artificial Intelligence (AI), particularly Multimodal Large Language Models (MLLMs), is increasingly used to automate the understanding of scientific charts, promising more efficient and accurate analysis.
Robust benchmarks are also essential, setting standards and metrics that drive the development and evaluation of these AI tools.

Prior studies have introduced end-to-end neural models aimed at enhancing chart comprehension [, , ], such as masked table prediction [], chart question answering [], and chart de-rendering [].
These models each is specialized in handling one task within the domain of chart analysis.
Furthermore, advancements in Multimodal Large Language Models (MLLMs), exemplified by LLaVA [, ] and miniGPT [], have showcased their versatility in vision-language tasks.
These generalist models undergo a two-stage training process: initially learning visual-language alignment through image-caption pairs, followed by end-to-end fine-tuning using image-QA pairs.
This training not only enables LLMs to interpret visual data but also retains their extensive pre-trained knowledge, which supports their reasoning abilities and leads to strong performance across diverse visual language understanding tasks.

Recent advancements have further ignited interest in tailoring MLLMs to specialized domains such as scientific chart understanding.
Han et al. [], Liu et al. [] have explored collecting instruction-tuned chart data and low-rank adaptation [] to enhance MLLMs’ proficiency with unique chart characteristics, research on the fundamental-training regimes – namely, pre-training to align across modalities and comprehensive end-to-end fine-tuning – for chart-specific understanding remains scarce.
As shown in [Fig. 1](#S1.F1), existing MLLMs often struggle to extract the underlying data from charts when numerical values are not annotated.
We hypothesize that this issue stems from a gap in vision-language alignment between natural image-caption pairs and digital chart-data pairs.
Without targeted pre-training for chart-data alignment, models may resort to relying on a “shortcut” of recognizing numeric annotations through OCR during fine-tuning with QA pairs, rather than truly understanding the visual subtleties of diverse charts.

This paper addresses the above issues by concentrating on the essential training methodologies for MLLMs, including cross-modal feature alignment pre-training and comprehensive end-to-end fine-tuning.
Our research is guided by the question, *“How significantly does fundamental MLLM training influence the enhancement of general MLLMs with chart-specific domain understanding?”*
Our findings indicate that:
(1) Raw data extraction are pivotal in alignment pre-training to bolster chart data comprehension;
(2) Substituting some chart images with purely textual data during end-to-end fine-tuning not only preserves LLM’s text-only reasoning ability but also augments chart interpretation capabilities;
(3) Augmenting QAs with data extraction tasks in the fine-tuning phase allows model to achieve the data prompting during testing, where it first extract data and then answer the QAs, further improving the correctness of its reasoning skills.
Furthermore, existing chart benchmarks are limited in chart and question types.
This prompt our introduction of a comprehensive chart benchmark, which includes 18 chart types and three QA levels to better measure MLLM performance and advance future research in this field.

Our key contributions are summarized as follows:

- •
We introduce ChopinLLM,(^1^11Chart Oriented Pretraining Integration in Large Language Models) a Multimodal Large Language Model tailored for comprehensive chart understanding. This model excels at interpreting various chart types including unannotated ones, underpinned by our detailed analysis and training guidance that emphasizes the importance of foundational training for chart-specific tasks.
- •
We propose a novel data generation pipeline using text-only Large Language Models to efficiently produce large-scale pairwise data. This approach significantly reduces the costs and complexity of data generation for MLLM training.
- •
We establish a robust benchmark that includes a diverse array of chart types and question answering levels, designed to rigorously evaluate MLLMs’ abilities in chart understanding.

### 2 Related works

##### Large language model

Large Language Models (LLMs) have seen remarkable advancements in recent years, primarily driven by transformers [] that have significantly scaled in model size and training data [, , , , , , , ].
These models excel in generalized reasoning and exhibit robust chain-of-thought reasoning [, , ] across a variety of tasks, largely attributed to extensive pre-training [, , ] and fine-tuning strategies [, , ]. The availability of using powerful LLMs with specialized capabilities – ranging from general assistance [, , , ] to coding [, , ] – has fueled diverse applications such as data augmentation [], data generation [, ], and providing training training guidance [, ]. These developments have markedly accelerated research and practical applications in the field.

##### Multimodal large language model

Building on the success of LLMs, recent research has expanded their application to multimodal tasks, including image [, , , ], video [, ], audio or speech [, , ], mixed-modal [], various tool and API usages [, , ], and robotics [, ].
In extending LLMs to image modalities, early studies combined LLMs with external vision models to convert visual information into text, enhancing image comprehension [, ].
Others have integrated visual encoders directly within LLM frameworks, developing end-to-end systems that transform images into textual tokens [, , , , , ].
While maintaining capabilities like reasoning and chain-of-thought processing across various tasks, these models often fall short in domain-specific tasks like chart analysis [, ].
This prompts further research into specialized data collection and fine-tuning for distinct domains.

##### Chart understanding

Current approaches to chart understanding fall into two main categories: models specifically designed for chart-related tasks [, , , , ], and those that utilize pre-trained LLMs and MLLMs [, , , ].
The first group involves models trained exclusively on chart-specific data, often limited by the scope of the training datasets thus cannot be applied to diverse chart scenarios.
The second group, which involves adapting existing LLMs and MLLMs through fine-tuning [] or integration with external models [], shows promising versatility across various questions and scenarios.
Yet, there is a scarcity of research on MLLMs’ pre-training, crucial for deep chart understanding and adaptability to multiple chart types in practical settings.
Typically, chart understanding models are evaluated against benchmarks focused on tasks like data extraction [, ], summarization [], and basic mathematical reasoning [], which predominantly feature basic chart types (*e.g*., bar, line, pie charts) and lack nuanced differentiation in QA levels to thoroughly assess models’ understanding capabilities.
Addressing these gaps, our work not only explores effective pre-training strategies for MLLMs on chart data but also introduces a new benchmark with a variety of chart types and differentiated QA levels (*e.g*., literal, inferential, reasoning) to evaluate MLLMs’ comprehensive abilities.
Concurrently, CharXiv [] is proposed for evaluating general understanding of real-world scientific charts, including complex compositions with multiple subplots.
In contrast, our benchmark focuses on single-plot chart images, evaluating the raw data understanding and mathematical reasoning of an MLLM.

### 3 Generating data for chart understanding

To build a chart understanding MLLM and study its fundamental training process, a comprehensive dataset containing chart images paired with captions and raw data is essential for pre-training, alongside different types of question-answer pairs for end-to-end fine-tuning. However, no existing dataset provides the necessary variety of chart types, topics, and styles. To bridge this gap, we introduce a novel data generation pipeline for large-scale chart data generation ([Sec. 3.1](#S3.SS1)) and QAs generation ([Sec. 3.2](#S3.SS2)). With the data at hand, we then explore various training strategies in the later sections, including feature alignment pre-training and end-to-end fine-tuning for LLMs.
[Figure 2](#S3.F2) presents an overview of our framework.

#### 3.1 Efficient data generation with quadratic scaling

Our data generation leverages the promising text content generation and coding abilities of large language models, *e.g*., GPT-4, to generate chart images and data. Specifically, LLMs allow us to synthesize raw data for chart images, and then the generated Python script turns the raw data into a chart image. In this way, we can produce image data without accessing costly multimodal LLMs like GPT-4V. Unlike previous and concurrent works [, ] that prompt LLMs to iteratively generate CSV data, QAs, and Python script for each chart image – a process that is costly to massively scale – our pipeline features parallel code and data generation through shared templates and READMEs for consistent definitions and formats across the same chart types. Most importantly, since all code script and data share the same structure, our generated data can be universally applied to any generated code and vice versa, significantly enhancing scalability without exhaustedly prompting LLMs. We detail the pipeline further below.

Figure: Figure 2: Overview of (a) the proposed data generation pipeline and (b) Training strategies of ChopinLLM. Generating code and data points conforming to a shared JSON template enables quadratic scaling of the data size (w.r.t. to #GPT calls). The 3-stage training equips our model to grasp the underlying data, thereby achieving a fundamental understanding of charts. ($N$ and $M$ denote the number of generated scripts and data, respectively.)
Refer to caption: /html/2407.14506/assets/x2.png

##### Shared template and README

As shown in [Fig. 2](#S3.F2) (a), given a chart type (*e.g*., line) sampled from a predefined chart type database, the JSON expert GPT-4 first generates a JSON template for the given chart type, along with a README file. In detail, the JSON template contains general information for the chart image, including the title, x-axis, y-axis information, and raw data. The README contains the definition of the chart type and the meanings of the keys and values to enhance understanding of the JSON template. Please refer to [Appendix F](#A6) for some examples. We note that the JSON template, together with the README, ensures the consistency of data generation so that further data and code generation can follow the explicit format and definition guidance of the template data. Note that we choose JSON as our primary data representation format, in contrast to previous works [, , , ], which used CSV. The JSON format allows us to incorporate not only numerical data but also additional chart information, such as titles and the scales of x and y axes, which is beneficial for pair-wise pre-training tasks. Moreover, JSON data is structured, and when paired with a README file, it minimizes ambiguity in data descriptions, which is particularly valuable for complex chart types. For instance, in candlestick charts, we can clearly define a data point as a dictionary containing “open”, “close”, “high”, and “low” values, rather than a list where the meaning of each number might be unclear.

##### Orthogonal data and code generation

With the template files at hand, we can generate data and code independently. For the data generation branch, to ensure the generated data covers diverse topics, we jointly input the produced template files (*i.e*., JSON template and README) and a topic sampled from a pre-defined topic set (e.g., energy production and market share) into a data expert GPT-4 module. For the complete topic list, please refer to [Appendix G](#A7). We require the data expert GPT-4 to follow the definitions in the template files and generate $M$ JSON data along with different kinds of questions and answers (*e.g*., summary QA) based on the raw data. As for code generation, another code expert GPT-4 is utilized to produce $N$ Python code based on the given chart type, data template, and Python library. Note that to prevent generating simple code repeatedly for the given chart type, we explicitly ask the code expert GPT-4 to introduce visual variations in aspects such as color, legend, grid, font, and mark texture, *etc*. More details can be found in the appendix.

Figure: Figure 3: Examples of generated three-level QAs with long and short answers, accessing the understanding of charts from various perspectives. Best viewed in color.
Refer to caption: /html/2407.14506/assets/x3.png

#### 3.2 Diverse QA synthesis

Based on the parallel data generation pipeline, we are able to collect massive amount of chart image and JSON raw data pairs for the feature alignment pre-training.
Now, we details how we generate different types of QAs for end-to-end fine-tuning.
Specifically, having each JSON data as input, we use text-only LLM to generate question-answer (QA) pairs.
To cover various question-anwser for chart data, we include general QAs, containing not only description and summary QA but also three different level of QAs: literal QAs, inferential QAs, and reasoning QAs (as illustrated in [Fig. 3](#S3.F3)).
Furthermore, to enhance the training of chart understanding, we introduce two additional augmented QAs (for training only): text-only QAs and data-driven QAs.
We detail each QA type as follows:

- •
Description QAs: Generate objective descriptions based on the chart data.
- •
Summary QAs: Summarize the chart, highlighting key findings.
- •
Literal QAs: Extract specific values directly from the data.
- •
Inferential QAs: Infer global insights, such as identifying extreme values.
- •
Reasoning QAs: Perform calculations to derive answers from chart data.
- •
JSON-only QAs:
Replace images with JSON raw data to augmented previous QAs.
- •
Data-driven QAs:
Prompt the model to extract JSON raw data before answering the question.

These QAs encompass a range of questions for chart images, covering abilities from basic data understanding and global concept comprehension to advanced reasoning, allowing us to further assess the abilities of MLLMs. Note that, for each QA pair, we use GPT-4 to generate both long and short answers. The long answer, generated first, includes a step-by-step explanation to derive the answer, while the short answer, generated later, contains only the final answer derived from the long explanation. Short answers contain only numerical values or Yes/No response for convenient evaluation purpose. For more examples of generated chart and QAs, please refer to [Appendix J](#A10).

##### Composition for quadratically scaled data

As shown in [Fig. 3](#S3.F3) (a), we consider 18 different chart types. For each chart type, we collect 400 different Python codes ($N=400$) and 1000 different JSON data files ($M=1000$) covering various topics. Note that we exclude bad data based on predicted file structure’s correctness, Python code execution errors, and OCR tools. Please refer to the supplementary materials for detailed information. After filtering, we have approximately 5 million images, with the distribution for each chart type displayed in [Fig. 3](#S3.F3) (a). For each chart image, we collect the raw data in JSON format, a shared README file, the corresponding Python script, $17$ general question-answer (QA) pairs: one description QA, one summary QA, five literal QAs, five inferential QAs, five reasoning QAs, 2 augmented QAs: 1 JSON-only QA, and 1 data-driven QA.

#### 3.3 A new benchmark for comprehensive chart understanding

Existing chart benchmarks [, ] contains only a limited range of chart types (*e.g*., line, bar, and pie charts) and lack of comprehensive QAs to access a model’s understanding of charts from various perspectives, including raw data comprehension, inferential abilities, and mathematical reasoning capabilities.
To bridge this gap, we propose a comprehensive benchmark derived from the aforementioned synthetic dataset.
It covers 18 different chart types, three different levels of QAs (literal, inferential, and reasoning QAs), and provides both long and short answers.
Notably, the chart images in the benchmark are not all annotated, allowing assessment of the model’s ability to understand the underlying data of a chart as humans do.
To ensure the quality of the images in the benchmark, we employed human evaluations to filter the data and obtain a high-quality test set.
The evaluations are based on two criteria: Validity: Whether the essential components of the images (*e.g*., title, x, y-axis, labeling) are clearly visible and not missing.
Extractability: Whether the evaluator can extract the raw data from the given chart image, and whether the extracted data matches the generated one.
After human evaluation and filtering, we obtain a test set consisting of $\approx 300$ image-QA paris for each chart type, resulting in a total of $\approx 5k$ pairs. Note that these QAs equally cover literal, inferential, and reasoning questions for measuring chart understanding of MLLMs.

### 4 Experiments and model analysis

#### 4.1 Experimental setup

##### Benchmark

Our evaluation utilizes four classical benchmarks to compare our model against previous works.
We specifically use the ChartQA dataset [], which includes 1.5k chart images in its test set, divided into human-written and machine-generated questions with 1.2k QA pairs each.
The human-written questions often require mathematical reasoning.
ChartQA also provides CSV data for each image, enabling us to conduct a Chart-to-Table (or Chart Extraction) task to assess the ability of Multimodal Large Language Models (MLLMs) to extract raw data from charts, following previous studies [, ].
Additionally, we use the PlotQA dataset [] where images generally lack numerical value annotations, necessitating value inference relative to the Y-axis.
For evaluating the models’ capability to capture global concepts, we assess on the Chart-to-Text task using the Pew and Statista splits from the dataset [].
The Pew split contains 9k images accompanied by descriptions written by professional editors, while the Statista split includes 33k images that often feature descriptive text within the charts themselves, making it an easier split than Pew.

##### Metrics

For ChartQA and PlotQA, we adopt the relaxed accuracy metric for numeric answers, allowing a 5% margin of error from the exact value, and use exact match for non-numeric answers as per the standard in previous studies [, ].
In the Chart-to-Table task, we measure performance using F1 score of Relative Mapping Similarity (RMS) and Relative Number Set Similarity (RNSS) to evaluate numeric accuracy and table similarity, respectively.
For the Chart-to-Text task, we use BLEU-4, an N-gram matching score, following the previous work [].

##### A 3-stage training process

Unlike previous approaches that convert a general MLLM into a chart-specific expert by only applying LoRA fine-tuning on limited high-quality data [], training ChopinLLM unfolds in three stages, illustrated in [Fig. 2](#S3.F2) (b).
The 3-stage training enables our model not only to understand chart QAs and downstream tasks but also to capture the underlying data, thereby achieving a fundamental understanding of charts.
In the initial pre-training stage, we fix the ViT and LLM while training the projector from scratch using original LLaVA data alongside our newly generated chart-description and chart-json pairs.
The second stage involves freezing ViT and jointly fine-tuning the projector and LLM with both original LLaVA QA pairs and our generated chart QA pairs, enabling the LLM to comprehend visual tokens and facilitate chart question answering.
Finally, we apply LoRA fine-tuning to align the LLM’s response distribution with the target downstream dataset. Each stage is carefully studied and the results are presented in the following subsections.
In the following study, we ablate 1 stage at a time and use the full-training setting for the other 2 stages.

#### 4.2 Stage 1: Pre-training for chart feature alignment

In the first training stage, the goal is to align visual and linguistic features so that visual data can be seamlessly translated into the textual domain for LLM comprehension. Employing a strategy from Liu et al. [], we use a projector to translate visual features from ViT [] into the textual domain, training it with pairwise image-caption data to enhance its capability to capture visual information.
We explore three configurations: utilizing only LLaVA CC3M Pretraining data,(^2^22[https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K](https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K)) combining LLaVA data with chart-description pairs, and using LLaVA data with both chart-description and chart-raw data pairs. The data for stage two training remains consistent across these settings, summary QAs, description QAs, three-level QAs, text-only QAs, and data-driven QAs, as depicted in [Fig. 2](#S3.F2) (b). In stage three, all models undergo LoRA fine-tuning on the downstream dataset, using LLaVA-7B as the baseline for this comparison. Results are detailed in [Table 1](#S4.T1).

##### Dense data alignment is beneficial for both chart data comprehension and reasoning.

For chart images, chart-description pairs act as standard image-caption pairs. However, to more effectively bridge the visual-textual gap, we also utilize chart-json pairs that encompass the underlying numerical data and its schema of the charts. This approach not only aligns visual features with textual descriptions but also significantly enhances model performance, as demonstrated by improvements of approximately 2% in literal QAs and about 1% in reasoning skills, according to results in [Table 1](#S4.T1).

**Table 1: Ablation of stage-1 training. This empirically verifies that pre-training basic chart visual perception is still important, even with abundant stage-2 instruction fine-tuning data. Moreover, learning to predict JSON data is beneficial, even on top of pre-training with descriptive captions.**
| Training data | ChartQA | Our benchmark |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | human | augmented | literal | inferential | reasoning |
| LLaVA-CC3M-Pretrain pairs [] | 44.80 | 83.92 | 41.45 | 34.09 | 22.31 |
| + Chart-description pairs | 48.56 | 86.89 | 42.71 | 33.68 | 23.51 |
| + Chart-JSON data pairs | 52.28 | 87.68 | 44.96 | 34.94 | 24.61 |

**Table 2: Ablation of stage-2 training. Each type of new instruction / QA data improves the final performance consistently across almost all metrics. Best result is highlighted in Bold and the second best is underlined. ^† denotes inference technique without extra data.**
| Training data | ChartQA | Our benchmark |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | human | augmented | literal | inferential | reasoning |
| LLaVA-Instruct-150K QAs | 45.84 | 86.48 | 16.54 | 15.99 | 6.57 |
| + description and summary QAs | 47.04 | 87.76 | 19.90 | 15.69 | 5.26 |
| + Literal / infer. / reasoning QAs | 48.96 | 87.52 | 40.55 | 33.33 | 21.30 |
| + JSON-only QAs | 49.60 | 87.36 | 41.45 | 34.84 | 22.36 |
| + Data-driven QAs | 52.28 | 87.68 | 44.96 | 34.94 | 24.61 |
| + Data Prompting^† | 56.96 | 87.60 | 52.00 | 41.75 | 31.90 |

#### 4.3 Stage 2: End-to-end fine-tuning

The second stage, end-to-end fine-tuning, trains the MLLM to actually understand the aligned visual tokens so that it follows the user instruction and reason about the answer, on top of the inherent language capability from the original LLM.
We utilize a significant number of image-QA pairs to jointly tune the LLM and the projector.
To evaluate the effectiveness of incorporating chart QAs during fine-tuning, we conduct ablation studies starting with a baseline that uses only LLaVA Instruct-150K data,(^3^33[https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K)) incrementally adding extra QA pairs.
All methods leverage the same pre-training weights, derived from training on LLaVA data with both chart-description and chart-raw data pairs (the best setting in [Sec. 4.2](#S4.SS2).
In stage three, all models undergo LoRA fine-tuning on the downstream dataset. [Table 2](#S4.T2) presents the result.

##### JSON-only QAs allow transferring pure text reasoning abilities to multimodal chart understanding.

The chart understanding of MLLMs can be seen as two stages: visual and text raw data alignment (which is done in the training of the first stage) and question answering with reasoning ability on the raw textual data (JSON).
Thus, with a well-aligned first stage training, we hypothesize that re-blending some pure textual QAs, preserving the ability of reasoning on text raw data, can also benefit the reasoning abilities in visual-text scenarios.
As detailed in [Sec. 3.2](#S3.SS2), for JSON-only QAs, rather than utilizing chart images and QAs, we replace the chart image with JSON data and a README, resulting in purely text-based QAs for training.
[Table 2](#S4.T2) demonstrates the effectiveness of each QA type.
We discover that re-blending JSON-only data during the end-to-end fine-tuning stage improves chart reasoning skills, matching the assumption.

##### Data-driven QAs in the fine-tuning stage enable MLLMs to enhance prediction accuracy through data prompting.

As detailed in [Sec. 3.2](#S3.SS2), data-driven QAs are multi-turn QAs, which require models to extract raw data before answering given questions. Combined with the raw data reasoning abilities enhanced via JSON-only QAs, the model can perform data prompting during inference, where models achieve better reasoning robustness by first extracting raw data and then answering the given question based on the data. Please refer to [Appendix I](#A9) for some examples.
As shown in [Table 2](#S4.T2), data-driven QAs significantly enhance the model’s ability to capture visual information.
Furthermore, leveraging data prompting in inference significantly improves performance across all downstream tasks.

**Table 3: Comprehensive evaluation across four chart benchmarks. ChopinLLM achieves best QA results on both (mostly) annotated benchmark, ChartQA, and non-annotated benchmark, PlotQA. H and A denote the human and augmented branch in ChartQA, respectively. Stat. represent the statista split. ^†: our reproduction using the official code. Note that for fair comparison, we don’t use chain-of-reasoning in the inference. The best result is highlighted in Bold and the second underlined.**
| Method | ChartQA | Chart-to-Table | Chart-to-Text | PlotQA^∗ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | H | A | Avg. | F1 | RNSS | Pew | Stat. | v1 | v2 |
| Pix2struct [] | 30.50 | 81.60 | 56.00 | - | - | 10.30 | 38.00 | - | - |
| Matcha [] | 38.20 | 90.20 | 64.20 | - | - | 12.20 | 39.40 | - | - |
| Unichart [] | 43.92 | 88.56 | 66.24 | 52.71 | - | 12.48 | 38.21 | - | - |
| DePlot [] | - | - | - | 87.22 | 94.28 | - | - | - | - |
| LLaVA${}_{\text{7B}}$^† [] | 36.00 | 67.44 | 51.72 | 56.96 | 91.83 | 8.50 | 21.50 | 27.26 | 30.64 |
| LLaVA${}_{\text{13B}}$ | 37.68 | 72.96 | 55.32 | 48.95 | - | 7.16 | 24.65 | - | - |
| LLaVA${}_{\text{13B}}$^† | 42.56 | 73.60 | 58.08 | 63.18 | 93.18 | 8.83 | 22.39 | 27.68 | 30.98 |
| ChartLlama${}_{\text{13B}}$ [] | 48.96 | 90.36 | 69.66 | 89.84 | 94.65 | 14.23 | 40.71 | 29.76 | 29.93 |
| ChopinLLM ${}_{\text{7B}}$ | 52.28 | 87.68 | 69.98 | 83.63 | 95.27 | 11.50 | 38.97 | 30.06 | 31.08 |
| ChopinLLM ${}_{\text{13B}}$ | 54.11 | 88.67 | 71.39 | 88.12 | 95.95 | 12.66 | 40.81 | 33.98 | 33.96 |

#### 4.4 Stage 3: Downstream fine-tuning

We build ChopinLLM with the best setting based on the previous observation (the data used in each stage can be refered to [Fig. 2](#S3.F2) (b)), and we compare ChopinLLM with existing chart understanding approaches, including Pix2struct [], Matcha [], Unichart [], Deplot [], LLaVA [], and ChartLlama []. The results are shown in [Table 3](#S4.T3).

##### Classical question-answering on ChartQA.

We find that ChopinLLM achieves the best performance on the human split of ChartQA, as shown in [Table 3](#S4.T3), with $\approx 5\%$ improvement, while achieving comparable accuracy on the augmented split with the previous state-of-the-art model, ChartLlama, making ChopinLLM the best model in terms of average accuracy. We note that the human split in ChartQA is more challenging than the augmented split, as it contains more reasoning questions, suggesting that ChopinLLM is better at performing reasoning tasks.

##### Raw data and global concept understanding.

As listed in [Table 3](#S4.T3), ChopinLLM achieves the second-best F1 score and the highest RNSS result, indicating that ChopinLLM can capture not only the structure but numerical values of raw data of chart images.
We note that the performance on the chart-to-table task may have been saturated, as the images are mostly annotated.
In this context, this primarily measures the OCR capability and does not assess the ability to capture the underlying data.
As for the Chart-to-Text, shown in [Table 3](#S4.T3), ChopinLLM performs comparable in the global concept capturing and can caption chart image with meaningful texts.

##### Performance on unannotated chart images.

Most of the images in ChartQA [] are annotated, which means the numerical values of data points are explicitly shown on the images.
We observe that existing chart MLLMs, such as ChartLlama [], seem to heavily rely on this annotation for chart understanding, which is not ideal since real-world charts may be unannotated.
We further evaluate them using the PlotQA dataset, and the results are shown in the last column of [Table 3](#S4.T3).
Notably, since training previous models like ChartLlama on PlotQA is infeasible, we load the model weights as used in ChartQA and perform zero-shot prediction on PlotQA.
The results show that our model performs significantly better ($\approx 3\%$ improvement) on unannotated chart images, suggesting that our methods with fundamental training rely less on numerical annotations on chart images.

Figure: Figure 4: Qualitative comparison on Chart-to-text and Chart-to-Table task. ChopinLLM effectively captures the underlying data and generates accurate chart descriptions with correct numerical values. Green denotes correct numbers, while red indicates mistakes. Best viewed in color. Note that the gold table is the noisy ground truth provided in ChartQA dataset.
Refer to caption: /html/2407.14506/assets/x4.png

Figure: Figure 5: Qualitative comparison of multi-turn chart question-answering. Green denotes numbers that match groundtruth number, while red indicates mismatches. Best viewed in color.
Refer to caption: /html/2407.14506/assets/x5.png

#### 4.5 More model analysis

##### Qualitative examples

We provide a qualitative comparison of chart-to-text and chart-to-table tasks, with results depicted in [Fig. 4](#S4.F4).
In the chart-to-table task, our model accurately captures values from chart images, unlike LLaVA and ChartLlama.
It is important to note that the gold data tables for ChartQA are not always directly accessible, leading to the use of existing models or OCR tools for data extraction.
This process can introduce errors, such as misreporting the value $91.5$ for the UK as $915.0,$ which can adversely affect the performance of MLLMs fine-tuned on such data.
Despite these dataset inaccuracies, our model remains robust, correctly outputting values where ChartLlama does not.
In the chart-to-text comparison, both ChartLlama and our model grasp the overall concept of the charts, but our model excels at accurately capturing and summarizing exact numerical values.

Additionally, as a multimodal chatbot, we emphasize preserving human-like multi-turn conversation abilities.
[Figure 5](#S4.F5) presents a qualitative comparison on chart images with multi-turn QAs.
Although ChartLlama extracts accurate numerical values, it fails to provide coherent explanations or reasonable text outputs.
In contrast, ChopinLLM not only extracts accurate data but also provides logical reasoning and coherent explanations, showcasing the effectiveness of our training approach.

##### Performance across different chart types

**Table 4: Performance comparison on different chart types. Overlapped denotes the chart types that are in both the ChartLlama training set and our dataset.**
| Method | Basic | Overlapped |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Line | Bar | Pie | Funnel | Gantt | Heatmap | Scatter | Box | Candle. |
| LLaVA [] | 21.0 | 18.0 | 27.0 | 17.6 | 9.3 | 16.3 | 7.7 | 13.3 | 23.0 |
| ChartLlama [] | 27.6 | 16.7 | 32.7 | 26.0 | 12.0 | 16.7 | 8.0 | 12.0 | 26.3 |
| ChopinLLM | 48.0 | 37.0 | 62.7 | 52.6 | 24.0 | 30.0 | 24.6 | 25.7 | 41.3 |

Our model, trained extensively across a variety of chart types, was evaluated to assess its performance against the previous state-of-the-art model on the same chart types.
For an unbiased comparison, we focused on the short answer format in QA pairs to avoid variations in output preference.
The results, detailed in [Table 4](#S4.T4), reveal that our model consistently outperforms the state-of-the-art across both overlapping and basic chart types.
Notably, our benchmark, which features unannotated images, poses a greater challenge than ChartQA.
The substantial performance improvement indicates that our model is adept at inferring data directly from charts and demonstrates superior reasoning capabilities.

### 5 Conclusion

In this paper, we explore the impact of fundamental training strategies in adapting generalist Multimodal Large Language Models (MLLMs) to chart understanding.
We offer practical guidance for optimizing feature alignment pre-training and end-to-end fine-tuning.
Leveraging these enhanced training strategies, we introduce a specialized chart MLLM, named ChopinLLM, capable of interpreting diverse chart types independently of numerical annotations.
Extensive experiments confirm that ChopinLLM surpasses the previous state-of-the-art across four benchmarks, validating our framework’s effectiveness.
Additionally, we present a new benchmark specifically designed to evaluate MLLMs’ comprehension across various chart types and multiple levels of understanding.

### Acknowledgement

This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs, NSERC CRC, and NSERC DGs. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, the Digital Research Alliance of Canada,(^4^44[alliance.can.ca](alliance.can.ca)) companies sponsoring the Vector Institute,(^5^55[https://vectorinstitute.ai/#partners](https://vectorinstitute.ai/#partners)) and Advanced Research Computing at the University of British Columbia. Additional hardware support was provided by John R. Evans Leaders Fund CFI grant and Compute Canada under the Resource Allocation Competition award.

## 

### Appendix A Details of the dateset generation

#### A.1 Dataset filtering

In Section [3.1](#S3.SS1), we introduce a novel data generation pipeline that leverages text-only LLMs. This pipeline enables us to collect chart images along with various data and QA pairs without extensive human effort, thereby reducing the cost of creating pairwise data. However, LLMs are not perfect and can make mistakes in either data generation or code script generation. Thus, in this section, we discuss the data filtering techniques we use to improve the quality of the synthetic dataset. The generation pipeline is split into three parts: shared template generation, data and QA generation, and code script generation. We now detail the filtering process for each part.

##### Shared template and README

The shared template and README file for each chart type form the core of the entire data generation process, as the subsequent raw data, QA, and Python script are based on the shared template. Therefore, for the shared template, we deploy a human check to ensure the template contains necessary elements for the chart (i.e., title, x-axis, y-axis, data). Additionally, humans are required to verify the correctness of the chart type definitions in the README. Note that we consider 18 different chart types; thus, there are 18 template JSON files along with the READMEs in our dataset.

##### Data and QA generation

Since all the data should follow the template JSON, for the data generation part, we apply filtering based on the JSON structure. Specifically, we remove generated data that deviates from the template file by comparing the elements in the keys of the JSON dictionary and the data types of all the values. As for QA generation, we check the structure of the output dictionary. In detail, the keys of the output QA dictionary should contain summary, description, literal, inferential, and reasoning QAs. We filter out QAs with missing attributes.

##### Code script generation

We predefined a code expert GPT-4 to use four different Python libraries to plot the chart images: Matplotlib, Plotly, Pygal, and Seaborn. The advantage of these libraries is that if the Python code or input data is incorrect in terms of structure or other errors, the generated image will either be missing with a Python error or display a "No Data" icon. Thus, we apply a two-step filtering process: (1) Python Error Filtering: If there is an error while running the Python script to generate the image, we will remove the script and the corresponding JSON data. (2) OCR Tool Filtering: If the image is generated but there is some other error, the output image will display a "No Data" icon on it. To this end, we further use an OCR tool to detect whether there is any "No Data" icon in the images. If so, we will remove the data and script accordingly.

#### A.2 Details of expert GPT-4

In the GPT-4 module, before feeding the prompts for generation, we must specify the system message for GPT-4. These system messages explicitly inform GPT-4 about the environment for the generation and the role it must play in this task, helping the model to output precise responses that match users’ needs. Thus, in our data generation pipeline, we encounter three different circumstances, and we have three different system messages for GPT-4, tailoring it to become an expert accordingly. We provide the details of these system message in Table [5](#A1.T5) for reference.

**Table 5: System messages for GPT-4.**
| Model | System messages |
| --- | --- |
| JSON expert GPT-4 | You are an AI chatbot designed to help users generate a JSON template file along with a README for a specific chart type. Once users specify the chart type, you will have to determine the necessary attributes for plotting a chart, including but not limited to the title, x-axis, y-axis, and data. The data part should have a general structure that covers both simple and complex examples for the chart type. As for the README, it should explain the meaning and type of each attribute or label, and also the definition for this chart type. The generated template and README will then be used in creating raw data and a Python script for visualizing the chart. |
| Data expert GPT-4 | You are an expert in generating question and answer pairs based on raw chart data. Your role involves carefully examining chart data, which is presented in JSON format, and creating relevant question and answer pairs. These pairs will be used for instructional tuning of a vision and language model. Along with the chart data, you will receive a JSON template and a README file that provides additional information on the meaning of each attribute in the JSON data. It is important to review all the provided materials thoroughly to ensure the question and answer pairs are accurate and useful for model training. |
| Code expert GPT-4 | You are a Python code AI assistant and good at chart image plotting. Now, you are asked to modify the Python code to a general version that can take any JSON data matching the definition in the template. |

### Appendix B Implementation details

In this paper, if not specified, we employ the same framework as LLaVA-7B for our model architecture. For details regarding the model architecture, hyperparameters, and the type of optimizer, we adhere to the configuration settings found in the official LLaVA repository.(^6^66LLaVA: [https://github.com/haotian-liu/LLaVA](https://github.com/haotian-liu/LLaVA).) Regarding computational resources for training, we use 8*A100 GPUs for both the pretraining and finetuning stages. Considering the amount of data utilized in these stages, it takes approximately one day for pretraining and four days for finetuning using 8*A100 GPUs.

### Appendix C Experimental results

**Table 6: Performance comparison with synthetic data using in different training stages.**
| Idx | Model | Use syn. data in | ChartQA | Our benchmark |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| human | augmented | literal | inferential | reasoning |  |  |  |
| (1) | LLaVA-7B | - | 36.00 | 67.44 | 14.69 | 18.50 | 7.32 |
| (2) | LLaVA-7B | stage 3 | 37.60 | 70.40 | 27.57 | 29.17 | 12.48 |
| (3) | LLaVA-7B | stage 1 & 2 | 52.28 | 87.68 | 45.91 | 34.74 | 23.01 |

##### Effectiveness of synthetic data in different training stages.

In the previous section, we showcase that chart data generated by text-only LLMs enhances MLLM learning in chart understanding. Here, we compare the performance of using the same data amount in the third stage of LoRA downstream fine-tuning, similar to Chartllama []. The results are in Table [6](#A3.T6). Specifically, Model (1) is the baseline, trained with LLaVA data in pre-training and fine-tuning stages, followed by LoRA fine-tuning on ChartQA. Model (2) is similar but includes synthetic data with ChartQA in the LoRA fine-tuning stage. Model (3), our best 7B model, uses the same synthetic data as Model (2) but incorporates it in pretraining and fine-tuning before LoRA fine-tuning on ChartQA. As shown in Table [6](#A3.T6), Model (2) shows an $\approx 2\%$ improvement over Model (1), indicating the benefit of synthetic data. However, Model (3) shows a significant improvement over Model (2), suggesting synthetic data is more effective in fundamental training rather than fine-tuning. We propose two possible reasons for this. First, alignment issues cannot be effectively resolved via LoRA tuning, as it only adjusts a small portion of the model’s parameters. Second, the output preference of synthetic data may differ from that of the downstream dataset. Joint tuning might shift the output preference away from the downstream task, resulting in limited performance improvement.

### Appendix D Social impact

Our model is capable of chart understanding and can interpret the raw data of a chart like a human, without relying on annotations, while also performing various levels of QA tasks. Thus, our model can be used in many data analysis scenarios, such as market research, healthcare trend analysis, and other data science areas. With the help of our model, humans can process large volumes of chart data more efficiently, make informed decisions, and enhance reporting accuracy. While our model provides benefits in chart understanding and analysis, there are potential negative impacts. For instance, it could be employed to create misleading data visualizations or generate false narratives when combined with other LLM tools. These fake charts and pieces of information can negatively affect decision-making processes.

### Appendix E Limitation

In this paper, we propose an MLLM model for chart understanding, fundamentally trained on synthetic data. However, since the synthetic data generated by LLMs cannot be perfect, sometimes incorrect data can be introduced into the dataset and may not be filtered out by our filtering process. These data can result in misalignments and incorrect mappings during pre-training and fine-tuning, potentially leading to incorrect responses and hallucinations. Thus, the performance of our chart MLLMs is limited by the LLMs’ generation capabilities. We can potentially include more advanced LLMs in the data generation pipeline to reduce the occurrence of incorrect data. Moreover, another limitation of our model is that it currently supports understanding only 18 chart types. However, there are many more chart types in the real world. Developing an open-domain, versatile chart understanding MLLM remains a task for future work.

### Appendix F Examples for JSON template and READMEs

#### F.1 Example 1

[Uncaptioned image]: /html/2407.14506/assets/x6.png

#### F.2 Example 2

[Uncaptioned image]: /html/2407.14506/assets/x7.png

### Appendix G Examples for pre-defined topics

[Uncaptioned image]: /html/2407.14506/assets/x8.png

We provide a word cloud in the figure above to show the frequency of each word in the defined topic set. A comprehensive list of all the topics is also provided at the bottom of this figure.

### Appendix H Examples of augmented QAs

#### H.1 JSON-only QA: example 1

[Uncaptioned image]: /html/2407.14506/assets/x9.png

#### H.2 JSON-only QA: example 2

[Uncaptioned image]: /html/2407.14506/assets/x10.png

#### H.3 Data-driven QA: example 1

[Uncaptioned image]: /html/2407.14506/assets/x11.png

#### H.4 Data-driven QA: example 2

[Uncaptioned image]: /html/2407.14506/assets/x12.png

### Appendix I Examples of data prompting

#### I.1 Example 1

[Uncaptioned image]: /html/2407.14506/assets/x13.png

#### I.2 Example 2

[Uncaptioned image]: /html/2407.14506/assets/x14.png

### Appendix J Examples from our benchmark

#### J.1 Example 1

[Uncaptioned image]: /html/2407.14506/assets/x15.png

#### J.2 Example 2

[Uncaptioned image]: /html/2407.14506/assets/x16.png

#### J.3 Example 3

[Uncaptioned image]: /html/2407.14506/assets/x17.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: In addition to the weaknesses:

1. What is the difference between annotated data and synthetic data that could be the major cause of the performance gap between CHOPINLLM and ChartAst-13B? What challenges exist in create synthetic data in comparable quality? 
2. Can the data generation method be generalized to other domains where annotated data is harder to obtain? Demonstrating this would help justify the advantage of using only synthetic data for training and emphasize its broader applicability.
soundness: 2
strengths: 1. The paper introduces efficient training techniques that significantly enhance chart comprehension.
2. CHOPINLLM, a model for chart understanding, demonstrates strong performance with various chart types.
3. A benchmark is established to evaluate MLLMs' comprehension of different chart types, aiding future research.
4. The data generation pipeline uses text-only Large Language Models to efficiently create diverse datasets, reducing costs and complexity.
summary: The paper introduces a pipeline to create a comprehensive dataset for fine-tuning the proposed MLLM, CHOPINLLM for chart understanding. It highlights that incorporating raw data values during pre-training, substituting images with textual data in fine-tuning, and prioritizing data extraction before answering questions significantly improve performance. Additionally, a benchmark dataset is developed to evaluate MLLMs’ comprehension of various chart types across different complexity levels.
weaknesses: 1. CHOPINLLM did not achieve state-of-the-art (SOTA) performance in Table 4. While the authors claim that higher-performing models benefited from using more data and annotated datasets, there is no evidence showing that the proposed synthetic data offers performance gains when combined with existing datasets. Demonstrating that such a combination improves results would strengthen the contribution of the synthetic data. Otherwise, the benefit of using only synthetic data to build an underperforming model appears limited. (this is my major concern)
2. The paper lacks comparisons with a broader range of SOTA MLLMs that are not specifically tailored for chart understanding, such as InternVL2 and Phi-3.5-V.
3. It omits comparisons with proprietary SOTA models like GPT-4o and Claude-3.5-Sonnet, which would help illustrate performance differences between open-source and proprietary models.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: - Line 300: The reference seems to be wrong (should be section C instead of 3.3?).
- Line 274: The “chart variation” terminology can be misleading without reading additional context e.g., it refers to having multiple styles of chart for the same data instead of the visual diversity of the charts. 
- Line 478: I wonder if a formal ablation is conducted with respect to reliance on numerical annotations. A stronger performance on unannotated chart images does not necessarily indicate that the model doesn’t rely on numerical annotations. There are many possible reasons, such as questions on unannotated chart images tend to be easier, etc. A formal controlled study is warranted for the suggestion that your methods rely less on numerical annotations compared to a well-controlled baseline.
- LLaVA 1.5 only supports resolution up to 336^2. Have you considered the resolution bottleneck for your training experiments and evaluations. Does training on your data become more effective if you scale up the training resolution?
- Line 522: there is one typo.
- Line 342: I interpret Data-driven QAs as the finetuning data for model to generate the JSON before giving the answer, and Data Prompting is a natural language prompt applied during inference time to elicit generation of the JSON before giving the answer. Is my interpretation correct? Does the model generate the JSON when there is no explicit prompting but when there are data-driven QAs?
soundness: 2
strengths: - Investigating ways to improve chart understanding of MLLMs from the “pretraining” (e.g., aligning the connector with captioning data) perspective is rarely explored, which sets this work apart from others that focus on chart understanding in supervised finetuning of the full model on chart QAs. Experiments demonstrate that having a curated chart understanding dataset for pretraining can significantly enhance the model’s performance when later supervised finetuned on the same set of visual QA dataset.
- The paper is clearly written and examples of data and the data curation process are well documented in the supplementary materials.
- The experiments on the effectiveness of different types of chart understanding data are well investigated, where the major contribution factor toward the performance boost is to learn to translate the entire chart into textual data sources and learn to use the pattern for inference.
summary: The paper investigates the design space of chart understanding pretraining of multimodal LLMs along with a fully automatic synthetic data generation pipeline to resemble real-world charts. The resulting model, ChopinLLM, when pretrained on a mixture of LLaVA pretraining data and the synthetic data and fine-tuned on a mixture of LLaVA QAs and the synthetic QAs, achieves competitive performance on its own chart understanding benchmark and decent performance on a variety of other chart understanding benchmarks. The authors well documented the data generation pipeline and mappings between data usage at different stages and model performance.
weaknesses: - A main argument from the paper seems to be that existing models could learn a shortcut that uses chart annotations to analyze the chart and answer questions (L73), while your methods result in a model that has less reliance (L478). Yet, there are no controlled experiments from the paper to support either claim.
- Lack of discussions and/or ablations on the effectiveness of orthogonal data and code generation compared to first generate the data then code. Generating code without knowing the data distribution/patterns limits the variations of the charts and may also create suboptimal layout of the charts. For example, if the data generator chooses to generate data that grows exponentially while the code generator chooses to create the corresponding axis in a linear scale, this can make the plot look awkward and it will also be hard to learn/interpret data from both human/model’s perspective. Some discussion and experiments on these scenarios (and how they could affect training) would be beneficial.
- Cost-effectiveness of data in terms of training is rarely discussed or compared with. While authors proposed a data pipeline that is cost-effective in synthesis, how much a fixed amount of data (or a fixed amount of compute) helps models learn chart understanding is not ablated. For example, when reducing ChartAst’s data to 5M, does model trained on your data perform better? Similarly, you can also reduce the amount of your training data to match the amount in ChartLlama, MMC or ChartInstruct and compare the performance.
- Adding chart-specific data to the pertaining dataset makes chart understanding data over-represented. As most multimodal LLMs tend to be used to solve a diverse range of tasks (i.e., not limited to chart understanding), it is unknown if such data imbalance affects models’ performance on other tasks that require visual perception and reasoning.
- I noticed that the most significant improvement of the performance on your benchmark happens when you add the same types of questions in stage-2 training, yet the performance gain on ChartQA is very small — which could indicate that your literal/inferential/reasoning QAs have a narrow and biased distribution. From a benchmarking perspective, this means that someone can easily gain huge performance boost by scaling up the amount of synthetic data under this distribution (which is easy to scale and can be fully automated as you documented), yet the models’ utility in real-world chart understanding can still remain low. I wonder if authors can provide some discussions on the validity of the numbers reported from your benchmark in terms of real-world chart understanding utility.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The motivation for introducing the benchmark is unclear, as it appears similar in structure and evaluation to MMC without offering additional insights or conclusions.
2. The introduction needs smoother transitions; while the motivation and insights are understandable, it is difficult to follow how the problem is specifically addressed.
3. The authors should better clarify their contributions. While the workload is evident, the innovation is not, making the paper feel more like a technical report. For instance, if one contribution is the training method, does it generalize to other MLLMs like LLaVA, InternXC, Qwen, etc.?
4. After training heavily on chart-specific data, does the model's performance on other MLLM tasks (e.g., those in MME or SEED) degrade? How did the authors balance these aspects? Or is CHOPINLLM solely focused on chart-based QA?
soundness: 3
strengths: 1. The authors present a clear and easy-to-understand workflow.
2. They provide a Chart instruction dataset that includes raw data and QA. The dataset creation process and its characteristics are well explained.
3. The authors offer a comprehensive summary and recommendations regarding MLLM training in the chart domain, particularly on instruction data selection and mixing.
summary: To enhance MLLM's ability to understand charts, the authors propose a process for generating charts and QA data and create a large training dataset. Based on this data, they introduce CHOPINLLM, a fine-tuned LLaVA-like model. Additionally, they propose a benchmark to evaluate the model's performance.
weaknesses: 1. Training aligned with raw data is already widely adopted (e.g., ChartAst, ChartReformer). Similarly, extracting chart data before QA has been explored (e.g., OneChart). 
2. The authors emphasize that their model handles unannotated charts well, but there is no specific design for addressing it. Furthermore, results on unannotated charts are not provided. Benchmarked datasets like PlotQA are overly simple and repetitive, while others such as MMC, ChartBench, and ChartX (all are provided in Table 1) include higher-quality unannotated charts and QA, yet the authors do not report results on them.
3. Although the authors claim their method is MLLM-based fine-tuning, it’s unclear which base model they fine-tuned, making it difficult to evaluate the effectiveness of their data and training approach.
4. The experimental comparisons are insufficient. Some recent works like TinyChart, OneChart, and those mentioned in related works (e.g., ChartGemma) are not included in the comparative tables. Based on the numbers reported in those papers, CHOPINLLM’s results do not appear to be significant.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see my feedback and suggestions above. I think the benchmark for CHART UNDERSTANDING presented in the paper is something that does promote the field, but for the first two contributions in the article, I don't see the obvious significance.
soundness: 3
strengths: **Relevance**: The paper addresses a important task in MLLMs (Chart Understanding). The paper should be of interest that transcends the vision & language community to the broader research community.

**Novelty**: 

- **Innovative Training Techniques**: The paper pioneers a set of training strategies (Three stages), notably using raw data in visual-language alignment, integrating text-only chart representations, and data-first reasoning in Q&A. These approaches contribute to making CHOPINLLM more adept at extracting and interpreting unannotated chart data, a significant advance over existing methods.
- **Data Generation Pipeline**: The paper propose a data generation pipeline, which addresses the challenge of obtaining diverse and high-quality chart data by using automated processes involving language models like GPT-4, which generate both the raw chart data and the Python code to produce chart images. 

**Significance**:  This paper introduces a novel approach to training MLLMs, enabling accurate comprehension and reasoning over complex, unannotated charts, which significantly advances AI's ability to autonomously interpret data visualizations.
summary: This paper proposes a new Multimodal Large Language Model (MLLM), named CHOPINLLM, designed to enhance chart comprehension, especially in scientific and complex data visualizations. The model is tailored to bridge the gap between typical image-caption training data and chart-specific data, aiming to improve MLLM capabilities in extracting underlying numeric values from both annotated and unannotated charts. Furthermore, the paper also introduces a novel data generation pipeline to automaticaly produce large-scale pairwise data about chart understanding tasks. Finally, the paper construct a new benchmark comprising a diverse array of chart types and question-answering levels for robustly evaluate the chart understanding capabilities of MLLMs.
weaknesses: My primary concern about this paper is the performance of CHOPINLLM in chart understanding:

**Baselines**: This paper uses ChartAst [1] as its primary baseline. However, some baselines, such as TinyChart [2], ChartGemma[3], etc are being ignored. After going through and comparing these baselines on Chart QA, I don't find a significant performance advantage with CHOPINLLM. 

**General MLLMs:** I don't get the practical significance of CHOPINLLM, the paper trained an MLLM by proposing a complex THREE STAGES TRAINING STRATEGY. For Stage 1, it's common to add CHART DATA to the image-text pair alignment stage, which has been used by several general MLLMs, e.g., LLama 3[4]. In Stages 2 and 3 (visual instruction tuning), adding chart QA data, cf. the above baselines is common. Therefore, I don't understand the significance of Contribution 1 shown in the Paper. 

**Data Generation Pipeline:** By comparing with TinyChart, the automated pipeline proposed in the paper generates 5M of synthetic data, but the synthetic data generated by TinyChart is about 1M, and there is a big gap between the two in terms of performance on ChartQA (71.39 vs 82.88). This makes it hard to convince me that the data generation pipeline proposed in the paper is more efficient.

Meta-review: The paper introduces CHOPINLLM, a multimodal large language model designed to enhance chart comprehension, particularly for unannotated charts. It presents a data generation pipeline that automatically produces a synthetic dataset tailored for chart understanding tasks and proposes a new benchmark to evaluate performance across diverse chart types and question-answering levels.

However, Reviewers noted a lack of demonstrated performance advantages over existing state-of-the-art models. The paper does not provide sufficient evidence that the proposed synthetic data improves performance when combined with existing datasets. There are also concerns about the model's generalizability beyond chart-specific tasks and its potential over-reliance on chart-specific data. Given these substantial issues related to originality, effectiveness, and broader impact, the paper does not meet the acceptance criteria at this time.

Decision: Reject