ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding

## Abstract

Abstract Automatic chart understanding is crucial for content comprehension and document parsing.
Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in chart understanding through domain-specific alignment and fine-tuning.
However, the application of alignment training within the chart domain is still underexplored.
To address it, we propose ChartMoE, which employs the mixture of expert (MoE) architecture to replace the traditional linear projector to bridge the modality gap.
Specifically, we train multiple linear connectors through distinct alignment tasks, which are utilized as the foundational initialization parameters for different experts.
Additionally, we introduce ChartMoE-Align, a dataset with over 900K chart-table-JSON-code quadruples to conduct three alignment tasks (chart-table/JSON/code).
Combined with the vanilla connector, we initialize different experts in four distinct ways and adopt high-quality knowledge learning to further refine the MoE connector and LLM parameters.
Extensive experiments demonstrate the effectiveness of the MoE connector and our initialization strategy, e.g., ChartMoE improves the accuracy of the previous state-of-the-art from 80.48% to 84.64% on the ChartQA benchmark.

## 1 Introduction

Charts serve as a fundamental tool for data visualization, with automated chart interpretation gaining prominence in domains such as text analysis Hoque et al. (), scientific research Hsu et al. (), and policy-making Wu et al. ().
Chart understanding is a complex task that demands the identification of visual cues, the comprehension of intricate interactions, and the precise inference of values informed by prior knowledge.
Previous work Lee et al. (); Liu et al. (, ) typically pretrained on structured datasets or domain-specific charts to improve the model performance and efficiency.
However, they are constrained by limited resources and narrow task focus, thereby restricting the application in diverse real-world scenarios.
In contrast, MLLMs Li et al. (); Liu et al. (); Bai et al. (); Ye et al. (); Chen et al. (); OpenAI () exhibit substantial potential in image comprehension and instruction following.
With the exponential growth of chart data, automated chart interpretation via MLLMs is emerging as a promising avenue.

The community has achieved advanced progress by creating chart understanding datasets Liu et al. (); Han et al. (); Masry et al. (); Xu et al. () and applying supervised fine-tuning on charts Meng et al. (); Yan et al. (); Carbune et al. ().
Recent studies advocate for alignment pre-training as a foundational step in this paradigm, which only updates the connector’s parameters to bridge the modality gap.
ChartAst Meng et al. () proposes to align chart with table, significantly improving the performance across several chart benchmarks.
ChartReformer Yan et al. () proposes to switch the alignment intermediary from table to information-rich JSON.
Some chart MLLMs, e.g., ChartLlama Han et al. (), employ code as an intermediary to achieve more accurate chart parsing and precise editing.

Figure: Figure 1: Overview and capabilities of ChartMoE: We introduce a MoE architecture connector and provide visualizations of the top-1 expert selection (refer to Fig. 6 and appendix for details). ChartMoE can extract highly precise values and provide flexible chart editing through code-based interactions.
Refer to caption: x1.png

However, few studies have explored how to select appropriate alignment tasks for chart understanding models.
For instance, DocOwl Hu et al. () advocates aligning all modalities with markdown, while Llama-v3.1 Team et al. () warns that excessive markdown data in alignment tasks could be detrimental to model performance.
In the chart domain, table and JSON formats primarily capture attributes like titles, axes, and numerical values but are less effective in conveying visual elements such as color and topology.
This limitation is especially evident in charts with implicit numerical information, such as network diagrams, where comprehension depends on the spatial relationships between visual elements.
In such cases, code effectively captures the relationships between visual marks.
Hence, we aim to embrace all these structured text formats.

We introduce ChartMoE, an instruction-tuned MLLM specifically designed for chart comprehension and reasoning.
As illustrated in Fig. 1&2, we replace the linear connector with the Mixture of Experts Zoph et al. () (MoE) architecture, each expert utilizes a two-layer MLP with GeLU activation Lee ().
Specifically, We train three distinct connectors with different alignment tasks independently, i.e., chart to table, chart to JSON, and chart to code, as shown in Fig. 1.
For better alignment, we construct ChartMoE-Align via existing chart datasets, which consist of over 900K quadruplets {chart, table, JSON, code}, as illustrated in Fig. 3.
Additionally, we retain the original vanilla connector to preserve the base model’s foundational knowledge and conversational abilities.
Consequently, we offer four distinct pathways for initializing the expert parameters.
Compared to random initialization, ChartMoE achieves faster and more stable convergence. Unlike Co-Upcycling initialization Komatsuzaki et al. (), ChartMoE is grounded in a more principled design to prevent experts’ homogenization effectively.

We conduct high-quality knowledge learning using the MMC instruction Liu et al. () to update the routing network, expert connectors, and LLM LoRA parameters while further employing annealing training with datasets like ChartQA Masry et al. () and ChartGemma Masry et al. ().
Following the previous work Zhang et al. (); Masry et al. (), we integrate the Program of Thought (PoT) prompting Chen et al. () to enhance the mathematical computation capabilities. ChartMoE can be deployed for inference on a single A100-40G GPU, achieving state-of-the-art results in areas such as question-answering, number extraction, and chart editing&highlighting. In summary, our contributions are:

- a)
We present ChartMoE for faithful and reasonable chart understanding, with an MoE architecture connector to bridge the visual and LLM branches, enabling differential expert initialization based on various alignment training tasks.
- b)
We introduce ChartMoE-Align, a large-scale dataset with over 900K chart-table-JSON-code quadruplets for chart alignment pretraining.
- c)
We propose to train ChartMoE with a three-stage training paradigm, including alignment pre-training, high-quality knowledge learning, and annealing chart tuning.
- d)
Extensive quantitative and qualitative studies demonstrate that ChartMoE significantly outperforms previous state-of-the-art across several benchmarks by a large margin.

## 2 Related Work

Multimodal large language models leverages a connector to bridge the gap between large language models Touvron et al. (); Radford et al. (); Brown et al. (); Zhang et al. (); Zheng et al. () and vision encoders Radford et al. (); Oquab et al. () to enable enriched capabilities of comprehension and instruction following. Approaches such as BLIP2 Li et al. (), Flamingo Alayrac et al. (), mPLUG-Owl Ye et al. (), and Qwen-VL Bai et al. () utilize QFormers or Resamplers to align modalities on extensive datasets of image-text pairs. LLaVA Liu et al. (, ) is the pioneering work to extend the instruction tuning paradigm to visual tasks with text-only GPT-4 OpenAI (), achieving tremendous performance using a simple MLP without compromising visual information to refine the multimodal alignment. Some works Lin et al. (); Tong et al. (, ) explore the combination of various vision encoders, complementarily enhancing visual representations to bolster the fine-grained visual perception of MLLMs. Despite efforts in structural design, training strategies and data quality remain crucial in the advancement of MLLMs.

Chart Reasoning refers to chart analysis, summarization, and etc. Existing methods can be categorized as 1) Two-stage methods use specialized extraction modules to generate intermediary representations of chart information, like tables, which are provided as textual prompts for LLMs. Pix2Struct Lee et al. () aligns markdown data with charts. MatCha Liu et al. () aligns various data formats (e.g., tables and code) with charts on several downstream tasks. DePlot Liu et al. () fine-tunes Pix2Struct for table extraction and uses LLMs to process queries based on the extracted data. ChartVLM Xia et al. () employs a discriminator to ascertain the necessity of intervention by LLMs for a given query. 2) End-to-end methods strive to tackle chart reasoning challenges with a unified model. ChartLlama Han et al. () incorporates diverse charts and downstream tasks based on LLaVA Liu et al. (). ChartPaLI Carbune et al. (), ChartAst Meng et al. (), and MMC Liu et al. () conduct alignment on table-chart pairs. UReader Ye et al. () aligns all data with markdown, while mPLUG-Owl2 Ye et al. () achieves superior performance with high-resolution inputs. ChartThinker Liu et al. () and DOMINO Wang et al. () propose the CoT Wei et al. () for chart reasoning. LaMenDa Zhuowan et al. () trains MLLMs via step-by-step reasoning QA. ChartReformer Yan et al. () introduces chart-JSON alignment, while OneChart Chen et al. () aligns charts with Python dictionaries. MiniGPT-v2 Chen et al. (), Doc-Owl Hu et al. (), and TinyChart Zhang et al. () tackle the reasoning efficiency for high-resolution charts by merging tokens.

## 3 ChartMoE

### 3.1 Architecture

The ChartMoE is based on InternlmXC-v2 Dong et al. () due to the concise LLaVA-like architecture Liu et al. () and performance on par with GPT-4 on text-image comprehension. The base model includes a vision encoder and a LLM connected by a two-layer MLP. In our ChartMoE  we replace the MLP with an MoE architecture as the connector to leverage diverse prior knowledge.

Vision Encoder.
We utilize CLIP ViT-Large Radford et al. () as the vision encoder, leveraging its rich prior knowledge gained from training on millions of image-text pairs. Considering the impact of chart resolution on performance, we set the input resolution to 490 $\times$ 490 to strike a balance between efficiency and performance. Formally, the visual encoder $\mathcal{M}^{V}(\cdot)$ will project the chart $\mathcal{I}$ into $N$ tokens $V:=\{v_{1},v_{2},\ldots,v_{N}\}$, where $N=1225$ in the ChartMoE.

Mixture-of-Experts Connector.
As illustrated in Fig. 2c, the MoE architecture employs a parallel multi-expert collaboration approach. This architecture comprises $L$ experts $\mathcal{M}^{E}(\cdot)$, each designed with the same linear layer as the baseline. For a visual token $v_{i}$ given by $\mathcal{M}^{V}$, the gating network $\mathcal{M}^{G}(\cdot)$ will calculate the routing weight $g_{j}(v_{i})$ of each expert $\mathcal{M}^{E}_{j}(\cdot)$ and select top-K to activate. Finally, the tokens processed by each expert $\mathcal{M}^{E}_{j}$ will be averaged according to the weight $g_{j}(v_{i})$ given by $\mathcal{M}^{G}$ to get the token $\hat{v}_{i}$ for the LLM branch $\mathcal{M}^{L}$.

Figure: Figure 2: Overview of proposed ChartMoE. (a) Examples of alignment instructions. (b) We conduct three different alignment tasks in parallel. (c) We initialize MoE connectors in four different manners and train the gate network, experts, and LoRA during the supervised fine-tuning stage.

Large Language Model.
Following the baseline, we employ the InternLM2-7B-ChatSFT variant as the LLM $\mathcal{M}^{L}$, implemented as a transformer decoder with a causal attention mask.
We concate the visual tokens $\hat{V}:=\{\hat{v}_{1},\hat{v}_{2},\ldots,\hat{v}_{N}\}$ given by MoE connector with the $M$ input text $\mathcal{T}$ tokens $T:=\{t_{1},t_{2},\ldots,t_{M}\}$ to form the input token sequence for the LLM $\mathcal{M}^{L}$. Formally, give the chart $\mathcal{I}$ and instruction $\mathcal{T}$, the output $\mathcal{O}$ of proposed ChartMoE can be formulated as:

$$ $\{v_{1},v_{2},\ldots,v_{N}\}=\mathcal{M}^{V}(\mathcal{I}),$ (1) $$

$$ $\hat{v_{i}}=\sum_{j}^{L}g_{j}(v_{i})\mathcal{M}^{E}_{j}(v_{i}),\quad g(v_{i})= \mathrm{Top}(\sigma(\mathcal{M}^{G}(v_{i}));K),$ (2) $$

$$ $\mathcal{O}=\mathcal{M}^{L}(\{\hat{v}_{1},\hat{v}_{2},\ldots,\hat{v}_{N};t_{1} ,t_{2},\ldots,t_{M}\}),$ (3) $$

where $\sigma$ indicates softmax and the $\mathrm{Top}(\cdot;K)$ will reset the non-Top K routing weight to 0.

Initialization of Expert.
Previous approaches initialize expert parameters via 1) Random initialization, which may lead to convergence difficulties during supervised fine-tuning, and 2) Co-upcycling initialization Komatsuzaki et al. (), i.e., copy baseline connector parameters to each expert, which may lead to homogenization of experts. ChartMoE introduces a more principled initialization strategy, enabling experts to effectively specialize different components of charts and enhance overall performance over the baseline.

### 3.2 Alignment Pretraining.

Unlike previous work, we obtain the initialization parameters for the various experts during the different stages of alignment pre-training. Specifically, as illustrated in Fig. 2b, we align expert connectors using three distinct alignment tasks, where only the connector parameters will be updated.

Alignment with Table.
ChartAst Meng et al. () first proposes performing chart alignment before supervised fine-tuning, effectively enhancing the model’s ability in numerical extraction and chart understanding. We align charts and tables (in CSV format, shown in Fig. 2a) to improve the model’s capability to extract tabular data.

Alignment with JSON.
ChartReformer Yan et al. () proposes using JSON instead of table data for alignment pre-training. JSON contains not only the values of the table but also drawing properties of the chart, such as font format, line type, color style, etc. This task is more difficult than chart-to-table, contributing to attribute-based chart editing and highlighting.

Alignment with Code.
Chart editing is primarily achieved by reverse engineering the plotting code and re-rendering the chart Han et al. (); Xia et al. (). Therefore, we propose the chart-to-code task. Specifically, the code contains precise data and plotting attributes, such as numerical values represented by Python lists and colors represented by hexadecimal codes, as shown in Fig. 2a. This task is more information-rich compared to JSON and Table alignment.

Fig. 3 illustrates how we generate the aligned data in ChartMoE-Align. 1) We filter charts with meta CSV from existing datasets Masry et al. (); Methani et al. () and data generated by LLMs Chen et al. (). 2) We use a fine-tuned Deplot Liu et al. () to inverse the plotting attributes following the templates provided by ChartReformer and randomly sample missing attributes from the predefined set. 3) We create code templates for different types of charts and generate plotting code based on the meta CSV and extracted JSON attributes. Note that all values and attributes in the code are explicitly represented. 4) We retain the (table, JSON, code, chart) quadruples that pass Python compilation. Tab. 1 shows the data sources and size in detail.

### 3.3 Supervised Finetuning.

We initialize ChartMoE using the structure shown in Fig. 2c after aligning the connectors across 3 distinct tasks separately. We also retain the vanilla connector to maintain the baseline’s excellent dialogue capabilities, which aligns with the principle of residual optimization He et al. (). We train the MoE connector and LLM during this stage with LoRA Hu et al. (), as shown in Fig. 2c. Considering the training principles proposed in LLaVA-NeXT Liu et al. (), this stage is divided into high-quality knowledge learning and chart-specific annealing training.

Figure: Figure 3: Overview of ChartMoE-Align data generation pipeline. The charts are plotted by Python matplotlib.

**Table 1: Datasets used for training ChartMoE. We conduct alignment pre-training with synthetic data and supervised tuning with high-quality, real-world data. Only ChartQA is used in the ablation due to GPU constraints.**
| Source | Data Type | Task Type | Samples |
| --- | --- | --- | --- |
| Alignment Training |  |  |  |
| ChartQA | synthetic | chart to table | 18.3K |
| chart to JSON | 18.3K |  |  |
| chart to code | 18.3K |  |  |
| PlotQA | synthetic | chart to table | 157K |
| chart to JSON | 157K |  |  |
| chart to code | 157K |  |  |
| ChartY | synthetic | chart to table | 763.6K |
| chart to JSON | 763.6K |  |  |
| chart to code | 763.6K |  |  |
| Total | 2.8M |  |  |
| Usage: Table = 500K JSON = 200K Code = 100K | 800K |  |  |
| High-Quality Knowledge Learning |  |  |  |
| MMC | synthetic | QA & reasoning | 410K |
| & real world | & summariztion |  |  |
| Chart Specific Annealing Tuning |  |  |  |
| ChartQA | real world | QA | 28.3K$\times$2 |
| ChartGemma | real world | QA & PoT & reasoning | 163.2K |
| & summariztion |  |  |  |
| Total | 220.8K |  |  |

High-Quality Knowledge Learning
We adopt MMC Liu et al. () to enhance the ChartMoE’s knowledge. MMC includes a variety of chart types and tasks such as chart-related question answering, translation, extraction, reasoning, and analysis. Considering data quality, we only utilize the MMC-Instruction subset, which has been manually verified. Notice that the quality of instruction data is more important than quantity in this stage.

Chart Specific Annealing Tuning
Following Llama-v3.1 Team et al. (), we perform annealing tuning before evaluating mainstream benchmarks. We increase the learning rate and conduct instruction tuning using the training sets of ChartQA Masry et al. () and ChartGemma Masry et al. () to adjust the query styles and answer formats of these benchmarks.

PoT Chen et al. () requires the MLLM to generate the variables and operation code rather than producing direct answers. This inference pipeline addresses the mathematical capabilities by employing Python to handle the logical computations, which is the shortcoming of all open-sourced models. With better numerical extraction abilities, PoT can significantly enhance our ChartMoE’s question-answering performance.

## 4 Experiment

### 4.1 Implementation Details

The ChartMoE builds on InternlmXC-v2 Dong et al. (), integrating ViT-Large Radford et al. () as the visual encoder and InternLM-v2 Cai et al. () as the language model. We replace the MLP with MoE architecture as the connector. During the alignment stage, we train the connector parameters and keep the visual encoder and LLM parameters fixed for 1 epoch. The learning rate is set to 5e-5 with a warmup phase covering the first 1% of training steps. In the supervised fine-tuning stage, we continue training the connector while employing LoRA to update the LLM parameters with the rank of 64. The learning rate is adjusted to 1e-5 for the high-quality knowledge learning period and 5e-5 for the chart-specific annealing tuning period. The weight decay is 0.1 for all stages. We use the cosine annealing learning rate schedule. The global batch size is set to 64 for all stages. The training process is conducted on A100-40G GPU, with the alignment stage taking approximately 240 GPU hours, the knowledge learning stage taking 138 GPU hours, and the chart-specific annealing tuning stage taking around 76 GPU hours.

### 4.2 Evaluation Metrics

ChartQA Masry et al. () test split consists of 1,250 questions in both human and augmented parts. The charts are three common chart types and are sourced from the real world. It features a variety of human-crafted questions and answers to evaluate models’ understanding, reasoning, and data extraction skills.
ChartQA adopts relaxed accuracy, which is highlighted shortcomings by recent studies Chen et al. (); Xu et al. (), such as simplistic string matching and direct float conversion. Therefore, we improve it by 1) using regular expression matching to extract number values, 2) optimizing string matching for short answers, and 3) demonstrating model performance under various relaxed margins. We adopt it for all experiment results in this paper.

ChartBench Xu et al. () focuses on charts without data point annotations. It includes a broader range of chart types, with 9 main categories and 42 subcategories, each containing 50 charts. ChartBench focuses on extracting numerical values, posing a greater challenge as models cannot depend on OCR for precise answers. It adopts Acc+ for judgments and relaxed accuracy for NQA tasks. The benchmark proposes to extract number values by LLMs first, which is omitted for the stratifying instruction-following ability of ChartMoE.

ChartFC Akhtar et al. () & ChartCheck Akhtar et al. () They use accuracy to verify whether the claim aligns with the input chart, marking a significant advancement in chart recognition and reasoning abilities. This also identifies the potential hallucinations in chart-related contexts. The ChartFC test set has 1,591 questions, and the ChartCheck test set has two splits, containing 937 questions and 981 questions.

### 4.3 Comparative Models

General MLLMs.
We compare PaliGemma Beyer et al. (), LLaVA-v1.5 Liu et al. () with an MLP connector, Qwen-VL Bai et al. () with a Qformer Li et al. () connector, DocOwl-v1.5 Hu et al. () that employs multi-level image resolution and token convolution techniques, and the current open-source SOTA, InternlmXC-v2 Dong et al. ().

Specialist Chart Models.
Previous works specifically design models and algorithms for chart question answering. We compare Pix2Struct Lee et al. (), Matcha Liu et al. (), UniChart Masry et al. (), and Deplot Liu et al. (). Notably, Deplot fails to handle questions in arbitrary formats, so we extract table information with Deplot and use LLaVA-v1.6 to answer the questions.

**Table 2: The relaxed accuracy (%) performance on ChartQA. Ada.: Adaptive input resolution. *: Multi-scale image feature, 448$\times$9 in DocOwl. $\dagger$: Employing token merging to reduce computational overhead.**
| Models | Para. | Resolution | Relax Acc. @0.05 | Relax Acc. @0.10 | Relax Acc. @0.20 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Human | Aug. | Avg. | Human | Aug. | Avg. | Human | Aug. | Avg. |  |  |  |
| General MLLMs |  |  |  |  |  |  |  |  |  |  |  |
| LLaVA-v1.5 | 13B | @336 | 25.36 | 18.56 | 21.96 | 28.56 | 23.52 | 26.04 | 32.56 | 30.72 | 31.64 |
| Qwen-VL | 9.6B | @448 | 40.48 | 79.76 | 60.12 | 43.20 | 82.56 | 62.88 | 47.52 | 85.76 | 66.64 |
| DocOwl-v1.5 | 8B | @448* | 47.44 | 91.52 | 69.48 | 51.92 | 92.08 | 72.00 | 56.72 | 93.12 | 74.92 |
| InternlmXC-v2 | 8B | @490 | 62.72 | 81.28 | 72.00 | 66.72 | 84.08 | 75.40 | 70.80 | 86.56 | 78.68 |
| Specialist Chart Models |  |  |  |  |  |  |  |  |  |  |  |
| Pix2Struct | 282M | Ada. | 30.08 | 76.88 | 53.48 | 31.68 | 78.40 | 55.04 | 37.28 | 81.12 | 59.20 |
| Matcha | 282M | Ada. | 37.12 | 86.64 | 61.88 | 39.84 | 87.36 | 63.60 | 43.52 | 88.56 | 66.04 |
| UniChart | 201M | @960 | 34.64 | 83.28 | 58.96 | 36.48 | 84.24 | 60.36 | 38.88 | 85.28 | 62.08 |
| Deplot + LLaVA-v1.6 | 282M+13B | Ada. | 53.44 | 87.68 | 70.56 | 56.80 | 88.48 | 72.64 | 60.64 | 90.08 | 75.36 |
| Chart MLLMs |  |  |  |  |  |  |  |  |  |  |  |
| ChartVLM | 13B | Ada. | 42.08 | 82.48 | 62.28 | 43.84 | 82.88 | 63.36 | 46.00 | 83.28 | 64.64 |
| OneChart | 125M+13B | @1024 | 54.48 | 87.12 | 70.80 | 57.60 | 87.84 | 72.72 | 62.00 | 88.64 | 75.32 |
| ChartLlama | 13B | @336 | 58.40 | 93.12 | 75.76 | 61.20 | 93.60 | 77.40 | 63.52 | 94.00 | 78.76 |
| ChartGemma+PoT | 3B | @448 | 67.84 | 85.28 | 76.56 | 68.64 | 85.84 | 77.24 | 69.84 | 86.32 | 78.08 |
| TinyChart | 3B | @768$\dagger$ | 58.72 | 94.88 | 76.80 | 62.56 | 95.28 | 78.92 | 67.04 | 96.16 | 81.60 |
| ChartAst | 13B | @448 | 64.88 | 93.12 | 79.00 | 66.24 | 93.84 | 80.04 | 67.44 | 94.32 | 80.88 |
| TinyChart+PoT | 3B | @768$\dagger$ | 70.24 | 90.72 | 80.48 | 71.20 | 91.44 | 81.32 | 72.40 | 92.56 | 82.48 |
| ChartMoE (Ours) | 8B | @490 | 71.36 | 91.04 | 81.20 | 75.12 | 92.48 | 83.80 | 78.16 | 93.68 | 85.92 |
| ChartMoE+PoT (Ours) | 8B | @490 | 78.32 | 90.96 | 84.64 | 80.16 | 92.32 | 86.24 | 82.08 | 93.60 | 87.84 |

**Table 3: The zero-shot performance on ChartBench. No methods are fine-tuned on the trainset for fairness. We exclude PoT because ChartBench mainly assesses numerical extraction accuracy without math calculation.**
| Models | Regular Type | Extra Type | ALL |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Line | Bar | Pie | Avg. | Area | Box | Radar | Scatter | Node | Comb. | Avg. |  |  |
| General MLLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaVA-v1.5 | 29.12 | 21.26 | 17.28 | 22.10 | 21.73 | 20.94 | 27.50 | 23.47 | 36.80 | 24.30 | 24.96 | 23.38 |
| Qwen-VL | 38.00 | 20.71 | 38.24 | 29.46 | 28.83 | 24.17 | 35.00 | 19.50 | 18.50 | 25.50 | 26.56 | 28.18 |
| DocOwl-v1.5 | 49.60 | 31.69 | 31.54 | 35.68 | 12.27 | 23.33 | 22.50 | 36.13 | 29.60 | 38.80 | 27.38 | 32.05 |
| Mini-Gemini | 34.88 | 36.12 | 40.40 | 36.77 | 31.20 | 23.33 | 30.60 | 35.20 | 43.60 | 27.90 | 30.61 | 34.37 |
| InternlmXC-v2 | 68.16 | 48.74 | 56.60 | 54.50 | 27.47 | 25.33 | 40.10 | 52.93 | 50.40 | 46.20 | 39.72 | 48.41 |
| Specialist Chart Models |  |  |  |  |  |  |  |  |  |  |  |  |
| Pix2Struct | 2.56 | 2.37 | 1.90 | 2.33 | 0.13 | 0.13 | 4.60 | 0.67 | 0.40 | 3.20 | 2.93 | 2.16 |
| Matcha | 6.80 | 5.05 | 3.60 | 5.18 | 0.27 | 1.60 | 6.20 | 3.46 | 5.40 | 4.80 | 5.81 | 4.84 |
| UniChart | 7.04 | 5.35 | 4.30 | 5.55 | 3.86 | 4.80 | 11.60 | 5.06 | 15.80 | 9.60 | 8.30 | 6.78 |
| Deplot+LLaVA-v1.6 | 31.20 | 26.46 | 24.00 | 27.09 | 21.34 | 13.34 | 24.00 | 41.34 | 42.00 | 31.00 | 31.57 | 27.62 |
| Chart MLLMs |  |  |  |  |  |  |  |  |  |  |  |  |
| ChartVLM | 21.92 | 14.16 | 10.50 | 15.16 | 7.47 | 7.87 | 8.00 | 7.87 | 5.40 | 10.50 | 8.38 | 11.96 |
| ChartLlama | 26.80 | 18.83 | 20.80 | 20.99 | 14.27 | 12.00 | 24.30 | 27.73 | 26.20 | 25.80 | 21.71 | 21.31 |
| TinyChart | 32.40 | 25.81 | 22.50 | 26.71 | 10.13 | 14.80 | 13.40 | 28.14 | 10.80 | 21.60 | 22.56 | 22.51 |
| OneChart | 41.28 | 30.28 | 29.60 | 32.65 | 19.07 | 13.20 | 24.60 | 38.53 | 34.80 | 27.90 | 31.91 | 29.93 |
| ChartGemma | 50.48 | 38.21 | 32.10 | 39.89 | 28.27 | 24.13 | 28.10 | 48.00 | 41.80 | 43.40 | 42.47 | 38.46 |
| ChartMoE (Ours) | 71.44 | 51.57 | 52.80 | 56.31 | 38.40 | 24.13 | 40.20 | 62.67 | 58.00 | 49.20 | 55.58 | 51.67 |

Chart MLLMs.
Chart-oriented MLLMs are the promising direction for utilizing prior knowledge of LLMs.
ChartLLaMA Han et al. () proposes to generate high-quality instruction data to improve chart question-answering capabilities.
ChartAst Meng et al. () suggests aligning the connector with chart-table pairs before supervised fine-tuning.
ChartVLM Xia et al. () uses different decoders to handle different questions based on their difficulty.
ChartInstruct Masry et al. () conducts large-scale chart instruction tuning based on general MLLMs.
OneChart Chen et al. () converts the chart to the table with a dedicated decoder and uses LLMs to answer questions.
ChartGemma Masry et al. () proposes more instruction data and achieves efficient chart reasoning based on SigLIP Zhai et al. () and Gemma-2B Team et al. ().
TinyChart Zhang et al. () adopts token merge to reduce visual tokens and enable high-resolution chart input.

### 4.4 Main Results

ChartQA.
Tab.2 presents detailed comparisons of ChartMoE on ChartQA. ChartMoE significantly improves the baseline (InternlmXC-v2) performance (72.00% vs. 84.64%, +12.64%$\uparrow$ in Acc.@0.05). Compared to previous SOTA (TinyChart+PoT @768 pixel), ChartMoE consistently surpasses it across all metrics. The PoT effectively enhances the model’s mathematical reasoning capabilities, which is a common shortfall in current MLLMs. ChartMoE integrates better with PoT, indicating that it accurately extracts fundamental elements from charts. The questions in Human part tend to be more computationally complex and challenging, and ChartMoE shows more significant improvement in this area, especially after incorporating PoT. Notably, our error analysis in the Augmented part reveals that many errors stem from limitations of the evaluation criteria, i.e., string matching. For instance, it is marked incorrect if the prediction is It is between 2003 and 2005 and the ground truth is (2003, 2005). Forcing performance improvement on it may lead to model overfitting.

ChartBench.
Tab. 3 presents detailed comparisons of ChartMoE on ChartBench. None of the models, including our ChartMoE, undergo supervised finetuning on the ChartBench trainset to ensure fair experimental comparison. Chart-specific models typically underperform due to their limited generalization ability, which prevents them from effectively managing charts with sparse data point annotations ($<10$%). Deplot shows a distinct advantage over these types of models (27.62%) with the assistance of LLaVA-v1.6. The baseline (InternlmXC-v2) demonstrates strong generalization on ChartBench (48.41%), which may benefit from pre-training instructions designed for unannotated charts. Without additional design, ChartMoE improves the baseline performance comprehensively (48.41% vs. 51.67%), especially on extra chart types (39.72% vs. 55.58%, +15.86%$\uparrow$).

ChartFC & ChartCheck.
Tab. 4 compares ChartMoE on the synthetic ChartFC and real-world ChartCheck. ChartMoE consistently outperforms SOTA (e.g., ChartGemma +4.4%$\uparrow$ on ChartFC) and significantly improves the performance compared to InternlmXC-v2 (+6.83%$\uparrow$ and +8.76%$\uparrow$ on ChartCheck T1 and T2, respectively). Note that this is implemented without using training data for supervised fine-tuning, demonstrating ChartMoE’s strong generalization capabilities.

| Models | ChartFC | ChartCheck |  |
| --- | --- | --- | --- |
| T1 | T2 |  |  |
| PaliGemma | 58.26 | 67.34 | 68.50 |
| LLaVA-v1.5$\dagger$ | 61.28 | 70.22 | 70.03 |
| InternlmXC-v2 | 65.93 | 72.04 | 70.44 |
| ChartInstruct-LLama2 | 69.57 | 70.11 | 68.80 |
| ChartInstruct-FlanT5XL | 70.27 | 72.03 | 73.80 |
| ChartGemma | 70.33 | 71.50 | 74.31 |
| ChartMoE (Ours) | 74.73 | 78.87 | 79.20 |

Table: Table 4: The accuracy performance on ChartFC and ChartCheck. $\dagger$: tuning with ChartGemma instructions.

Figure: Figure 4: Training loss of different initialization.

Figure: Figure 5: Top-2 selected expert distribution on ChartBench.

## 5 Further Study

### 5.1 Model Architecture Ablation

We investigate the impact of three factors on our MoE connector: the number of experts, the number of activated experts, and the expert initialization manner. All the experiments are conducted with ChartQA training data and evaluated on ChartQA test split with relax accuracy metric.

Effect of the Expert Initialization Manner.
The initialization strategy plays a crucial role in determining the performance of the MoE connector. Effective initialization is essential to ensure that each expert performs its designated function optimally. As illustrated in Tab. 6 row 1-3, we explore the impact of 3 initialization strategies for the MoE connector. Random initialization serves as a baseline but struggles with convergence (refer to Fig. 4), resulting in a suboptimal accuracy of 73.20% at Acc.@0.05. Following CuMo Li et al. (), we employ the Co-Upcycle strategy by replicating the table-JSON-code aligned connector for all experts. Given the same starting point, this approach lacks expert diversity, which limits its effectiveness, resulting in an accuracy of 77.48% at Acc.@0.05. In contrast, our initialization assigns distinct parameters to each expert. This tailored approach enables each expert to capitalize on its specific strengths, resulting in the highest performance, achieving 78.76% in Acc.@0.05.

Effect of Number of Experts and Activated Experts.
As shown in rows 3-4 of Tab. 6, we compare ChartMoE configurations with 4 and 8 experts, keeping 2 experts activated. The 8 experts are initialized in pairs using the 4 methods illustrated in Fig. 2c. ChartMoE achieves 78.76% in Acc.@0.05 with 4 experts, which is slightly higher than the 78.60% achieved with 8 experts, showing a marginal increase of +0.16%. In rows 4-5, we compare the performance of configurations with 2 and 4 activated experts, finding similar results: 78.60% vs. 78.64% in Acc.@0.05. This analysis indicates that simply increasing the number of experts or activated experts does not necessarily lead to better performance. The configuration with 4 experts and 2 activated experts effectively balances complexity and performance, making it a suitable choice for ChartMoE.

### 5.2 Training Strategy Ablation

We analyze the impact of the training strategy across alignment and supervised fine-tuning stages. We use InternlmXC-v2 with ChartQA fine-tuning as our baseline, maintaining the same hyperparameters as the chart-specific annealing tuning stage.

**Table 5: Ablation study on ChartMoE architecture w.r.t. the total / activated / initialization of connector experts. All experiments are conducted on ChartQA.**
| Total<br>Experts | Activated<br>Experts | Experts<br>Initialization | Relax Acc @0.05 |  |  |
| --- | --- | --- | --- | --- | --- |
| Human | Aug. | Avg. |  |  |  |
| 4 | 2 | Random | 59.68 | 86.72 | 73.20 |
| 4 | 2 | Aligned Co-Upcycle | 64.96 | 90.00 | 77.48 |
| 4 | 2 | Ours | 67.92 | 89.60 | 78.76 |
| 8 | 2 | Ours | 67.20 | 90.00 | 78.60 |
| 8 | 4 | Ours | 67.68 | 89.60 | 78.64 |

**Table 7: Ablation study on the expert of MoE connector. We ignore the gating network and adopt specific expert output.**
| Connector | Relax Acc @0.05 |  |  |
| --- | --- | --- | --- |
| Human | Aug. | Avg. |  |
| Expert 0 (Vanilla) | 69.76 | 89.84 | 79.80 |
| Expert 1 (Table) | 63.60 | 89.12 | 76.36 |
| Expert 2 (JSON) | 60.64 | 82.48 | 71.56 |
| Expert 3 (Code) | 66.88 | 89.36 | 78.12 |

Effect of Alignment Strategy.
As shown in rows 1-3 of Tab. 6, translating the chart image into structural text formats such as table, JSON, and code during the alignment stage significantly enhances performance in downstream chart understanding tasks. After applying table-JSON-code alignment, the model achieves 77.20% in Acc.@0.05, representing a notable improvement of +1.52%$\uparrow$. When combined with our proposed MoE architecture, performance further increases to 78.76%, a total gain of +3.08%$\uparrow$ in Acc.@0.05.

Effect of Supervise Fine-Tuning Strategy.
As shown in rows 4-5 of Tab. 6, we divide the supervised fine-tuning stage into two phases. By incorporating high-quality knowledge learning using the MMC dataset, ChartMoE achieves 79.04% Acc.@0.05, reflecting a 3.36% improvement. In the chart-specific annealing tuning phase, we introduce ChartGemma data to enhance the model’s reasoning and PoT capabilities, leading the model to peak performance (81.20%, +5.52%$\uparrow$).

### 5.3 In-depth Analysis

Effect of the Each Expert.
To explore the role of each expert in ChartMoE, we bypass the gating network and manually select the output of specific experts. As shown in Tab. 8, E0 performs the best (79.80%), which is consistent with the distribution in Fig. 5. However, this doesn’t mean other experts lack relevance, which may offer crucial insights at key moments (Fig.6).

Effect of Alignment Task.
As shown in Tab.8, we explore various alignment tasks based on the baseline (linear connector). After alignment, the performance significantly declines compared to the vanilla version. However, the aligned model exhibits a substantial improvement after supervised finetuning on the ChartQA train split, which is consistent with previous observations Meng et al. (); Yan et al. (). Specifically, alignment based on JSON and code exhibits remarkable improvement over the one based on the table.

Figure: Figure 6: Visualizations of top-1 expert selection. Only the boundaries of the merged tokens are plotted.

### 5.4 Expert Distribution Visualiztion

As shown in Fig. 5 and Fig. 6, we visualize the expert distribution in the MoE connector on the ChartBench test set. We designate the vanilla connector as E0, while E1-3 corresponds to connectors aligned with tables, JSON, and code. As depicted in Fig. 5, the trend is consistent across different chart types, with E0 and E3 being the most frequently selected connectors. The expert selection shows no extreme bias, as even the least chosen, E1, accounts for over 10%.

We further visualize the expert selection for each image token, revealing the preferences of each expert. As shown in Fig. 6, E0 is the primary choice for background tokens, explaining its dominance in Fig. 5. E1 and E2 are more frequently chosen by tokens from titles, axis labels, or legends, as these elements are commonly found in tables and JSON files. ChartMoE tends to use E3 to focus on the data points and visual elements within the chart, e.g., data points on the line, digital text, and edges in a node chart. These components are essential for accurately re-drawing the charts.

## 6 Conclusion

We introduce ChartMoE, a multi-task aligned and instruction-tuned MLLM designed for complex chart understanding and reasoning.
We replace the linear connector with the MoE architecture and initialize each expert with parameters derived from different alignment tasks.
We further present the ChartMoE-Align dataset, a synthetic collection of over 900K table-json-code-chart quadruples, to facilitate alignment training across different experts.
This approach preserves the strengths of each alignment task, ensuring efficient training and superior model performance.
ChartMoE outperforms the previous state-of-the-art on several benchmarks by a large margin and excels in real-world applications such as chart question answering, translation, and editing.

## Appendix A Additional experimental Settings and results

### A.1 Top-2 experts distribution

Our ChartMoEemploys MoE connector expert parameters initialized with various alignment tasks. To investigate the impact of these initialization methods on model performance, we present the comparisons in Tab. 6& 8& 8 and Fig. 4&5. For a deeper analysis, we explore how different initialization methods affect expert selection. As shown in Fig. 7, both random initialization and co-upcycle result in a more uniform distribution of experts. However, this uniformity does not inherently lead to improved performance or interpretability, possibly due to insufficient differentiation among the experts. In contrast, our ChartMoE clearly prefers specialized roles, as illustrated in Fig. 6& 8& 9.

Figure: (a) Random
Refer to caption: x4.png

### A.2 Summary of hyperparameter settings

The training process of our ChartMoEis structured into three distinct phases: Alignment Pretraining, High-Quality Knowledge Learning and Chart-Specific Annealing Tuning. Table 9 provides a comprehensive overview of the hyperparameter configurations employed during each of these training stages.

**Table 9: Training hyperparameters of ChartMoE for all stages.**
| Configuration | Alignment Pretraining | High-Quality Knowledge Learning | Chart Specific Annealing Tuning |
| --- | --- | --- | --- |
| Connector Initialization | InternlmXC-v2 | Table&JSON&Code Experts + InternlmXC-v2 | ChartMoE 2nd-stage |
| LLM Training | Freeze | LoRA | LoRA |
| Image Resolution | 490 | 490 | 490 |
| ViT Sequence Length | 1225 | 1225 | 1225 |
| Optimizer | AdamW | AdamW | AdamW |
| Optimizer Hyperparameter | $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e^{-8}$ | $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e^{-8}$ | $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=1e^{-8}$ |
| Peak Learning Rate | $5e^{-5}$ | $1e^{-5}$ | $5e^{-5}$ |
| Learning Rate Schedule | cosine decay | cosine decay | cosine decay |
| Weight Decay | 0.1 | 0.1 | 0.1 |
| Gradient Clip | 1.0 | 1.0 | 1.0 |
| Warm-up Ratio | 0.01 | 0.01 | 0.01 |
| Global Batch Size | 64 | 32 | 32 |
| Gradient Acc. | 8 | 8 | 8 |
| Numerical Precision | bfloat16 | bfloat16 | bfloat16 |
| Optimizer Sharding | ✓ | ✓ | ✓ |
| Gradient Sharding | ✓ | ✓ | ✓ |
| Parameter Sharding | $\times$ | $\times$ | $\times$ |
| Activation Checkpointing | ✓ | ✓ | ✓ |
| GPU Hours (A100-40G) | 240 | 138 | 76 |

## Appendix B Additional Vsualizations of top-1 expert selection

Figure: Figure 8: More visualizations of top-1 expert selection on general images randomly sampled from LLaVA-CC3M. These examples show the selection preferences of different experts in ChartMoE.
Refer to caption: extracted/5834836/Fig/figs/appendix_tokenwise_llava/logo/514.png

Figure: Figure 9: More visualizations of top-1 expert selection on chart images. The vanilla expert primarily handles background tokens, and the chart visual markers are handled by other experts.
Refer to caption: extracted/5834836/Fig/figs/appendix_tokenwise_chart/chartqa/588.png

## Appendix C Detailed Examples in ChartMoE-Align

Figure: (a) Charts in ChartMoE-Align.
Refer to caption: x9.png

## Appendix D Additional Capabilities on Chart Editing and Highlighting

Leveraging fine-grained alignment, particularly in translating input charts into corresponding Python code, ChartMoE demonstrates advanced capabilities in editing and highlighting key elements. As illustrated in Fig. 11, ChartMoE effectively modifies input charts based on user instructions, such as converting a vertical bar chart into a horizontal one, transforming a pie chart into a bar chart, and adjusting the values of specific elements.

ChartMoE also excels at adding auxiliary lines and highlighting elements within charts, facilitating the rapid extraction of valuable insights. For instance, in Fig. 12, ChartMoE draws a horizontal line through the lowest data point and both horizontal and vertical lines through each data point, aiding in the precise determination of values. Additionally, in Fig. 13, ChartMoE highlights the maximum data point using two different methods: employing a larger green scatter with a prominent marker and applying a bright color to make it stand out.

Figure: Figure 11: ChartMoE learns to convert the type and adjust the values of the input chart to redraw a new one, according to the user instruction.

Figure: Figure 12: ChartMoE can generate python code to draw some auxiliary lines on the input chart according to the user query, which can benefit parsing and understanding for both human and machine.

Figure: Figure 13: ChartMoE shows its ability on highlighting the key elements via understanding the requirement of user, which is a good way to showcase the most valuable insights.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 4
questions: Q1: Modality-Specific Contributions

It would be beneficial for the paper to elaborate on the unique contributions of different representations (JSON, code, and chart) in the context of chart-related tasks. Understanding how each representation impacts the model's learning and performance could provide insights into optimizing future models for similar tasks.


Q2: Necessity and Efficiency of Large-Scale Alignment Dataset

The heuristic approach to generating a large-scale dataset raises questions about the efficiency and necessity of such a volume of data. Is there potential to achieve similar performance with a smaller, possibly more curated dataset? This exploration could lead to more resource-efficient training processes and better generalization in practical applications.
soundness: 4
strengths: S1. Innovative Methodology: The introduction of the ChartMoE method, which utilizes a Mixture of Experts (MoE) architecture, represents a significant innovation in the field of automatic chart understanding. This approach addresses the modality gap effectively and could potentially set a new direction for future research in multimodal learning systems.

S2. Comprehensive Dataset: The creation of the ChartMoE-Align dataset is commendable. Its large scale and diversity are well-suited for robust pre-training in chart alignment tasks. This dataset not only serves the immediate needs of the study but also provides a valuable resource for the broader research community to explore complex multimodal tasks involving charts and text.

S3. Extensive Experimental Validation: The paper presents extensive experiments that demonstrate the effectiveness of the ChartMoE approach. The thoroughness of these experiments, which include a variety of scenarios and detailed performance metrics, establishes a strong benchmark for future comparative studies.

S4. Clear Writing: The manuscript is exceptionally well-written, providing clear explanations and methodical presentation of the concepts and methodologies involved. This clarity enhances the reader's understanding and appreciation of the work's contributions to the field.
summary: The paper introduces ChartMoE, a novel approach that leverages a Mixture of Expert (MoE) architecture to improve automatic chart understanding with Multimodal Large Language Models (MLLMs). ChartMoE addresses the limitations of existing MLLMs by using specialized linear connectors for diverse expert initialization, coupled with a unique dataset, ChartMoE-Align, containing nearly one million chart-related quadruples. This setup significantly enhances data interpretation from charts, pushing the accuracy on the ChartQA benchmark from 80.48% to 84.64%, showcasing a substantial improvement over previous methods.
weaknesses: W1: Details on Dataset Construction

The paper lacks critical details on the dataset construction process. Clarifications are needed regarding the criteria used to select and filter charts for inclusion in the dataset. Specifically, the process for generating meta CSV data via Large Language Models (LLM) requires more transparency.
More details on which LLMs were used and the code templates for different types of charts are missing. Such information is crucial for reproducibility and for understanding the dataset's applicability to other multimodal tasks.
The manuscript should discuss the steps taken to ensure the quality of the data, including any validation mechanisms or controls used during dataset assembly.

W2: Clarification of Experiment Results

The paper briefly mentions that the proposed method shows weaknesses in some settings compared to baselines, as detailed in Tables 2 and 3. However, these points are not adequately addressed or explained. A more thorough analysis of why ChartMoE underperforms in these instances would be valuable for readers and for future improvements to the method.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: see above
soundness: 2
strengths: 1. ChartMoE’s use of task-specific expert connectors in a Mixture of Experts (MoE) framework provides a  solution to multimodal chart understanding.

2. ChartMoE-Align, a large-scale dataset with varied chart alignments (table, JSON, code).

3. The three-stage training paradigm increass its accuracy in extracting and interpreting numerical data.
summary: This work introduces ChartMoE, a multimodal large language model  that enhances automatic chart understanding through a MoE architecture. Unlike traditional linear connectors, ChartMoE uses diverse expert connectors aligned with specific tasks (chart-table, chart-JSON, and chart-code) to bridge the gap between visual encoders and large language models. The paper also presents ChartMoE-Align, a dataset with nearly 1 million chart-table-JSON-code quadruples for training.
weaknesses: 1. The use of MOE in MLLMs is not particularly novel, as several prior works have already explored MoE structures to enhance model performance. From an innovation standpoint, this reliance on MoE does not introduce a distinctly new approach and could be considered a weakness in terms of contribution.

2. The multi-expert structure, along with the diverse alignment tasks, adds significant complexity to ChartMoE’s architecture. Also, the training data, being mostly synthetic, might limit the model’s ability.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. In line 369, the error analysis suggests that many errors stem from the limitations of the evaluation metric, namely string matching. Are other comparison models also limited by this?
2. Can you provide more detailed explanations of the data construction process, such as how code templates for different types of charts were obtained? Were they manually constructed?
soundness: 4
strengths: This paper incorporates a Mixture of Expert architecture to bridge the gap between charts and language models and offers a valuable insight for the expert initialization manner. The creation of the ChartMoE-Align dataset with nearly 1 million chart-table-JSON-code quadruplets is a significant contribution to the field, allowing for detailed and meticulous chart alignment pre-training.

The paper is clear and well written in general, is well motivated, and comes with an extensive and comprehensive ablation study.
summary: The paper introduces ChartMoE, a multi-task aligned and instruction-tuned MLLM designed for complex chart understanding and reasoning. The key contribution is the replacement of the traditional linear connector with the Mixture of Expert (MoE) architecture, which improves chart understanding by bridging the modality gap. Additionally, a new dataset called ChartMoE-Align is introduced, containing nearly 1 million chart-table-JSON-code quadruples for alignment training. The proposed three-stage training paradigm and high-quality knowledge learning approach result in significantly improved performance compared to the previous state-of-the-art on various benchmarks.
weaknesses: The paper focuses on conducting experiments on a single Vision Encoder and Large Language Model, which limits the generalizability of the proposed method. It would be beneficial to test the effectiveness of ChartMoE on a diverse set of MLLMs to ensure its applicability across different models and scenarios.

The paper does not thoroughly discuss potential limitations or challenges that may arise when implementing ChartMoE in practical applications.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Have you ever tried training on your alignment data with random initialization of expert parameters and balanced loss from scratch? It will better prove the significance of this work.
soundness: 3
strengths: 1. First introduce MoE MLLM for chart task, the model architechture design which using MoE in connector is novel.
2. Detailed experiments and analysis, extensivequantitativeandqualitativestudies.
3. Introduced a large dataset for chart pre-train data.
summary: This paper proposed ChartMoE, which employs the Mixture of Expert (MoE) architecture to replace the traditional linear projector to bridge the modality gap.
weaknesses: 1. limmited innovation: MoE on multimodal language models has been explored in other domains. There's a trade-off between the performance gain and the increase of model parameters and inference time. 
2. The contributions of MoE module is unclear, more theoretical analysis is needed such as the knowledge in the routing to differnernt experts.

Meta-review: The paper introduces ChartMoE, a model that enhances complex chart understanding through a Mixture of Expert (MoE) architecture, replacing traditional linear projectors. It also presents the ChartMoE-Align dataset, designed for three specific alignment tasks, demonstrating the model's practical utility. The experimental results are robust and convincing. I recommend that the authors strengthen the presentation of their novel approach in the Introduction to highlight the advancements over existing methods better.

Decision: Accept (Oral)