ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding
Authors: Zhengzhuo Xu
Bowen Qu
Yiyan Qi
SiNan Du
Chengjin Xu
Chun Yuan
Jian Guo
Venue: ICLR 2025 Oral (2025)
Keywords: Multimodal Large Language Models
Chart Reasoning
Mixture of Expert
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Automatic chart understanding is crucial for content comprehension and document parsing. Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding through domain-specific alignment and fine-tuning. However, current MLLMs still struggle to provide faithful data and reliable analysis only based on charts. To address it, we propose ChartMoE, which employs the Mixture of Expert (MoE) architecture to replace the traditional linear projector to bridge the modality gap. Specifically, we train several linear connectors through distinct alignment tasks, which are utilized as the foundational initialization parameters for different experts. Additionally, we introduce ChartMoE-Align, a dataset with nearly 1 million chart-table-JSON-code quadruples to conduct three alignment tasks (chart-table/JSON/code). Combined with the vanilla connector, we initialize different experts diversely and adopt high-quality knowledge learning to further refine the MoE connector and LLM parameters. Extensive experiments demonstrate the effectiveness of the MoE connector and our initialization strategy, e.g., ChartMoE improves the accuracy of the previous state-of-the-art from 80.48% to 84.64% on the ChartQA benchmark.

Published as a conference paper at ICLR 2025
CHARTMOE: MIXTURE OF DIVERSELY ALIGNED EX- PERT CONNECTOR FOR CHART UNDERSTANDING
Zhengzhuo Xu Bowen Qu Yiyan Qi Sinan Du² Chengjin Xu' Chun Yuan Jian Guo 14f
'International Digital Economy Academy TTsnnguua University *Peking University *Hong Kong University of Science and Technology (Guangzhou) https: /ttp///gttttt/t////e com/ IDEA- Fea-Finnii inAI I
ABSTRACT
Automatic chart understanding is crucial for content comprehension and docu- ment parsing. Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding through domain-speciffe alignment and fine-tuning. However, current MILMS still struggle to provide faithful data and reliable analysis only based on charts. To address it, propose ChartMoE, which employs the Mixture of Expert (MoE) architecture to replace the tradi- tional linear projector to bridge the modality gap. Specifically, we train several linear connectors through distinct alignment tasks, which are utilized as the foun- dational initialization parameters for different experts. Additionally, we introduce dataset with nearly 1 million Shart-table-JSON-eee ples to conduct three alignment tasks cchartiable//SSONcodde)) Combined with the vanilla connector, we initialize different experts diversely and adopt high- quality knowledge learning to further refine the MoB connector and LLM param- eters. Extensive experiments demonstrate the effectiveness of the MoE connector and our initialization strategy, e.g.. ChartMoE improves the accuracy of the previ- ous state-of the-art art from 80.48% to 84.64% on the ChartQA benchmark.
ChartMoF--llp
quadru-
INTRODUCTION
Charts serve as a fundamental tool for data visualization, with automated chart interpretation gain- ing prominence in domains such text analysis (Hoque et al., 2017), scientific research (Hsu al., 2021), and policy-making (Wu et al., 2024). Chart understanding is a complex task that demands the identification of visual cues, the comprehension of intricate interactions, and the precise inference of values informed by prior knowledge (Huang et al., 2024). Previous work (Liu et al., 2023b;a) typically pre-trained on domain-specific charts, which are constrained by limited resources and nar- row task focus. In contrast, Multi-modal Large Language Models (MLLMs) (Li et al., 2023; Liu al., 2023d; Bai et al., 2023a; Ye et al., 2023b; Chen et al., 2023; OpenAI, 2023) exhibit sub- stantial potential in image comprehension and instruction following. The community has achieved advanced progress by creating chart understanding datasets (Liu et al., 2023c; Han et al., 2023; Masry et al., 2024b; Xu et al., 2023) and applying supervised fine-tuning based on well-performed MLLMs (Meng et al., 2024; Yan et al., 2024). With the exponential growth of chart data, automated chart interpretation via MLLMs is emerging
promising avenue.
Recent studies advocate for chart alignment as a foundational step for VLaVk-like MLLMS (Liu et al., 2023d; Zhang et al., 2023; Xue et al., 2024), which bridge the visual encoder and LLM through MLP connector. They usually utilize chart-to-talle alignment task to train the connector effectively (Meng et al., 2024; Yan et al., 2024; Hu et al., 2024). However, tables only provide basic information, such as numerical values and titles, which fail to capture the full range of chart elements. Despite some efforts to align with more informative text (Yan et al., 2024), the heavy alignment tasks may lead to the erosion of the connector's s general capabilities, e.g., instruction fol- lowing and visual counting, which are derived from the pre-training on large-scale visual-languaae data. To mitigate knowledge forgetting, one intuitive approach is to further tune with its original data, which results redundant training and computational burden.
In this paper, we try to address these challenges via Mixture of Experts (MoE) architecture (Zoph et al., 2022). MoE enhances model capacity by activating subset of experts through router. Since
"Equal contributions. 'Correponndngg authors.

---

Published as a conference paper at ICLR 2025
Answer
Label
Repke
Edilig
LLM
Tekza-vit Tap-l Eapurt Sckeetin Mutl Conneder
11
Text
Tranffrraziin
Galle
Clarr Qu
Ascrnding in rablii Cat
dacs Chiia sne in 20007
Neialtes
Chart
Figure I: Overview and capabilities of ChartMoE: We introduce a MoE architecture connector and provide visualizations the top-1 expert selection (refer to Fig. 6 and Appendix for details). ChartMoE can extract highly precise values and provide flexible chart editing through code-based interactions.
the alignment tasks work on the connector, we replace only the MLP projector with MoE while keeping the vision encoder and L.LM frozen. Our insight lies in the expert initialization manner. Random initialization can lcad to training instability and convergence at sub-optimal points (Fig. 4). Recent co-upcycling initialization (Komatsuzaki et al., 2023) addresses this issue by duplicating the vanilla connector parameters across all experts. However, it fails to avoid the dilemma of expert homogenization, where the experts end up with similar functionalities.
In contrast, we attempt to inject distinct prior knowledge into each expert first to tackle these chal- lenges. Unlike natural images, charts can be represented in various text formats, e.g., tables, at- tribute JSON, and rendering code. As shown in Fig. 1& 2, in addition to chart-table, chart-ISON alignment to capture detailed elements like color or topological relationships and chart- code alignment to incorporate rendering details such as numerical values, color hex codes, and visual elements interactions (refer to Appendix C). We independently conduct various alignment tasks to capture more diverse chart features and thus obtain three distinct initialization approaches. We also retain the vanilla connector to preserve the capabilities of the MLLM on general tasks effectively.
introduce
Building upon the proposed four initialization manners, we introduce ChartMoE, an SFT-based MLLM with MoE connector for chart comprehension and reasoning. Our purpose in applying MoE connector not to increase model capacity, but rather to improve chart comprehension and di- verse representation through alignment tasks, while maintaining performance on other general tasks. Hence, we preserve the original connector parameters as one expert initialization manner. The MoE connector is extremely lightweight, so it adds negligible computational cost during both training and inference. Interestingly, we observe that experts in ChartMoB exhibit distinct visual token prefer- ences, e.g., the vanilla expert favors background tokens while other experts focus more tokens with legends or numbers (Fig. 6 and Appendix B). Considering that the distribution of visual to- kens is naturally imbalanced in chart scenarios, we remove the expert-balancee! loss in MoE and obtain further performance gain. Due to the scarcity of rich structural text for chart alignment, we design pipeline (Fig. 3) to generate nearly million quadruplets caart-tabl--SOON-cole the ChartMoE- Align dataset for alignment. We train ChartMoE in 3 stages. First, we initialize ex- perts via the proposed four manners. Then, we conduct high-quality knowledge learning using the MMC instruction (Liu et al., 2023c) to train the routing network, expert connectors, and LoRA (Hu et al., 2022) modules. Finally, we employ annealing training on ChartQA (Masry et al., 2022) and ChartGemma (Masry et al., 2024b). ChartMoE achieves state-of-the-aar (SOTA) performance and provides more precise numbers and comprehensive attributes of charts (Appendix E). Refer to Ap- pendix D.1 for detailed comparisons with other MoE works w.r.t. motivation, initialization, and complexity. In summary, our contributions are:
build
a) We present ChartMoE for faithful and reasonable chart understanding, with the connector based on Mixture of Expert architecture, to bridge the chart and LLM branches. All experts are initial- ized based on various aligmment training tasks to avoid expert homogenization.
b) We introduce ChartMoE-A Allign, a large-scale dataset with nearly 1 million meticulous chart- table-ISON-code quadruplets for chart alignment pre-training.
2

---

Published as a conference paper at ICLR 2025
We propose to train ChartMoE with three-stage training paradigm, including connector align- ment pre- pre-training, high-quality knowledge learning, and annealing chart tuning.
d) Extensive quantitative and qualitative studies demonstrate that ChartMoE significantly outper- forms previous state-of-the-art across several benchmarks by a large margin.
RELATED WORK
Multimodal large language models leverages connector to bridge the gap between large lan- guage models (Touvron et al., 2023; Radford et al., 2018; 8 Brown et al., 2020; Zhang et al., 2022; Zheng al., 2023) and vision encoders (Radford et al., 2021; Oquab et al., 2023) to enable enriched capabilities of comprehension and instruction following. Approaches such as BLIP2 (Li et al., 2023), Flamingo (Alayrac al., 2022), mPLUG-Owl (Ye al., 2023b). and Qwen-VL VL (Bai al., 2023b) utilize QFormers or Resamplers to align modalities on extensive datasets of image-text pairs. LLaVA (Liu al., 2023d; 2024b) is the pioneering work to extend the instruction tuning paradigm to visual tasks with text-only GPT-4 (OpenAI, 2023), achieving tremendous performance using simple MLP without compromising visual information to refine the multimodal alignment. Some works (Lin et al., 2023; Tong et al., 2024b;a) explore the combination of various vision encoders, complementarily enhancing visual representations to bolster the fine-grained visual perception of MLLMs. Despite efforts in structural design, training strategies and data quality remain crucial in the advancement of MILMSs
Chart Reasoning refers chart analysis, summarization, and etc. Existing methods can be cate- gorized as 1) Twoo-tage methods use specialized extraction modules to generate intermediary rep- resentations of chart information, like tables, which are provided as textual prompts for LLMs. Pix2Struct (Lee et al., 2023) aligns markdown data with charts. MatCha (Liu et al., 2023b) aligns various data formats (e.g., tables and code) with charts on several downstream tasks. DePlot (Liu et al., 2023a) fine-tunes Pix2Struct for table extraction and uses LLMs to process queries based on the extracted data. CLartVLM (Xia et al., 2024) employs a discriminator ascertain the necessity of intervention by LLMS for given query. 2) End-to-end methods strive to tackle chart reasoning chal- lenges with a unified model. Chartl.lama (Han al., 2023) incorporates diverse charts and down- stream tasks based on LLaVA (Liu et al., 2023d). ChartPaLI (Carbune et al., 2024), ChartAst (Meng et al., 2024), and MMC (Liu et al., 2023c) conduct aligmment on table-chart pairs. UReader (Ye et al., 2023a) aligns all data with markdown, while mPLUG-Ow12 (Ye et al., 2023c) achieves supe- rior performance with high-resolution inputs. ChartThinker (Liu et al., 2024c) and DOMINO (Wang et al., 2023) propose the CoT (Wei et al., 2022) for chart reasoning. LaMenDa (Zhuowan et al., 2024) trains MILMs via step-by-step reasoning QA. ChartReformer (Yan et al., 2024) introduces chart-JSON alignment, while OneChart (Chen et al., 2024) aligns charts with Python dictionaries. MiniGPT-v2 (Chen et al., 2023), Doc-Owl (Hu et al., 2024). and TinyChar (Zhang et al.. 2024) tackle the reasoning efficiency for high-resolution charts by merging tokens.
CHARTMOE
3.1
ARCHITECTURE
The ChartMoB is based on InternlmXC-v2 (Dong et al., 2024) due to the concise LLaVA-like ar- chitecture (Liu et al., 2023d) and performance on par with GPT-4 on text-image comprehension. The base model includes a vision encoder and a LLM connected by two-layer MLP. ChartMoE replaces the MLP with a MoE architecture as the connector to leverage diverse prior knowledge.
Vision Encoder. We utilize CLIP ViT-Large (Radford et al., 2021) as the vision encoder, leveraging its rich prior knowledge gained from training on millions of image-text pairs. Considering the impact of chart resolution balance between efficiency and performance. Formally, the visual encoder M (-) will project the chart I into N tokens V := {11, .2, UN), where N 1225 in the ChartMoE.
performance, we set the input resolution to 490 490 to strike
Mixture-of-Expertss Connector. As illustrated in Fig. 2c, the MoE architecture employs a parallel multi-expert collaboration approach. This architecture comprises L experts (.),.), each designed with the same linear layer as the baseline. For a visual token v, given by M the gating network MG)) will calculate the routing weight (v;) of each expert Mf and select top-K activate.
3

---

Published as a conference paper at ICLR 2025
Ttanalase the chan BND Ierras
Aerpres
Layer
Layen
dedty 17t,
LLM *
LLM
LoKA
Toanlate adrii allon
anilible
Type antirt
Teat Tokan
Vizzal Toionn
Vical Tokens
Weigheed
Inaracione
J5ON
Alggene Ieamections
Uonnactor
Contort the chart tynher code
Chart
Ganneke
wi Bronks
Voali Takao
Visii Takes
Cide
Charr
(5) Alignment Instructions
Alignment Pretraining
Supervised Finetuning
Figure 2: Overview proposed ChartMoF. (a) Examples of alignment instructions. (b) conduct three different alignment tasks in parallel. (c) We initialize MoF connectors in four different manners and train the gate network, experts, and LoRA during the supervised fine-tuning stage.
Finally, the tokens processed by each expert M will be averaged according to the weight gj (u:) given by M get the token i; for the LLM branch M
Large Language Model. Following the baseline, we employ the InternLM2-77 7B.F---S--- variant as the LLM M implemented as a transformer decoder with a causal attention mask. We concate the visual tokens V {61, . "e....,ūr)) given by MoE connector with the M input text T tokens T ;= (t1,t2,... tv } to form the input token sequence for the LLM M Formally, given the chart I and instruction T, the output O of proposed ChartMoE can be formulated as:
{ur, 02 ................................ UN M (I),
L
i = Σ (1) (v.), g(v;) = (m))
(2)
o = (2, NN; la..,,,,.................. tM));
where a indicates softmax and the Top(: K) will reset the nop-Top K routing weight to 0.
Initialization of Expert. Previous approaches initialize expert parameters via 1) Random initializa- tion, which may lead to convergence difficulties during supervised fine-tuning, and 2) Co-upcycling initialization (Komatsuzaki et al., 2023), i.e., copy baseline connector parameters to each expert, which may lead to homogenization of experts. ChartMoE proposes initializing experts' parameters through distinct alignment tasks. We eliminate the load-balancing loss typically used in standard MoE. architectures equalize expert activation frequencies, as initialization approach allows each expert to specialize in its preferred visual tokens, which inherently exhibit biased distributions.
3.2
ALIGNMENT PRE-TRAINING.
The key insight of ChartMoE is the experts' initialization parameters from the different alignment pre-training (Fig. 2a). Specifically, as illustrated in Fig. 2b, we align expert connectors using three distinct alignment tasks, where only the connector parameters will be updated. We visualize the visual token preferences of each expert for both chart (Fig. 6& 12) and non-chart (Fig. 11) images.
Alignment with Table. Charts convey key information that can more precisely expressed in tab- ular form, and LLMs are particularly adept at processing such structured data. Hence, we introduce a chart-table alignment task, aiming to translate chart content into tabular format. The connector trained to convert chart information into corresponding CSV tables, thereby improving model performance in numerical data extraction and chart interpretation.
Alignment with JSON. Although tables capture the numerical information from charts, they miss semantic details such as colors, shapes, and fonts. To fill this gap, we propose a chart-JSON align- ment task, which represents chart attributes in ISON format. This task requires the connector to
4

---

Published as a conference paper at ICLR 2025
Filler out mata CSV fooet cuisting datlasets
inversion Giscuree Depioe model
Get execunable oode with pralefinee templates and properiess
Failed
ali
Discard the quadruple
CSV data
Rendom simpll prodefinal attribution sets
Success
by large language models
Keep the quadlupl
CSV Table
Attribution ISON
Python Code
Render
Chart
Figure 3: Overview ChartMoE Align data generation pipeline. The charts are plotted by Python matplotlib.
focus not only on the numerical data but also on visual and semantic properties. Accurately extract- ing chart attributes is essential for tasks like chart redrawing and editing.
Alignment with Code. To fully align with charts, we further introduce a chart-code alignment task. Since the underlying drawing code fully defines chart, this approach enables the connector to con- vert the chart's visual tokens into representations in the LLM domain. Notably, we provide the drawing code explicitly, including precise numerical values and rendering attributes, e.g.. numbers represented in Python lists and colors hexadecimal code. Re- fer to Fig. 13 for more detailed cases. The code en- ables the model to perform in-depth summarization, analysis, and editing of charts. This expert is signifi- cantly more sensitive to the trends and key elements in the charts.
Table I:
Datasets used for training ChartMoE. We conduct alignment pre-training with synthetic data and supervised tuning with high-quality, real- world data. Only ChartQA is used in the ablation due to GPU constraints.
Source
Data Type
Task Type
Samples
Alignment Training
chart to table
18.3K
ChamQA
synthetic
chart JSON
18.3K
chart code
18.3K
chart to table
157K
PlotQA
synthetic
chari JSON
157K
chart code
157K
chart to
ChartY
synthetic
chari to JSON
763. 6K
763.6
ChartMoE-Align generation pipeline. As Fig. illustrates, 1) We filter charts with meta CSV from existing datasets (Masry et al., 2022: Methani et al., 2020) and data generated by LLMs (Chen et al., 2024). 2) We use a fine-tuned Deplot (Liu et al., 2023a) inverse the plotting attributes following the templates provided by ChartReformer and randomly sample missing attributes from the predefined set. 3) We create code templates for different types of charts and generate plotting code based on the meta CSV and extracted JSON attributes. Note that all values and attributes in the code are explicitly represented. 4) We retain the (table, JSON, code, chart) quadruples that pass compilation. Tab. shows the data sources & size and refer to Appendix C for details.
Total
2.8M
Usage: Table 500K 2SON 200K Code 100K
800K
High-Qualit Knowledde Learning
synthetic
QA & reasoning
MMC
410K
real world
summariztion
Charr Specific Annealing Tuning
ChanQA
real world
QA
x
Q4 & PoT reasoning
ChartGiemma
real world
163.2K
summariztion
Total
220.8K
3.3 SUPERVISED FINE-TUNING.
We initialize ChartMoE using the structure shown in Fig. 2c after aligning the connectors across distinct tasks separately. We also retain the vanilla connector to maintain the baseline's 's excellent dialogue capabilities, which aligns with the principle of residual optimization (He et al., 2016). We train the MoE connector and LLM during this stage with LoRA (Hu et al., 2022), as shown in Fig. 2c. Considering the training principles proposed is divided into high-quality knowledge learning and chart-specific annealing training.
VAL-AAN (Liu al., 2024a), this stage
High-Quality Knowledge Learning. We adopt MMC (Liu et al., 2023c) to enhance the Chart- MoE's knowledge. MMC includes a variety of chart types and tasks such as chart-related question answering, translation, extraction, reasoning, and analysis. Considering data quality, we only utilize the MMMCInsttructio subset, which has been manually verified. Notice that the quality of instruction data is more important than quantity in this stage.
Chart Specific Annealing Tuning. Following Llama-v3.1 (Team et al., 2024b), perform anneal- ing tuning before evaluating mainstream benchmarks. We increase the learning rate and conduct instruction tuning using the training sets of ChartQA and ChartGemma adjust the query styles and answer formats of these benchmarks.
Program of Thought (PoT) Inference. We require the model generate the variables and opera- tion code rather than producing direct answers. This inference pipeline addresses the mathematical capabilities by employing Python handle the logical computations, which is the shortcoming of
5

---

Published as a conference paper at ICLR 2025
Table 2: The relaxed accuracy (%) performance ChariQs Ada.: Adaptive input resolution. *: Multi-scale
image feature, 448x448 DocOwl. i: Employing token merging to reduce computational overhead.
Relax Acc.
@0.05
Relax
@0.10
Relax Acc. @0.20
Models
Para
Resolution
Human
Avg.
Human
Aug.
Avg.
Human
Aug.
Avg.
Generul MLLMs
v1.
13B
69336
25.36
18.56
21.96
28.56
23.52
26.04
32.56
30.72
31.64
Qwen-VL VL
9.6B
@448
40.48
79.76
60.12
43.20
82.56
62.88
47.52
85.76
66.64
DocOw1-v1.5
SB
@-448*
47.44
91.52 69.48
51.92
92.08
72.00
56.72
93.12
74.92
InternimX--v2
8B
69490
62.72
81.28
72.00
66.72
84.08
75.40 40
70.80
86.56
78.68
Specialist
Chart Models
Pix2Struct
282M
Ada.
30.08
76.88
53.48
31
78.40
55.04 14
37.28
81.12
59.20
Matcha
282M
Ada.
37.12
86.64
61.88
39 99.84
87.36
63.60
43.52
88.56
66.04
UniChart
34.64
36.48
84.24
60.36
Deplot LLaVA-v16
282M-13B
Ada.
53.44
87.68
70.56
56.80
88.48
72.64
60.64
90.08
75.36
Chart MLLMs
CLarVLM
13B
Ada.
42.08
82.48
62.28
43.84
82.88
63.36
46.00
83.28
64.64
OneChart
125M-13B
1024
54.48
87.12
70.80
57.60
87.84
72.72
62.00
88
75.32
Churtllam
13B
333
58.40
93.12
75.76
61 2.20
93.60
40
63.52
94,00
78.76
ChanCemma-PPoT
3B
@448
67.84
85.28
76.56
68.66
85.84
77 24
69.84
86.32
78.08
TinyChart
3B
@7681
58.72
94.88
76.80
62.56
95.28
78.92
67.04
96.16
81,60
ChartAst
13B
@448
64.88
93.12
79.00
66.24
93.84
80.04
67.44
94_32
80.88
TinyChart+Po
3B
@7681
70.24
90.72
80.48
71.20
91.44
81.32
72.40
92.56
82.48
ChunMoE (Ours)
SB
@490
71.36
91.04
81.20
75.12
48
83 80
16
93.68
85,92
ChartMoEsPPo (Ours)
8B
69440
78.32
90.96
84.64
80.16
92.32
86.24
82.08
93.60
87.84
all open-sourced models. With better numerical extraction abilities, PoT can significantly enhance
our ChartMoE's question-answeringg performance.
EXPERIMENT
4.1 IMPLEMENTATION DETAILS
During the alignment stage, train the connector parameters and keep the visual encoder and LLM parameters frozen. In the supervised fine-tuning stage, we continue training the MoE connector while employing LoRA update the LLM parameters. All training processes are conducted on X A100-40G GPUs You can refer to Appendix A.2 for more details.
4.2 EVALUATION METRICS
ChartQA (Masry et al., 2022) test split consists of 1,250 questions in both human and augmented parts. The charts are three common chart types and are sourced from the real world. It features variety of human-crafted questions and answers to evaluate models' understanding, reasoning, and data extraction skills. ChartQA adopts relaxed accuracy, which highlighted shortcomings by recent studies (Chen al., 2024: Xu et al., 2023), such as simplistic string matching and direct float conversion. Therefore, we improve it by 1) using regular expression matching to extract number values, 2) optimizing string matching for short answers, and 3) demonstrating model performance under various relaxed margins. We adopt it for all experiment results.
ChartBench (Xu al., 2023) focuses charts without data point annotations. includes broader range of chart types, with 9 main categories and 42 subcategories, each containing 50 charts. Chart- Bench focuses on extracting numerical values, posing greater challenge as models cannot depend on OCR for precise answers. It adopts Acc+ for judgments and relaxed accuracy for NQA tasks. The benchmark proposes extract number values by LLMS first, which is omitted for the stratifying istrucciio-followwer ability of ChartMoF.
ChartFC (Akhtar al., 2023a) ChartCheck (Akhtar et al., 2023b) adopt accuracy to verify whether the claim aligns with the input chart, marking significant ad advancement in chart recognition and reasoning abilities. This identifies potential hallucinations in chart-related contexts. The ChartFC test set has 1,591 questions, and the ChartCheck test set has two splits, containing 937 questions and 981 questions.
4.3 COMPARATIVE MODELS
General MLLMs. We compare PaliGemma (Beyer et al., 2024), LLaVA-v1.5 (Liu et al., 2023d) with MLP connector, Qwen-VI. (Bai et al., 2023b) with Qformer (Li al., 2023) connector,

---

Published as a conference paper at ICLR 2025
Table 3: The zero-shot performance ChartBench. No methods fine-tuned on the trainset for faireess.
exclude PoT because ChartBench mainly assesses numerical extraction accuracy without math calculation.
Regular Type
Extra Type
Models
ALL
Line
Bar
Pic
Avg.
Arca
Box
Radar
Seatter
Node
Comb.
Avg.
General ML
Ms
LLaVA-v1.5
29.12
21.26
17.28
22.10
21.73
20.94
27.50
23.47
36.80
24.30
24.96
23.38
Qwen-VL
38.00
20.71
38.24
29.46
28.83
24.17
35.00
19.50
18.50
25.50
26.56
28.18
DocOwl-v1.5
49.60
31.69
31.54
35.68
12.27
23.33
22.50
36.13
29.60
38.80
27.38
32.05
Mini-Gemini
34.88
36.12
40.40
36.77
31.20
23.33
30.60
35.20
43.60
27.90
30.61
34.37
IntemlmXC-v2
68.16
48.74
56.60
54.50
27.47
25.33
40.10
52.93
50.40
46.20
39.72
48.41
Specialist
Chart Models
Pix2Struct
2.56
2.37
1,90
2.33
0.13
0.13
4.60
0.67
0:40
3,20
2.93
2.16
Matcha
6.80
5.05
3.60
5.18
0.27
1.60
6,20
3.46
5.40
4.80
5.81
4.84
UniChart
7.04
5.35
4.30
5.55
3.86
4.80
11.60
5.06
15.80
8.30
6.78
Deplot-LLaVA-vL.6
31.20
26.46
24.00
27.09
21.34
13.34
24.00
41.34
42.00
31.00
31.57
27.62
Chart MILLMs
CL
21.92
14.16
10.50
15.16
7.47
7.87
8.00
7.87
5,40
10.50
8.38
11.96
ChartLlama
26.80
18.83
20.80
20.99
14.27
12.00
24.30
27.73
26.20
25.80
21.71
21.31
TinyChart
32.40
25.81
22.50
26.71
10.13
14.80
13.40
28.14
10.80
21.60
22.56
22.51
OneChart
41.28
30.28
29.60
32.65
19.07
13.20
24.60
38.53
34.80
27.90
31.91
29.93
CharGGemma
50.48
38.21
32.10
39.89
28.27
24.13
28.10
48.00
41.80
43.40
42.47
38.46
ChartMoE (Ours)
71.44
51.57
52.80
56.31
38.40
24.13
40.20
62.67
58.00
49.20
55.58
51.67
DocOwl-v1.5 (Hu al., 2024) that employs multi-level image resolution and token convolution techniques, and the current open-source SOTA, InternImXC-v2 (Dong et al., 2024).
Specialist Chart Models. Previous works specifically design models and algorithms for chart question answering. We compare Pix2Struct (Lee et al., 2023), Matcha (Liu et al., 2023b), UniChart (Masry et al., 2023), and Deplot (Liu et al., 2023a). Notably, Deplot fails to handle questions in arbitrary formats, so we extract table information with Deplot and use LLaVA-v1.6 to answer the questions.
Chart MLLMs. Chart-oriented MLLMs are the promising direction for utilizing prior knowledge of LLMs. CharLLaMA (Han et al., 2023) proposes to generate high-quality instruction data to capabilities. ChartAst (Meng et al., 2024) suggests aligning the
improve chart question connector with chart-table pairs before supervised fine-tuning. ChartVLM (Xia et al., 2024) uses different decoders handle different questions based their difficulty. ChartInstruct (Masry al., 2024a) conducts large-scale chart instruction tuning based on general MLLMs. OneChart (Chen al., 2024) converts the chart to the table with dedicated decoder and uses LLMs to answer questions. CharlGemma (Masry et al., 2024b) proposes more instruction data and achieves efficient chart reasoning based on SigLIP (Zhai et al., 2023) and Gemma-2B (Team al., 2024a). Tiny- Chart (Zhang et al., 2024) adopts token merge to reduce visual tokens and enable high-resolution chart input.
4.4 MAIN RESULTS
ChartQA. Tab.2 presents detailed comparisons of ChartMoE on ChantQA ChartMoE signifi- cantly improves the baseline (InternlmXC-v2) performance (72.00% vs. 84.64%, +12.64%* in Acc. Acc.@0.055. Compared previous SOTA (TinyChart+PoT @768 pixel), ChartMoE consistently surpasses it across all metrics. The PoT effectively enhances the mathematical reasoning capa- bilities, which common shortfall in current MLLMs. ChartMoE integrates better with PoT, indicating that it accurately extracts fundamental elements from charts. ChartMoE shows more sig- nificant improvement on Human part, especially after incorporating PoT. where the questions are more computationally complex and challenging. Notably, our error analysis in the Augmented part reveals that many errors stem from limitations of the evaluation criteria, i.e., string matching. For instance, it is marked incorrect if the prediction is It is between 2003 and 2005 and the ground truth is (2003, 2005). Forcing performance improvement may lead to model overfitting.
ChartBench. Tab. 3 presents detailed comparisons of ChartMoE on ChartBench. None of the models, including our ChartMoE, undergo supervised fine-tuning on the ChartBench trainset to ensure fair experimental comparison. Chart-specific models typically underperform due to limited generalization, which fails to manage the annotated charts effectively (< 10%). Deplot shows a distinct advantage over these types of models (27.62%) with the assistance of L.aVA-v1.6. The

---

Published as a conference paper at ICLR 2025
baseline (InternlmXC-v2)) demonstrates strong generalization ChartBench (48.41%) which may benefit from pre-training instructions designed for unannotated charts. Without additional design, ChartMoE improves the baseline performance comprchensively (48.41% vs. 51.67%), especially extra chart types (39.72% vs. 55.58%, + +15.86%1).
ChartFC & ChartCheck. Tab. 4 compares ChartMoE the synthetic ChartFC and real-world ChartCheck. ChartMoE consistently outperforms SOTA (e.g.. ChartGemma +4.4% on ChartFC) and significantly improves the performance compared InternlmXC-v2 (+6.83%1 and +8. 1 on ChartCheck TI and T2, respectively). Note that this is implemented without using training data for supervised finettuning, demonstrating ChartMoE's strong generalization capabilities.
Models
CharFC
ChanChuck
Randam
IXC=22
Cu-lipeydl:
Radar
Tañe
PaliCiemma
S8.26
67.34
68.50
CharMMBl
Pic
JSON
IntemimXC.v2
LLaVA-vi.5t
61.28
65.93
72.04
70.22
70.44
70.03
g.
Node
Line
Charllsstuct-
Comb.
Chartinstrcc- Mantinstrucc-FaanISSS SXL
70.27
72.03
73.80
ChariGemma
70.33
71.50
74.31
ClunMol: (Ours)
74.73
78.87
79.20
Are
40%5
600
1025
Table 4: The accuracy performance ChartFC and ChartCheck.
Ieemtion
Figure 4: Training loss of dif- ferent initialization.
Figure 5: Top-2 selected expert distri- bution on ChartBench.
ing with ChartGemma instructions.
5 FURTHER STUDY
5.1 MODEL ARCHITECTURE ABLATION
We investigate the impact of three factors on our MoE connector: the number of experts, the number of activated experts, and the expert initialization manner. All the experiments conducted with ChartQA training data and evaluated on ChartQA test split with relax accuracy metric.
Effect of the Expert Initialization Manner. The initialization strategy plays crucial role in de- termining the performance of the MoE connector. Effective initialization is essential to ensure that each expert performs its designated function optimally. As illustrated in Tab. 5 row 1-3, ex- plore the impact of 3 initialization strategies for the MoE connector. Random initialization serves as a baseline but struggles with convergence (refer to Fig. 4), resulting in a suboptimal accuracy of 73.20% at Acc. @0.05. Following CuMo (L.i et al., 2024), we employ the Co-Upeycle strategy by replicating the table-ISON- approach lacks expert diversity, which limits its effectiveness, resulting in accuracy of 77, 77.48% at Acc. @0.05, contrast, our initialization assigns distinct parameters to each expert. This tai- lored approach enables each expert to capitalize its specific strengths, resulting in the highest performance, achieving 78%76% Acc. Ac0.@..05.
aligned connector for all experts. Given the same starting point, this
Effect of Number of Experts and Activated Experts. As shown in rows 3-4 of Tab. 5, we com- pare ChartMoE configurations with 4 and 8 experts, keeping 2 experts activated. The 8 experts initialized in pairs using the methods illustrated in Fig. 2c. ChartMoF achieves 78. 78% in Acc.@0.05 with 4 experts, which is slightly higher than the 78.60% achieved with 8 experts, show- ing a marginal increase of of.00.155 16%. In rows 4-5, we compare the performance of configurations with 2 and 4 activated experts, finding similar results: 78.60 33.60% vs. 78.64% in Acc. @0.05. This analysis suggests that merely increasing the number of experts or the activation experts does not guarantee improved performance. The configuration with 4 experts and activated experts effectively balances complexity and performance, making a suitable choice for ChartMoE.
5.2 TRAINING STRATEGY ABLATION
We analyze the impact of the training strategy across alignment and supervised fine-tuning stages. We use InternlmXC-v2 with ChartQA fine-tuning rameters as the chart-specific annealing tuning stage.
baseline, maintaining the same hyperpa-
Effect of Alignment Strategy. As shown in 1-3 of Tab. 6, translating the chart image into structural text formats such as table, JSON, and code during the alignment stage signifi- cantly enhances performance in downstream chart understanding tasks. After applying table-ISON.
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study on ChartMoE architecture w.r.t. the total activated initialization of connector experts. All experiments are conducted on ChariQA.
Table 6: Ablation study on the proposed training
strategy and connector architecture on the align- ment, high-quality knowledge learning, and chart- specific anneal tuning stages.
Total
Activated
Espasts
Relax 8008
Expens
Exgerts
Initialization
Human
Aug.
Avg.
ChartMol Recipe
Relax Acc 80.05
Random Init.
59.68
86.72
73.20
Human
Aug.
Avg.
2
Randoen Align
62.32
88.88
75.60
Baseline: IntemlinXC-22 CharQA
63.68
87.68
75.68
Co-Upeycle Init.
64,96
90.00
77.48
tuble Boole-JSON--cooe Aligned Connector
64.24
90.16
77.20
Diversely Align
Topp--in44 ChurtMoB
67.92
78.76
Divensly Align
67,20
90.00
78.60
High-Quality Knowladge Learning
79.04
Divensely Align
67,68
89.60
78.64
ChariGioama Instructious
71.36
91.04
81.20
Table 7: Ablation study on the expert of MoE connector. We ignore the gating net- work and adopt specific expert output.
Table 8: Ablation study on alignment pre-training tasks. We adopt different alignment tasks for baseline (linear connector) and fur- ther conduct supervised fine-tuning on the CharlQA train set.
Relax Ace @0.05
w/o CharQQA SPT
wi ChamQA SFT
Connector
Alignment
Human
Aug.
Avg.
Humam
Aug.
Avg.
Human
Aug.
Avg.
Expert (Vanilla)
69.76
89.84
79.80
Vanilla
62.72
81.28
72.00
63.68
87.68
75.68
Expert (Table)
63.60
89.12
76.36
Table
51.28
71.76
61.52
63.92
89.28
76.60
Expert 2 (SSON)
60.64
82.48
71.56
JSON
44,40
65.12
54.76
64.88
89.84
77.36
Expert (Code)
66
89.36
78.12
Code
50.16
71.20
60.68
90.16
77.20
mot Ray
30
0 84
0.0
P
Balo
8
dP
Papert iniralival with ziver endo Figure Visualizations of top-1 expert selection. Only the boundaries of the merged tokens are plotted.
Pxpen minsool with IntemimXC-x0 w0
Pepert intioolood with chstt cable
Expen intillionl with chert JSCI
code alignment, the model achieves 77.20% in Acc. (cc.@0.05, representing notable improvement (+ (+1.52%1). When combined with our proposed MoE connector, performance further increases to 78.76% total gain of +3.08% in Acc. @0.05.
Effect of Supervise Fine-Tuning Strategy. As shown in rows 4-5 of Tab. 6, we divide the super- vised fine-tuning stage into two phases. By incorporating high-quality knowledge learning using the MMC dataset, ChartMoE achieves 79.04% Acc. @0.05, reflecting 3.36% improvement. In the chart-specific annealing tuning phase, we introduce ChartGemma data enhance the model's reasoning and PoT capabilities, leading the model to peak performance (81.20%, +5.52%1).
5.3 IN-DEPTH ANALYSIS
Effect of the Each Expert. To explore the role of each expert in ChartMoF, we bypass the gating network and manually select the output of specific experts. As shown in Tab. 7, EO performs the best (79.80%), which is consistent with the distribution Fig. 5. However, this doesn't mean other experts lack relevance, which may offer crucial insights at key moments (Fig. 6).
Effeet of Alignment Task. As shown in Tab. 8, we explore various alignment tasks based on the linear connector. After alignment, the performance ChartQA declines compared to the baseline. However, the aligned model exhibits substantial improvement after supervised fine-tuning on the ChartQA train split, which is consistent with previous observations (Meng et al., 2024; Yan et al., 2024). Specifically, the JSON and code tasks exhibit remarkable improvement over the table.
Expert Distribution Visualization. As shown in Fig. 5& 6, we visualize the expert distribution in the MoE connector on the ChartBench test set. We designate the vanilla connector as E0, while E1-3 corresponds to connectors aligned with tables, JSON, and code. As depicted in Fig. 5, the

---

Published as a conference paper at ICLR 2025
220
3000
Scores
200
Direct SFT
ChartMoE
IXC-v2
Direct SFT
ChartMoE
IXC-22
2500
2000
160-
1500
140
120
1000
100
500
Color Existence
Landmark
Scane
Position
OCR
Artwork
Posters
Celebrity
Count
commonsanse
Code Calculation Transtation
Perception
Cogrition
Total
Figure 7: The performance on the general VQA tasks (MME Fu et al. (2023)). With supervised fine-tuning extensive shart-structured data, the directly tuned IXC-v2 shows significant performance drop, while
ChartMoE maintains a satisfying performance by keeping the vanilla connector as the expert MoE.
100
Discc: SPT
CharbleB
IXC-22
Duant
Sare
Don't
70.28
Pecorooge 150
60
50
CR
9.9 9.
systom Oe Waneltt velc
kondar Upr1" ve-
Figure 8: The performance on the general VQA tasks (MMBench Liu et (2023e)). Please refer to its paper for each task's details. The observations and conclu- sions are consistent with the MME benchmark.
Figure 9: The performance with/without bz-loss on ChartQA. Left: The bz-loss lcads to more even ex- pert selections. Right: A more balanced distribution does not yield better performance.
trend is consistent across different chart types, with EO and E3 being the most frequently selected connectors, The expert selection shows no extreme bias, as even the least chosen, EI, accounts for over 10%. We further visualize the expert selection for each image token, revealing the preferences of each expert. As shown in Fig. 6, EO is the primary choice for background tokens, explaining its dominance in Fig. El and E2 are more frequently chosen by tokens from titles, axis labels, or legends, as these elements are commonly found in tables and JSON files. ChartMoB tends to use E3 to focus on the data points and visual elements within the chart, e.g., data points the line, digital text, and edges in node chart. These components are essential for accurately re-drawing the charts.
Performance on General Tasks. While ChartMoB designed enhance chart understanding. it does not compromise other capabilities, e.g., instruction following and object recognition. In Fig. 7& 8, we show the comparisons of directly fine-tuned InternlmXC-v2 (short for Direct SFT) with data from Tab. and the baseline (short for IXC-v2) on general benchmarks (Fu et al., 2023; Liu et al., 2023e). The direct SFT model shows diminished general capabilities. In contrast, ChartMoE preserves it nearly intact by retaining the original connector one of its experts.
Effect of Balance Loss in MoE. The standard MoE (Zoph et al., 2022) employs balanced loss and router z-loss (short for bz-loss) to prevent certain experts from dominating the model training. In Fig. 9, we compare the effects of with and without bz-loss. While bz-loss promotes a more equitable selection of experts, it fails enhance ChartMoE's performance further. As shown in Fig. 6, the expert initialization in ChartMoE results in each expert having its own preference for visual token selection (refer to Appendix B for detail). Consequently, the bz-loss might hinder the model's convergence to the optimal point because the distribution of visual tokens is inherently imbalanced.
6 CONCLUSION
We introduce ChartMoE, a multi-task aligned and instruction-tume MLLM designed for complex chart understanding and reasoning. We replace the linear connector with the MoE architecture and initialize each expert with parameters derived from different alignment tasks. We further present the ChartMoB- Align dataset, a synthetic collection of nearly million table-json-code--hart quadruples, to facilitate alignment training across different experts. This approach preserves the strengths of each alignment task, ensuring efficient training and superior model performance. ChartMoE outperforms the previous state-of-tt -fet on several benchmarks by large margin and excels in real-works ap- plications such as chart question answering, translation, and editing. Please refer to Appendix A.3 for the reproducibility statement.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work was supported by the National Key R&D Program of China (2022YTB4701400/477144 KJZD200339923 14916032, GJHZ220240221111366000 08), and Beijing Key Lab of Networked Multimedia.
2), SSTIC (raatKKIZD20230031155
REFERENCES
Mubashara Akhtar, Oana Cocaraseu, and Elena Paslaru Bontas Simperl. Reading and reasoning over chart images for evidence-based automated fact-checking. arXiv preprint:2001. 2023a.
Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu, and Elena Simperl. Chartcheck: An evidence-based fact-checking dataset over real-world chart images. arXiv preprint:2311.07453. 2023b.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. Flamingo: visual language model for few-shot learning. proceedings of NeurlPS, volume 35, pp. 23716-23736, 2022.
Alibaba. Qwen-vl 2.5: multimodal large language model. https: .awpyyllllll.... qianwen, 2024, Accessed: 2024-09-17.
al aliyun, com/
Jinze Bai, Shuai Bai, Yunfei Chu, et al. Qwen technical report. arXiv preprint:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, et al. Qwen-vl: A frontier large lasion- model with versatile abilities. arXiv preprint:2308. /2966, 2023b.
Lucas Beyer, Andreas Steiner, André Susano Pinto, al. Paligemma: versatile 3b vlm for transfer. arXiv preprint:24007.07726 2024.
Tom Brown, Benjamin Mann, Nick Ryder, al. Language models few-shot learners. In pro- ceedings of NeurlPS, volume 33, pp. 1877-1901, 2020.
Victor Carbune, Hassan Mansoor, Fangyu Liu, et al. Chart-based reasoning: Transferring capabili- ties from Ilms vims. In proceedings of NAACL, 2024.
Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Onechart: Purify the chart structural extraction via auxil- iary token. arXiv 2reprint:2444009990 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, et al. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv 2reprint:231000944: 2023.
Tri Dao. FlashAttention-22 2: Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations (ICLR), 2024.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, et al. Internlm-xcomposeer Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint:2401. 16420, 2024.
Chaoyou Fu, Peixian Chen, Yunhang Shen, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint:2306, 13394, 2023.
Yucheng Han, Chi Zhang, Xin Chen, et Chartllama: A multimodal Iim for chart understanding and generation. arXiv 2reprint:2311.66222 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. proceedings of CVPR, pp. 770-778, 2016.
Enamul Hoque, Vidya Setlur, Melanie Tory, and Isaac Dykeman. Applying pragmatics principles for interaction with visual analytics. IEEE TVCG, 24(1):309-318, 2017.
Ting- Yao Hsu, C Lee Giles, and Ting-Har Hao' Kenneth' Huang. Scicap: Generating captions for scien- tific figures. In Findings of ACI, 2021.
11

---

Published as a conference paper at ICLR 2025
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docow/ 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint:2403.1288 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, al. LoRA: Low-rank adaptation of large language models. In proceedings of ICLR, 2022.
Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih- Fu Chang, and Heng Ji. From pixels to insights: A survey on automatic chart understanding in the era large foundation models, 2024.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, et al. Sparse upcycling: Training mixture- of-experts from dense checkpoints. In proceedings of ICLR, 2023.
Kenton Lee, Mandar Joshi, Iulia Raluca Ture, et al. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In proceedings of ICML, pp. 18893-18912, 2023.
Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal Ilm with co-upeycled mixture-of-exxe -f-eeertss...........ee arXiv preprint:2405 240500999, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In proceedings of ICML, volume 202, pp. 19730-19742, 2023.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947 2024.
Ziyi Lin, Chris Liu, Renrui Zhang, et Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preppin::3311.07555, 2023.
Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, et al. Deplot: One- shot visual language reasoning by plot-to-table translation. In Findings of ACL, pp. 10381-10399, 2023a.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, et al. Matcha: Enhancing visual language pre- training with math reasoning and chart derendering. In proceedings of ACL, pp. 12756-12770, 2023b.
Fuxiao Liu, Xiaoyang Wang. Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Ya- coob, and Dong Yu. MMC: advancing multimodal chart understanding with large-scale instruc- tion tuning. In proceedings ACL, 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In proceed- ings of NeurlPS, 2023d.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:// Llava-vl.ge thub. lo/blog//024 00-30 llava-nex:
Haotian Liu, Chunyuan Li, Yuheng Li, et al. Improved baselines with visual instruction tuning. In proceedings of CVPR, 2024b.
Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, and Ying Shen. Chartthinker: A contextual thaoughf- approach to optimized chart summarization. arXiv 200p 12336, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, al. Mmbench: Is your multi-modal model all- around player? arXiv preprint: 2023c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseck-vl: towards real-world vision-lane languuge understanding. arXiv preprint:2403.0555 05525, 2024.
12

---

Published as a conference paper at ICLR 2025
Ahmed Masry, Do Xuan Long, Jia Qing Tan, et al. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In proceedings of ACL, 2022.
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. Unichart: A uni- versal languale pretrained model for chart comprehension and reasoning. proceedings of EMNLP, 2023.
Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning. arXiv preprint:2403..0008.. 2024a.
Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shatiq Joty. Chartgemma: Visual cutructiio-tuning for chart reasoning in the wild. arXiv preprint:2407.04172. 247.2400..42..222. 2024b.
Fanqing Meng, Wenqi Shao, Quanfeng Lu, et al. Chartassisstant: universal chart multi- modal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint:2401.02388. 2024.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. proceedings of CVPR, pp. 1527-1536, 2020.
OpenAl. Gpt-4 technical report. arXiv
2023.
OpenAl. Gpt-4o: A multimodal large language model. https: com, 2024. Accessed: 2024-09-17.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, al. Improving language under- standing by generative pre-training, OpenAl blog, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, al. Learning transferable visual models from natural language supervision. In proceedings of ICML, pp. 8748-8763, PMLR, 2021.
Gemma Team, Thomas Mesnard, Cassidy Hardin, et al. Gemma: Open models based on gemini research and technology. arXiv preprint:2403.0 08295, 2024a.
Gemma Team, Thomas Mesnard, Cassidy Hardin, et al. Gemma: Open models based on gemini research and technology. arXiv preprint:2407.227788 2024b.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of CVPR, pp. 9568- 9578, 2024b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint:2302.13377,, 2023.
Peifang Wang, Olga Golovneva, Armen Aghajanyan, al. DOMINO: dual-system for multi-step visual language reasoning. arXiv preprim::2110.2288 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. Chain-of-thought prompting elicits reasoning in large language models. In proceedings of NeurIPS, volume 35, pp. 24824-24837, 2022.
Yifan Wu, Lutao Yan, Yuyu Luo, Yunhai Wang, and Nan Tang. Evaluating task-based effectiveness of milms on charts. arXiv preprint:2405.07700 2024.
Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, et al. Chartx chartvlm: A versatile bench- mark and foundation model for complicated chart reasoning. arXiv preprint:2402.1 12185, 2024.
13

---

Published as a conference paper at ICLR 2025
Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint:2312.15915, 2023.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael Ryoo, al. xgen-mm (blip-3): A family of open large multimodal 2024.
models. arXiv 24eprini:2408.480
Pengyu Yan, Mahesh Bhosale, Jay Lal, Bikhyat Adhikari, and David S. Doermann. Chartreformer: Natural language-driven chart image editing. arXiv preprint:2403.0m 2403.0009, 2024.
Jiabo Ye, Anwen Hu, Haiyang Xu, et al. Ureader: Universal ocr-free visually- situaate language understanding with multimodal large language model. In Findings ACL,, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint:2344141172 2023b.
Qinghao Ye, Haiyang Xu, Jiabo Ye, et al. mplug-ow12: Revolutionizing multi modal large language model with modality collaboration. arXiv preprint:2311 1.04257, 1.00257 2023c.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In proceedings of ICCV, pp. 11975-11986, 2023.
Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and program-of-thone learning, arXiv preprint: 2404. 16635, 2024.
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, et al. Internlm-xcomposer:: A vision-language large model for advanced text-image comprehension and composition. arXiv preprint:2309.111 15112, 2023.
Susan Zhang. Stephen Roller, Naman Goyal, et al. Opt: Open pre-trained transformer language models. arXiv preprint:2205000066 2022.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, al. Judging llm-as-a-judge with mt-bench and chatbot arena. In proceedings of NeurlPS, 2023.
Li Zhuowan, Jasani Bhavan, Tang Peng, and Ghadar Shabnam. Synthesize step-by-step: Tools, templates and 1lms as data generators for reasoning-based chart vqa. In proceedings of CVPR, 2024.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam M. Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint:2202.089088 2022.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTAL SETTINGS AND RESULTS
A.1 TOP-2 EXPERTS DISTRIBUTION
Our ChartMoE employs MoE connector expert parameters initialized with various aligament tasks. To in- vestigate the impact of these initialization methods on model performance, we present the comparisons in Tab. 6& 7& 8 and Fig. 4&5. For deeper analysis, we explore how different initialization methods affect expert selection. As shown Fig. 10, both random initialization and co-upeycle result more uniform distribution of experts. However, this uniformity does not inherently lead to improved performance or inter- pretability, possibly due to insufficient differentiation among the experts. In contrast, our ChartMoB clearly prefers specialized roles, as illustrated in Fig. 6& 11& 12.
Exper
Exper
EXt122
DNC-V
Tak
Neto
2can2
Frpet
S
2008
Dat
Dut
Dut
Dug
18/55
69%
(a) Random
(b) Co-Upeyele
(c) ChartMoE with bo-loss
(d) ChartMoB
Figure 10: The distribution of Top-2 experts after supervised fine-tuning with three expert initialization meth- ods. We calculate the proportion of the top experts selected by the router on the CharlBench.
A.2 SUMMARY OF HYPERPARAMETER SETTINGS
The training process of ChartMoE is structured into three distinct phases: Alignment Pre- trainingg High- Quality Knowledge Learning. and Chart- Specitie Annealing Tuning. Table 9 provides a comprehensive overview of the hyperparameter configurations employed during each training stage. It should be noted that using flash-attention (Dao, 2024) can effectively reduce GPU hours.
Table 9: Training hyperparameters of ChartMoF for all stages.
Configuration
Alignment Pre-training
Iligh-Qualiy Knowledge Learning
Chart Specilic Anncaling Tuning
Connector Initialization
InteralmXC-22
Table& Experts InterninXC-v2
ChartMol: 2nd-stage
Training
Freeze
T.RA
Image Resolution
490
490
490
VIT Soquence Length
1225
1225
1225
Optimizer
Adamn
Adam
Adamm
Optimizer Hypeqparaecter 8, 0.9, 3g
0.95, 16-8
81 0.9, 3g 0.95, 1e-5 Se-
3, 0.9, 0g 0.95, 16-9 le-t
Peak Learning Rate
Se
Learning Schedule
cosine decay
cosine decay
cosine decay
Weight Decay
0.1
0.1
0.1
Gradicat Clip
1.0
1.0
1.0
Warm up Ratio
0.01
0.01
0.01
Global Batch Size
256
64
64
Gradient Acc.
Numerical Procision
blooat16
b6
blloat16
Optimizer Sharding
Gradient Sharding
Parameter Sharding
Activation Checkpointing
GPU Hours (A 100-40G)
210
100
A.3 REPRODUCIBILITY STATEMENT
We have included the architecture of ChartMoE in Section 3.1 and the complete training procedure in Section 3.3. The training data recipe is listed in Tab. in detail. Hyper-parameter settings are shown in Appendix A.2. We also introduce the generation pipeline for ChartMoE-Align in Section 3.2, and some detailed examples in Appendix C. Furthermore, our ChartMoF-Align dataset and checkpoints of ChartMoF. will be released soon on GitHub and Huggingface.
3.2 and Section
15

---

Published as a conference paper at ICLR 2025
B ADDITIONAL VSUALIZATIONS OF TOP-1 EXPERT SELECTION
In this section, we randomly sampled images from natural image datasets (LLaVA-CCC) (Liu et al., 2023d)) and chart datasets (ChartQA (Masry et al., 2022), ChartBench (Xu et al., 2023)) illustrate ChartMoE's token selection preferences. As shown Fig. 11, the vanilla expert focuses more on the background, the table expert concentrates on dctails such as the boundary between the background and the subject, the JSON expert focuses on textures (e.g., maps and objects). and the code expert specializes in curves and trends (e.g.. logos and tex(). Fig. 12 further demonstrates that while the vanilla expert continues to atlend to background tokens, critical visual elements are handled by the aligned experts, with the code expert being notably more prominent.
Doo
EO VEWOOM
TR892 HT
EEMETTEE
DUROW BESNEUE EO
0
20020
0
YOURE BARRED
OP
sttck
ZIRRET TH A2R0 EEM MOIZENOR WT
TIEN
LENAAR GERARDE TRUDI HE
Waandaayo Toou gUAP
00
00
alamy stock photo
9 5
Expert mitialioa with Xrir2
Peperi inttialined with to
Pepert intisstined with chorr ISON
Expert inviolional with chart code
Figure 11: More visualizations of top-1 expert selection general images randomly sampled from LLaVA- CC3M. These examples show the selection preferences different experts in ChartMoE.
16

---

Published as a conference paper at ICLR 2025
Ind
Refalm
8
1009
2011
2012
46 a
80 000
C
0.0
[5c
0 0
9P STP
Comeertt
2
1
0
0.00
JP
1d
cl
65
D
0
0 D0mB
Tbb
0 iemma
0g
D
0.6
d
f
y
0
d
D
20
6
4
E
intollzeed with code Figure 12: More visualizations of top-1 expert selection on chart images. The vanilla expert primarily handles background tokens, and the chart visual markers are handled by other experts.
Intialened Inver
Eispo initialied chort
intiislied with chort ISUN
17

---

Published as a conference paper at ICLR 2025
C DETAILS OF CHARTMOE-ALIGN
C.I OVERVIEW
ChartMoF-AIl Aligg is a datasct we introduced for different experts aligning pretraining. It consists of nearly million Chart Table JSON Code quadruples and supports three alignment tasks: Chart to Table. Chart to ISON, and Chart to Code. Unlike other chart datasets, ChartMoE-Align focuses solely these three fundamental alignment tasks without considering the diversity of instruction tasks.
C.2 TABLE DATA COLLECTION
We primarily collect table data from three sources: the ChartQA training set (Masry et al., 2022), the PlotQA training set (Methani al., 2020), and ChartY provided by OneChart (Chen al., 2024).
ChartQA includes 18.3K real- world charts and provides accompanying meta tables. While the charts are of high quality and manually curated, they lack fine-graing line-grained attribute annotations and executable plotting code. As a result, we only retained the tables from ChartQA in CSV format.
PlotQA comprises 157K charts, primarily focusing on three common types: line, har, and pic charts. These charts are generated using Python code with limited formatting and style diversity. Consequently, we did not utilize the charts from PlotQA but retained its 157K tables. These tables originate from sources like World Bank Open Data, Open Government Data, and the Global Terrorism Database, covering statisties on various indicators such as fertility rates, rainfall, coal production, and more across years, countries, and districts.
YhartY is a chart datasct containing 2.7M charts in both Chinese and English proposed by OneChart. Notably, YhartY also includes charts from ChartQA and PlotQA, which filtered out in ChartMoE-Alignn Addition- ally, ChartY primarily consists of common chart types such line, bar, and pie charts (or their combinations) and suffers from significant data imbalance. To address this, we sampled subset ensure roughly equal number of charts for each type. As the tables in ChartY are mainly generated by GPT-3.5 based on templates, we ultimately retained 763K samples from this source.
C.3 PAIR DATA CONSTRUCTION
JSON provides structured format distinct from CSV, designed to retain chart attributes beyond numerical data, such as title position, font size, element colors, legend styles, and more. We adopt the template provided by ChartReformer (Yan et al.. 2024) and further enhance it. We add chart type-agnostic attributes like title position and gridlines. For chart type-specific attributes, we aim to remain consistent with ChartReformer's definitions while accommodating all chart types present in ChartMooE-Align. With this framework, generate corresponding JSON files for all tables. extract chart type-specific attributes, we fine-tune a Deplot (Liu et al., 2023a) model, leveraging the original chart to extract their properties. Missing attributes are filled in using random sampling to ensure completeness.
Code refers to Python scripts based on matplotlib for rendering the charts. Leveraging the rich attributes defined in the JSON, the code is designed faithfully represent every attribute ensure diversity in the resulting charts. During generation, we explicitly specify all default parameters, such as the hexadecimal color codes each line/bar, default font sizes, text positions, etc. We provide basic code templates for type- agnossic attributes. For type-specific attributes, rules are used to automatically generate the corresponding code.
Chart is produced by executing the generated code. Given the number of table, JSON, and code pairs, we filter out any quadruples with execution errors or warnings during the chart generation process, retaining only valid and error-free samples.
Considering the alignment task, we directly employ several templated questions to define the Chart-to-X tasks (X is the ground truth). Ultimately, each quadruple corresponds to three QA pairs. Note that ChartMoFFAllin only serves for alignment training to initialize different expert projectors, thus emphasizing the diversity of charts and aligned modalities. To improve model performance and iustruction-followee still require more diverse instructions for supervised fine-tuning to update the MoE connector and LLM.
Instruction.
C.4 QUALITY CONTROL
We first remove all duplicate entries from the meta table and then eliminate quadruples that cause errors or warnings during rendering. To further assess the quality of ChartMoF-AAlgn, ples and ask GPT-4o and annotation experts (with least three experts reviewing each quadruple) to evaluate the clarity and readability
randomly sample 200 quadru-
the charts, as well as the alignment hetween the charts and table/ISON/code, scoring them as 0 or 1. The results show that nearly all charts are clear, unambiguous, and free from obstruc- tions (GPT-4o: 96.5%, Experts: 99%). Over 90% of the pairs are matching and suitable for instruction tuning (GPT-4o: 91% 1. Experts: 94.5%)
18

---

Published as a conference paper at ICLR 2025
C.5 EXAMPLE VISUALIZATION
(a) Charts in ChartMoE Align.
Cheracterietto
wtt.tr.ttmaaatttttoooo teals Hoal
www.ettrtttt..... Teeale
charactert stttc "Deaths per 1,000
Characteriate Dew.eettt, Interrat
2878 24833 188333
2210. 662 766
Soma Lay 38:85
7877-20 Cir 1,50
2015-24713
2017-402
Pour stonib, Comores, 7.21
2013 25.4-
21161 70656
461, 155
22235. 7738
2515. 452, 378
Sudan, 7.30
201) 203 8. 6.2%,2-55
2015e, 27912 0858
2233, 258, 351
Tunisio, 5.28
2820. 1. 54,
28144, 18987 -1122
2317- 161-
Temen.S-00
-8y-1-35
28174, 70015, (331
7318. 161 154
Syrte,5-11
F817.9 S5,55.7.
2381.
Libya,5.1
2015,2-3N, -2.25
2826, 21651, 4715
7087. 133 106
1,78
7.1 38,8.
2200 404,
Algeria
8013,8-7,-7-20
2AST, 2513
1.36
2026, 20015, 5212
2384, 425, 200
3..dan.3.89
1011, 40.1,,4.1
2282. 287
Vest Bunk and 2.29,
2281,181,
41241112-77
Bohrsin,2.35
gnited Arso Earrates,1.47
(b) Tables in ChanMoE-Align.
"Type agnsctie";
"yye spavttiets "Kdtrt starge
trya agpersse's
"oufent sleef: "a-larg""
"C aiees Marges
Test stief' "anlarge"
Test sue's tentaree",
resstionn
LLA "Sopens 1557 "oaer conter",
"upens, "Souer Vett", sonter
lagend "szacr
elnppe reales
"Maile, Fart rene": "moroopace",
TEiite, feet_aane' *ra303pace" "yrad Si57t Lras,
Feltla, fort Janel: "Sar21"
"eatle, taae "Sari"", "youl Yi57: Ink
Tyeas maatt leibs,
"grad Fut leloo.
"orte
"prid Strret atryaa "socud', To01" "sploc"
1or2 untnee Reapar"
"prii afteett "aaeer Lire_atlee' "Sanef 200_5 "dffeit""
tengatyees "wolue, Seat's "dcfact""
"Mated"
"vaa_Tast : "spplae"
rcolernane Bileee,
"colomep": "eurto,
coolomar' 1 "plaaes',
Scalormpph: sctolete"
"uldge"s "cariee"
"atylee:
"ullge" "cator"
"alggrs "castor"
Maycurt
Mayurt's
"plat_lamate::
pplet_ Laseltt II
labelatt "Saenttte" iieat"
Macctree",
GGrsaner Mulitheare
ISON' in ChanMoB-AAign, JSON is combined with table during aligoment pre-training.
pyplot pit
caport rarpy
sepoir ranpy an ep
yis too:
a via 1op3 (oophart)
12010" 20171, 12000e", (pease,, 120144', '2012"' 120120 2001"",
12000,
2005, 2000, 2007, 2000, 3000, 2010, 2011, 2012, 2013, 2024, 2015, 201)
Taner 2000r. uree 209011
Sasan, 1618), Seill. pa90o 1854, 1958, 3189, (808, 1335, dist, 1731, 548), 2186, 19911
2000, 23834,
191), 1153, 200, 186, Day, 165, 197, 103, 199, 151, 153, 183, 171,
401, 321, AE,,
att. ylll, www..hhaaeatt and vacclneet, cston//massss'', hatcni'a, cectert! c. plsl, tectampp.uulllii mnaett, labeleeCrrouerr Sealtheere", colore vebxads",
Taset/766le" catar: "eidSaa2", sorkerste', sarbersteell, stl.
Loaettyle att.ttoci,yyss. Lirouatioe.
1.4)
43312200" nerborets',, sareessaa00
conter")
the ri0k x/8 /tt.yslckkt/ssttitee esal11)
set The wwwyreeeetfootiiiee
o-1a6111)
let.legea lattl, reali),, fuetcion./evelie))
sct 2000l lugend
uut de g00ul logend
amal-, ferretre (1
stt.g
Mttgriee-sstbbarrrrr whiche novor', "anstyter'sttidt, ada
resazE Lmege by Light (ieeoti
resaze angge by (ayoo)))))) stb-tigte Layoe()
2ik.wwefigl oolpet.ppgg
(d) Codes in ChamMonF All values and attrihutes are expressed explicitly.
Figure 13: Detailed Examples in ChartMoE-AAlign. Each quadruple contains the chart, table, JSON and code.
19

---

Published as a conference paper at ICLR 2025
D FURTHER DISCUSSION
D.1 CONTRIBUTION OF CHARTMOR
Some prior work, such as MoE-LLaVA (Lin et al., 2024), DeepSeck-VL (Lu al., 2024), and CuMo (Li et al., 2024), has employed MoE architectures in MLLMs. However. these approaches all apply MoE to LLMs or ViTs to increase model capacity, introducing large number of learnable parameters to boost performance. In contrast, our ChartMoE introduces several distinctive innovations:
enhance the model's chart comprehension through alignment tasks while preserving performance on other general tasks. Hence, we retain the original connector parameters as one expert initialization manner.
Motivation:
Our goal not to expand model capacity
2) Initialization: Unlike previous methods that rely random co-upeycle initialization, leverage multi- ple alignment tasks for expert (connector) initialization. This approach enables ChartMoE to exhibit remarkable interpretability (Fig. 11& 12).
3) Complexity: We are the first to apply MoE exclusively to the MLP connector (projector) in LLaVA-like MLLMs. In ChartMoE (based on InternlmC-v22,, the MoE architecture introduces minimal additional param- eters (model size 8.364B + 8 8.427B, + 63Mf only) and training complexity (Fig. 4). It also shows negligible impact inference speed (0.945 0.952 seconds per QA on ChartQA test set) and peak memory usage (23.72
GB + 23.86 fp on A100-40G GPU).
D.2 CHARTMOE BASED OTHER MILMS
Our ChartMoB based on InterlmXC-v2, but our proposals (MoE connector, diverse alignment, etc.) are general approaches. Therefore, we use 10% of the alignment data (Tab. 1) and the ChartQA training data to train proposals based on LLaVA-v1.5-7B to further demonstrate their effectiveness. As shown in Tab. 10, proposals significantly improve the base model. This is partly because LLaVA is trained with fewer chart data, leading a lower baseline, and indicates that the additional alignment data greatly enhances chart understanding.
Table 10: Performance comparison ChanQA with LLaVA-v1.5-7B base MLLM.
Relax Ace @0.05
Relax Ace @0.10
Relax Ace 600,20
Models
Human
Aug
Avg
Aug
Avg
Aug
Avg
v..-7-v
7.60
7.36
7.48
7.92
8.08
8.00
9.04
9.52
9.28
LLaVA-v1.5-7B- CharQA
6.08
23.04
14.56
8.24
32,96
20.60
10.32
42.16
26.24
LLaVA-v1.5-7B- ChartMoE
18.13
32.11
25.12
20.20
42.32
31.36
24.24
52.12
38.18
D.3 PERFORMANCE ON CHARTQA
Tab. 2, our ChartMoE significantly outperforms SOTA. However, some models perform better than ours on the Augment part of the ChartQA test set. Given that the Augment part of ChartQA is considerably casier than the Human part, conduct more detailed analysis. We analyze the performance of various models on numeric (Human: 43%, Augment: 39%) and non-numeric (Human: 57%, Augment: 61%) questions. As shown Tab. 11, ChartMoF: excels all subcategories except for non-numeric questions in the Augment part. find that ChartMoE's errors primarily occur in string-matching tasks. For instance, a prediction of ft is between 2003 and 2005 is marked incorrect if the ground truth is (2003, 2005). High accuracy in this category may indicate overfitting instead.
Table 11: Fine- Fine-grained performance comparison on ChartQA with error margin 5%.
Human
Augment
Method
Acc
Numerie
Non-Numeric
Avg
Numeric
Non-Numeric
Avg
CharChart
58.52%
58.03%
58.24%
92.43%
96,25%
94.32%
28%
ChurtAst
67. 57.04%
65.35%
66.08%
93.20%
93.07%
93.12%
19.00%%
ChartMoR (Ours)
73.89%
75.49%
74.80%
93.20%
90.98%
91.84%
84.64%
D.4 LIMITATIONS
ChartMoE has two limitations: 1) Dependency on alignment tasks. ChartMoE requires chart- Table/JSON/Code aligmment tasks for initialization. Non-chart multimodal tasks need new alignment designs initialize MoE experts. 2) Limited flexibility. Medifying the projector into a multi-expert architecture makes ChartMoB non- plug-and-play like LoRA. We required retrain the router network when new experts are coming.
20

---

Published as a conference paper at ICLR 2025
E CAPABILITY COMPARISONS ON CHART EDITING AND HIGHLIGHTING
ChartMoF,, leveraging carefully designed alignment training, excels in accurately extracting chart values and attributes, facilitating precise image reconstruction, chart element editing, highlighting. this section. we provide a visual comparison with the SOTA MLLMs, Qwen-2.5 (Alibaba, 2024), and GPT-40 (OpenAI, 2024). While all three models perform the required instructions effectively, ChartMoE demonstrates superior accuracy in value and attribute extraction.
Instruction
Table: Metal, Percentage Glod,30
Chart
aevt Freem Coperr
Renett
Redraw the chart with python matplottib, convert it into bur chari, giving the code to reflect the fact that the price of 'Gold' has been reduced to 27% and the ''Silver' has been increused 28%.. Please keep the colors and legend according the input chart.
Platinum,
30
Silver,25
Pouine
Isport entplotiib. pyp pit
ChartMoE
d Derine the Labels and values for the bar chart
Labels I'Stlve'' "cold", 'cosper', "Platinun"
Vetal TrMe Compebtee Goba Naraa
Values 27, 35,
'awkab80"
d Create a tar chart
bars pitebarttabell,, vatues, colar-colors)
title
pit.xlabel! [Motal")
pit-yltbell (Percentage')
pit.tittel Metat Prices Comparison is Global Narket")
pit. Legend sagenddaars, ('Silver', 'Gold", "Copper', "Platinur"
9
Bisplay the chart
Ieport ratp Patploctib. Pyplot pit
Qwen-2.5
ANEAI 'suplee', 'gild", "Copprr' grewn Platiamn's 'gray"
# Miimmn
JeN ea vulues ('Sitver's 28, "Gold: 27, 'Cooper's 35, 'Platinue's : 18
value sunlnew values.values())
percentages (ental: Tvalue / total, total_value value 188). for wetal, value in new wwl.es..
wessi congattan 1844 Sutel
A MMMRM
for ratall percentage percentages, :):
baibarhee,aa,, percentage, colon-cotors
8 12
a6. Laabll Percentape (6)")
titlel 'Wetal Prices Comparison Global Parket!)
12:0 phow!
Inport ratplottib, pyplot a5 pit
GPT-40
setals (GGod'', 'Stlver', 'Copper', 'Platinun!)
PrICeS 127, 28, 35, 18)
colors Iredunsesgreen', 'rediuepurpte"' "gote', 'p
Plat
pit. (tgpsieefipsieeer1.... 613
plt.bbrisetall prices, colarralorss
Labels
Prices Conparison Stotal Norket")
plt.x Metteeleeetala Label] Pca Nercontage!)
pit. Inetats!
0 Show pl&t
showl
Figure 14: Chat demo involves modifying chart types and values. All models successfully convert the chart type, but only ChartMoB handles the legend correctly. No model makes errors in this task due to the simplicity of the values and the presence of data point labels.
21

---

Published as a conference paper at ICLR 2025
Instruction
Table
Chart
Year, GPA 2.3
Redraw the chart with python matplotibb giving the code to highlight the column corresponding to the year in which the student got highest score (painting red), Please keep the sume colors and legend as the input chart.
2017,
3.6
2019,3.8
2020,3.7
3.6
2022,3.8
teport iuspllt Hhbpypllo
ChartMoE
data 13.3, 3,5, 3,6, 3.8, 3.81
yours 1818 2017 20.8 2019 20281, '2023", 128221]
Labels
('Sudden: Average
# plotting the chart
(ee-bar caca, colorwcotors) 1it-ll
Performance
YYar''
pla-ylabel SSSudient Average GPK'1 pln- pir.legendi (abelss)
highest apa pls.berlyeers (tighessiigsstsssttcc indes,
in
index), color='red"
3850
2017
7200
2003
2007
plt.show()
import seport masplottab.yploe as
Qwen-2.5
(138181, 13817" 1288 2019 2008 12823', 128221]
gpas [3.2, 3.4, 3.6, 3.8, 3.3, 4.80
1101er10PP3300 2012
highest_gpe, indec
inden (hightst._gs..yme
AJvorage
e Plotting the chart *1p, pit.sibplots))
35-
se.barlyears [=-1], qpas]:-1]), (olare'blue))
ss.bart Inights _epa_year,, lopes thighes index]]] colorw "red")
pltuylabel (Studint plt. titlel Studen: Perfomance" "1
2.
Adding legend legeeeggdd Average
left!)
* Shou the plct plt.shool)
05
2036
2037
2008
2009
2720
2321
Import matstorTab.pyploe as teport numpy as up
GPT-40
years nowarroy 12016, 12016,
2017, 2018, 2010 2028, 2821, 2022)
g005 10.array][334. 3.5, 3.8, 319, 3.7, 3.8, 4.01]
your
tndex
Cotorst blue for the tars, red for the highest score inx_i else
colors
range years)]]
s Plaz
(nr.figgue((tgsseeeiii 6)1
bars years, gows, colorneetors, Aabel="Student A?]
Hightight
bars Lndclmaxcidddxl..ettt corcococorddd
Labelss
pltuxlabell
Graaee Point GPA')
l "Studen: Performance ")
plu. legund()
plct plt.show[]
200 200 200
200
Figure 15: Chat demo involves modifying chart editing. The bar chart without labeled data points, and all methods provide reasonable and executable drawing code. Qwen-2.5 directly identifies the highest element, while the other methods make it by code. Note that ChartMoE delivers the most accurate values, thanks extensive alignment training and proposed MoE architecture.
22

---

Published as a conference paper at ICLR 2025
Instruction
Table
Chart
Year, GDP Growth
Redraw chart with python matplotli,, giving the code highlight point with lowest
2010,3.5
2012,4.3 4.3
2814,3.8
growth (draw horizontal dotted purallel
2016,2.9 2.9
through the point and legend anchor). Please keep
2028, 2028,3.2 3.2
2020, -4.4
2022,5.2
same colors legend input chart.
watplactib. pyplot
pandes
ChartMoE
Datar years and corresgonding growth rates
2028, 5,213
pdd.ettaFraneeelai
growth rate years
(otipl "Yaars"y del 'Growt Addele"Grodd Rate Parcentage")
Percertage"),
Befine the year and value for the lowest potet twestt_vllur
Add dotted herizontal, line for the Lovest point plt.asal line lowes vallu,, on cnlor=lgrey Linestyl lam ddttte", label="Loons"
plt.tesi (il.teri(looss Lppins__ppiiit lowest Loa 0.5, "Lowest' calor='gray')
Show Legend and display the chart plt. Legendi? plt.show(1
Import nurpy 25 Tp import kutplotkib. pyplat as pll
Qwen-2.5
years (2.00 2012, 2016, 2010, 2018, 2020, 2022]1
growth rates a me-array([-iis 8.5, 2, 22. K.y, +3.5, 4:511
RURROKERS
ap..rggiiiggneehh rates)
Global GDP Grown
px.plotlyears,
rarker="o', color- bilue" Labele "Groth Bate Percentage" on az.achiine aggovtturatss (nin grovin idc), smiemb.1,
seusmb.d 0, deshese(S,
gray' alalu-b.5, sunder=-1) Wy-lyear growes,
18x1,
Inin wnw.ttrortthhpooooooooo... sodt_ sytext-(-10, -19), textcoondse" offset points', har right vae'botton bottoe
Mbosedict (oassyy round" foe'white") ())
as ar.sse_xlabell "Vars!)
Percentage!)
tel iGlebal CDP Grouth")
ax set, (iaariiaaannnnnaaaaa (years)-1 raxtyearste))
ax setset waar.iiarrennnnn lg wwweetsrttttteeieeoo lel:i, ax(prowahlreteesee...
aw.grádlTruel
-3
Orech
legend ax, Loce Lac- Lover right', chacov-Truee Legend.get frennddee_stteeeee frt_errrel.seee facecooor white") pit. show!!
2800
2006
2008
2000
2002
J Let's use satplatlib to create similar chart, and incorporate the requested changes. import watploctib..yppe as pli
GPT-40
Data global 6DP growth years 12610, 2012, 2014, 2016, 2018, 2820, 2022 a Sumple data sinilar to the graph in the Lagge grouth_rate [5:2, 4.1, 4.3, 3.7, 3:3, 44,3, 3:5]
Conate the plat Attrrtaurerrpssiee. pituplotlyears, growth_rate, label=/Grovth Rate in Percentage", color=! blue', worker='s")
abdal NEW IOOLTh
sin.growih -isinfgrowll, rate)
vin_year years years.growwh, atte.innex(esnngrrwann
horizontsl ptt.aaxiiineyyeei...oo
growth Tabele "lowes")
Syuchaa aa tarerage
color='redt Lisestyle
e Ass chart title and Labels att-tinleliGioon GDP Growth' ' fontstrect4)
pit.ylabel/ Growth
Percentage', fonssize-12)
Add the legend and place it with as anchor plt. legend uppprr right" fentsire=ia)
plat
3000
plt. show!!
Figure 16: Chat demo involves modifying chart editing. The line chart is without labeled data points, and all methods provide reasonable and executable drawing code. The values extracted by all models differ from the ground truth, but both ChartMoE and GPT-40 captured the correct data trends. Additionally, ChartMoE successfully completed editing tasks specified the instructions.
23


Reviews:
[{"review_id": "Z1EtKrX3wK", "reviewer": "Reviewer_jNMd", "summary": "The paper introduces ChartMoE, a novel approach that leverages a Mixture of Expert (MoE) architecture to improve automatic chart understanding with Multimodal Large Language Models (MLLMs). ChartMoE addresses the limitations of existing MLLMs by using specialized linear connectors for diverse expert initialization, coupled with a unique dataset, ChartMoE-Align, containing nearly one million chart-related quadruples. This setup significantly enhances data interpretation from charts, pushing the accuracy on the ChartQA benchmark from 80.48% to 84.64%, showcasing a substantial improvement over previous methods.", "questions": "Q1: Modality-Specific Contributions\n\nIt would be beneficial for the paper to elaborate on the unique contributions of different representations (JSON, code, and chart) in the context of chart-related tasks. Understanding how each representation impacts the model's learning and performance could provide insights into optimizing future models for similar tasks.\n\n\nQ2: Necessity and Efficiency of Large-Scale Alignment Dataset\n\nThe heuristic approach to generating a large-scale dataset raises questions about the efficiency and necessity of such a volume of data. Is there potential to achieve similar performance with a smaller, possibly more curated dataset? This exploration could lead to more resource-efficient training processes and better generalization in practical applications.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1. Innovative Methodology: The introduction of the ChartMoE method, which utilizes a Mixture of Experts (MoE) architecture, represents a significant innovation in the field of automatic chart understanding. This approach addresses the modality gap effectively and could potentially set a new direction for future research in multimodal learning systems.\n\nS2. Comprehensive Dataset: The creation of the ChartMoE-Align dataset is commendable. Its large scale and diversity are well-suited for robust pre-training in chart alignment tasks. This dataset not only serves the immediate needs of the study but also provides a valuable resource for the broader research community to explore complex multimodal tasks involving charts and text.\n\nS3. Extensive Experimental Validation: The paper presents extensive experiments that demonstrate the effectiveness of the ChartMoE approach. The thoroughness of these experiments, which include a variety of scenarios and detailed performance metrics, establishes a strong benchmark for future comparative studies.\n\nS4. Clear Writing: The manuscript is exceptionally well-written, providing clear explanations and methodical presentation of the concepts and methodologies involved. This clarity enhances the reader's understanding and appreciation of the work's contributions to the field.", "weaknesses": "W1: Details on Dataset Construction\n\nThe paper lacks critical details on the dataset construction process. Clarifications are needed regarding the criteria used to select and filter charts for inclusion in the dataset. Specifically, the process for generating meta CSV data via Large Language Models (LLM) requires more transparency.\nMore details on which LLMs were used and the code templates for different types of charts are missing. Such information is crucial for reproducibility and for understanding the dataset's applicability to other multimodal tasks.\nThe manuscript should discuss the steps taken to ensure the quality of the data, including any validation mechanisms or controls used during dataset assembly.\n\nW2: Clarification of Experiment Results\n\nThe paper briefly mentions that the proposed method shows weaknesses in some settings compared to baselines, as detailed in Tables 2 and 3. However, these points are not adequately addressed or explained. A more thorough analysis of why ChartMoE underperforms in these instances would be valuable for readers and for future improvements to the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hBXWZkGh4g", "reviewer": "Reviewer_6BC6", "summary": "This work introduces ChartMoE, a multimodal large language model  that enhances automatic chart understanding through a MoE architecture. Unlike traditional linear connectors, ChartMoE uses diverse expert connectors aligned with specific tasks (chart-table, chart-JSON, and chart-code) to bridge the gap between visual encoders and large language models. The paper also presents ChartMoE-Align, a dataset with nearly 1 million chart-table-JSON-code quadruples for training.", "questions": "see above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. ChartMoE’s use of task-specific expert connectors in a Mixture of Experts (MoE) framework provides a  solution to multimodal chart understanding.\n\n2. ChartMoE-Align, a large-scale dataset with varied chart alignments (table, JSON, code).\n\n3. The three-stage training paradigm increass its accuracy in extracting and interpreting numerical data.", "weaknesses": "1. The use of MOE in MLLMs is not particularly novel, as several prior works have already explored MoE structures to enhance model performance. From an innovation standpoint, this reliance on MoE does not introduce a distinctly new approach and could be considered a weakness in terms of contribution.\n\n2. The multi-expert structure, along with the diverse alignment tasks, adds significant complexity to ChartMoE’s architecture. Also, the training data, being mostly synthetic, might limit the model’s ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89EUlR3um3", "reviewer": "Reviewer_WXLS", "summary": "The paper introduces ChartMoE, a multi-task aligned and instruction-tuned MLLM designed for complex chart understanding and reasoning. The key contribution is the replacement of the traditional linear connector with the Mixture of Expert (MoE) architecture, which improves chart understanding by bridging the modality gap. Additionally, a new dataset called ChartMoE-Align is introduced, containing nearly 1 million chart-table-JSON-code quadruples for alignment training. The proposed three-stage training paradigm and high-quality knowledge learning approach result in significantly improved performance compared to the previous state-of-the-art on various benchmarks.", "questions": "1. In line 369, the error analysis suggests that many errors stem from the limitations of the evaluation metric, namely string matching. Are other comparison models also limited by this?\n2. Can you provide more detailed explanations of the data construction process, such as how code templates for different types of charts were obtained? Were they manually constructed?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper incorporates a Mixture of Expert architecture to bridge the gap between charts and language models and offers a valuable insight for the expert initialization manner. The creation of the ChartMoE-Align dataset with nearly 1 million chart-table-JSON-code quadruplets is a significant contribution to the field, allowing for detailed and meticulous chart alignment pre-training.\n\nThe paper is clear and well written in general, is well motivated, and comes with an extensive and comprehensive ablation study.", "weaknesses": "The paper focuses on conducting experiments on a single Vision Encoder and Large Language Model, which limits the generalizability of the proposed method. It would be beneficial to test the effectiveness of ChartMoE on a diverse set of MLLMs to ensure its applicability across different models and scenarios.\n\nThe paper does not thoroughly discuss potential limitations or challenges that may arise when implementing ChartMoE in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXI7VJ9ewx", "reviewer": "Reviewer_Ka6r", "summary": "This paper proposed ChartMoE, which employs the Mixture of Expert (MoE) architecture to replace the traditional linear projector to bridge the modality gap.", "questions": "Have you ever tried training on your alignment data with random initialization of expert parameters and balanced loss from scratch? It will better prove the significance of this work.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First introduce MoE MLLM for chart task, the model architechture design which using MoE in connector is novel.\n2. Detailed experiments and analysis, extensivequantitativeandqualitativestudies.\n3. Introduced a large dataset for chart pre-train data.", "weaknesses": "1. limmited innovation: MoE on multimodal language models has been explored in other domains. There's a trade-off between the performance gain and the increase of model parameters and inference time. \n2. The contributions of MoE module is unclear, more theoretical analysis is needed such as the knowledge in the routing to differnernt experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)