ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation
Authors: Cheng Yang
Chufan Shi
Yaxin Liu
Bo Shui
Junjie Wang
Mohan Jing
Linran XU
Xinyu Zhu
Siheng Li
Yuxiang Zhang
Gongye Liu
Xiaomei Nie
Deng Cai
Yujiu Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset and Benchmark
Code generation
Chart Understand and Reasoning
Primary area: datasets and benchmarks
TL;DR: A benchmark for evaluating LMM’s cross-modal reasoning capability via chart-to-code generation

Abstract: We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering.
ChartMimic includes $4,800$ human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span $18$ regular types and $4$ advanced types, diversifying into $201$ subcategories.
Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts.
Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of $3$ proprietary models and $14$ open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of $82.2$ and $61.6$, respectively, indicating significant room for improvement. 
We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.

Published as a conference paper at ICLR 2025
CHARTMIMIC: EVALUATING LMM'S CROSS-MODAL REASONING CAPABILITY VIA CHART-TO-CODE GEN-
ERATION
Cheng Yang** Chufan Shi
Yaxin Liu
Bo Shui
Junjie Wang
Mohan Jing! Linran Xu
Xinyu Zhu
Siheng Li
Yuxiang Zhang' Gongye Liul
Xiaomei Nie!
Deng Cai
Yujiu Yang
TTsinghua University Tencent Al Lab
chartmimi wtw.:a1..mmmmmmmmmmmmmmmmm.....m
ABSTRACT
We introduce a new benchmark, ChartMimic, aimed at assessing the visually grounded code generation capabilities of large multimodal models (LMMs). Chart- Mimic utilizes information-intensiee visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. Chart- Mimic includes 4, 800 buman-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found scientific papers across various domains (e.g.. Physics, Computer Science, Economics, etc). These charts span 18 regular types and advanced types, diversifying into 201 subcategories. Fur- thermore, propose multi-level evaluation metrics to provide an automatic and thorough assessment the output code and the rendered charts. Unlike existing code generation benchmarks, ChartMimic places emphasis evaluating LMMs' capacity to harmonize blend cognitive capabilities, encompassing visual un- derstanding, code generation, and cross-modal reasoning. The evaluation of 3 proprietary models and 14 open-weight models highlights the substantial chal- lenges posed by ChartMimic. Even the advanced GPT-4o, Intern VL2-LLLma3337 only achieved an average score across Direct Mimic and Customized Mimic tasks of 82.2 and 61.6, respectively, indicating significant room for improvement. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.
INTRODUCTION
Code generation (Sun et al. 2024) a rather demanding task that requires advanced abstract thinking and logical reasoning, reflecting the unique intelligence of human beings. Recently, advances in arti- ficial general intelligence (AGI) have demonstrated the potential of large foundation models (Google)
2023 OpenAT 2024 Anthropic 2024 AT Meta 2024 to solve the tasks that once the exclusive
domain of human abi lities (Achiam et al. 2023 hu et al. 2024). However, existing code generation
benchmarks (Chen et al. [021]
[ Austin et al./ |222 Hendryeks et 2021 Lai al., (023) solely
use text as input, while humans receive information from multiple modalities when coding (Liang
al. 2023
Fan et
2024). Such real-life scenarios have yet to be fully explored.
Taking common scene in Fig. researchers often need to write code for data visualization and may already have preferred chart templates hand. However, they usually lack either the source code or the expertise to reproduce these chart templates. As result, they turn to large multimodal models (LMMss as assistants to aid code generation. In this scenario, coding for scientific charts entails code generation grounded on visual understanding (i.e., chart-to-code generation), which necessitates LMMs to integrate a variety of advanced cognitive capabilities, including visual understanding, code generation, and cross-modal reasoning. Therefore, evaluating the performance of LMMs on this real-world task also enables researchers to pinpoint potential areas for improving models' capabilities.
"Equal Contribution.
Data and code are available at https://githui: com/ Soom/ChartMiiiie
ChartMimic
Corresponding author.

---

Published as a conference paper at ICLR 2025
It's de hand properly when aerting a papar..
remember there wene zome F'gunes read before..
The style of this how get the code drew
Let's LMMss with
With little medification, con get the style ward!
Fhe dealt of
dana
Figure 1:
The real-world example. LMMs assist scientists and rescarchers in understanding, inter- preting and creating charts during the reading and writing of academic papers. These models serve as assistants that enhance the comprehension and presentation of data in scholarly communications.
To this end, we present ChartMimic (Fig. 2 a multimodal code generation benchmark. ChartMimic characterized by its (1) informationnintensiiee visual inpuls, (2) diverse chart types, and (3) multi- level evaluation metrics. Specifically, compared to natural images, scientific charts convey nuanced semantic meanings through intricate visual logic, thereby exhibiting higher information density, Based on this, we define two tasks, Direct Mimic and Customized Mimic (Sec. 2. which utilize charts and textual instructions as inputs. These tasks challenge LMMS to generate the corresponding code for given chart or to incorporate new data specified in the instructions, respectively. Through the collection of academic documents and scientific papers, we identify 22 commonly used chart types and 201 subcategories. Subsequently, we manually annotate a total of 4, 800 (figure, instruction, code) triplets for these types (Sec. 2.2 Furthermore, we establish automatic evaluation metrics from both high-level and low-level perspectives to thoroughly assess the performance of LMMs (Sec. 2.4).
We conduct the examination of 17 LMMs on ChartMimic (Sec. 3.2), including 3 proprictary models and 14 open- apen-weight models across parameter sizes from 2.2B to 76.0B. We observe that while several open- geenhw models can match the performance of proprietary models such as GPT-4o on public leaderboards (OpenCompass) 2023), significant performance gap still persists ChartMimic. Specifically, the best open-weight model, Intern VL2-LLama337668 lags behind GPT-4o, with an average score gap of 20.6 on two tasks, indicating substantial room for improvement in the open- source community. Our analysis of prompting methods (Sec. 4.2) reveals that GPT-4o can improve itself through self-reflection, which is a key manifestation of System 2 reasoning (Sloman 1996] Kumar et al. 2024). This discovery highlights the vital role that the System 2 reasoning plays in LMMs when tackling the complex challenges presented by ChartMimic. Meanwhile, Correlation analysis (Sec. 4.3) demonstrates a high correlation between our automatic metrics and human evaluation, validating the effectiveness of these metrics. Further error analysis (Sec. [.4) reveals that hallucinations notably hinder the performance of LLMs on ChartMimic, as they lead to the insertion of non-existent text into ground-trutt figures and confusion between similar types of charts.
We envision ChartMimic as a comprehensive suite of benchmarks designed to guide researchers in understanding their LMMs' capabilities. By providing a comprehensive evaluation framework. ChartMimic aims to facilitate the growth of foundation models for the community, offering insights into various aspects such as visual understanding, code generation, and cross-modal reasoning.
2 THE CHARTMIMIC BENCHMARK
In this section, we first introduce the definition of two tasks involved in ChartMimic (Sec. 2. and then delineate the data curation process (Sec. 2.2) Subsequently, conduct quantitative analysis to assess the quality and diversity of ChartMimic (Sec. 2.3). establish evaluation metrics (Sec. 2.4).
and compare it with existing related benchmarks (Sec. 2.5
2.1 TASK DEFINITION
LMMs' ability to generate chart-rendering code demonstrates their visual understanding, coding, and cross-modal reasoning skills. Specifically, given the chart X and the instruction I, the LMM f is expected to output the code C that satisfies the requirements outlined in the instruction:
C = f(,
(I)
As shown in Fig. 2 based on the information provided in the instructions, we propose two tasks:
Direct Mimic. The LMMs are tasked to directly generate code that can reproduce the provided chart, thereby assessing their visual comprehension and reasoning capabilities.
2

---

Published as a conference paper at ICLR 2025
DATASET
TASKS
RESULTS EVALUATION
ChartMimic
Trisk I: Direct Himic
Generated Code
Rendered Chart
Types
malzlotih repodlaze reforsss
Masse image
LLeth Degaals HL 4.5.
Flgur
NSeans Eegiadt + (1-ds Ra 3.4]
www.ceeCCCCnccnaaennenoo sccdall
Chart Types
18 Regular
|
with har aid Tabel Secaullry
Emroppoer
f
Taask 2:
Custornized Mimic
Toxt
0.78
Layout
1.00
Hiss
Vislin
Hill
Hoautty
Plase cranc clast
Type
0.60
Color
0.84
accident cliforeet aress. Cata is Sillows
Bhe impust spool us
00
112
level Scores
G2 High Sooriss
Trecer
CITT-li Sconer
Costemnnn Bate
Twxt
0.82
Layoud
Taill
iied Untta 100 10, 3t, 40,
Type
1.00
Color
0.82
GPT-4e 0.50
7D
Pre
Radar
Treerap
Danndrred Chari
lecn to [ee, at, 1o, on ne]
For Auralg Dr dy bal, Tal
Hearnap
Casaoaa
betukters 8
Chart Types 4 Advanced
stoee) "Speed Lteie OWP?"
clanel Meccdent Fate Change (N)
et ar.hel ber.addry L.T, Utben, aid
PIP
HB
Al sec. s1a002
Figure 2:
The pipeline of ChartMimic. We provide 4, 800 human-curater (figure, instruction, code) triplets. We use ChartMimic to evaluate LMMs' proficiency in the multimodal chart-to-code generation, resulting in both high-level and low-level evaluation results.
Customized Mimic. The LMMs are requested to generate code for a new chart that incorporates customized data provided in the instruction while preserving the original chart's aesthetic and design, assessing their ability to integrate visual and textual information.
After obtaining the code generated by LMMs, we execute it to render the corresponding chart and subsequently compare its similarity with the ground-truth chart. Illustrative examples of the two tasks are shown in Fig. 2 To accommodate the defined tasks above, we propose ChartMimic, a benchmark designed to evaluate the comprehension of charts and their conversion into executable code.
2.2 DATA CURATION PROCESS
ChartMimic distinguishes itself through 4 fundamental considerations: (1) diversity of chart rypes, (2) balance of chart complexity, (3) reduction of data leakage (4) integration of authentic user requirements. We keep on these four principles to complete the data curation for ChartMimic through a five-step pipeline. Here, we provide an overview here and more details in Appendix B
General Filtering. We scrape figures from source files of publications on arXiv that hold BC 4.0 license with a publication date after February 2024 with PDF format. This yields approximately 174, 100 figures across various domains (e.g., Physics, Computer Science, Economies, etc). We then filter the figures based on the criteria of how designers select inspiring visualization examples, including clarity and visual appeal, color schemes (Bako et al.) 2022 Quispel et al. 2018). and uniqueness of the chart within its category, resulting in refined collection of about 15, 800 figures.
Diversity and Information Density Filtering. The filtering process involves two stages. In the first stage, we establish a data pool and categorize chart types. Charts with significant differences in complexity or information density are added to ensure diversity and effective communication (Bako) ct al. 2022). In the second stage, five experts from various fields independently evaluate the data, creating separate selection pools. We preserve the intersection of their selections and finalize the set through a voting process. This meticulous approach refines collection to 279 charts.
Manual Selection and Filling. In addition to sourcing from arXiv, we also collect charts from various platforms. including the Matplotlib gallery, Stack Overflow, and plotting-related forums on Twitter and Reddit. These charts are selected for their unique styles not represented in our arXiv curated 279 charts. To mitigate the risk of data leakage, we rigorously process the data and color styling of these charts, replacing existing color schemes with those unseen in our data pool while maintaining their aesthetic appeal. Consequently, we obtain 600 prototype charts for ChartMimic.
https://arxiv..rgg)
3

---

Published as a conference paper at ICLR 2025
Table I: Statistics of ChartMimic. We measure code length in terms of tokens, utilizing the Llama3
tokenizer. In the level count, "A/B/C" denotes the number of "easy/medium/hard level, respectively.
Type
Line
Heammpp
Hiis
Rodar
EnurFuint
Violin
Coust
280
120
801
Subcategoriie
12
6
Lnggh (AVG))
689.6
794.0
681.2
6958
6890
655.0
5296
779.
418.4
634.3
975.6
Code Lengto (STD)
237,8
244,4
146.7
258.7
228.2
253.0
147.1
146.3
241.4
99.5
197.7
252.3
Level Count
1767120/2 120/24
/6//6//0
68 65/52/0
0/76/44
60 40/0
80/20/0
52/ 25/0
52 28/0
87 8/48/24
52/28/0
/O
28/8
32/ 44/4
Type
Area
Contour
Density
Grngh
Quiver
Treenap
Combisation
HR
Muitiliff
PIP
Total
Count
80
30
80
80
80
80
120
100
100
80
2400
Subeatzgoiics
3
4
4
50
25
25
20
201(101 4 100)
Clod Lesgih (AVE.)
774.4
489.4
$40.0
564.5
895.4
342.2
697.4
718.9
798.2
1053,9
696,0
Code Length (STD.)
161
67.8
104.7
117.5
6310
36.3
163.6
265.5
271.2
290.1
278.4
Lovel Count
52/28/0
0/23/52
44/32/4
567
26/2670
0/52/26
52/28/0 /O 12/76/32
4/16/80
07 07/8/52
0/0/80
1100 808
Code and Instruction Writing. We propose to manually write codes and instructions for ChartMimic based on the collected 600 prototype charts. Initially, annotators are tasked with meticulously reproducing the 600 prototype charts using Python code, resulting in set of 600 (figure, code, instruction) triplets for the Direct Mimic task. Although other coding languages such as JavaScript and R can be used to create charts, current LMMs perform poorly when doing chart-to-code using these other languages (Sun et al. 2024). Therefore, we focus on Python code generation in the current version of ChartMimic. Subsequently, to simulate the scenario of Customized Mimic, annotators are instructed to modify the chart data in the Direct Mimic task by integrating new data from various domains. They are then required to modify the corresponding code and instructions, leading to the 600 (figure, code. instruction) triplets for the Customized Mimic task. Consequently, we establish the ChartMimic benchmark, comprising 1, 200 high-quality seed data.
Data Augmentation. Following the development of seed triplets, we initiate a process of manual data augmentation. Annotators are tasked with altering various elements of each seed triplet, including data, color schemes, mark styles, etc., produce augmented triplets. For each seed triplet, we create 3 additional augmented triplets. This process enhances our dataset, yielding a total of 4, 800 triplets that reflect wide range of realistic and practical chart use cases.
2.3 DATA STATISTICS AND DIVISION
The (figure, code, instruction) triplets for both Direct and Customized Mimic tasks share the same figure. Therefore, we detail the data statistics for the 2, 400 triplets in the Direct Mimic task. As depicted on the left side of Fig. 2 ChartMimic encompasses total of 22 categories, with 18 types of regular charts and 4 types of advanced charts. For the 18 regular chart types, we identify 101 subcategories, with the definitions and examples of each subcategory provided in Appendix D The advanced chart types such as Plot-in-Ploy (PIP), Multidiff, and Combination are distinct forms of amalgamating multiple chart sets into a singular chart. Given the variety of their internal combination elements, each of these can be treated as unique subcategory. Meanwhile, the Hard-to- Hard-to-Recognizz (HR) category encapsulates unclassifiable charts, with each chart in the seed data being considered category unto itself. When we factor in the additional 100 subcategories represented by these advanced chart types, ChartMimic encompasses a total of 201 subcategories. This extensive diversity underscores the comprehensive nature of our benchmark. We employ Llama3 (AI@ Meta 2024) tokenizer to measure the code length. As shown in Tab. ChartMimic has an average code token length of 696.0 with a standard deviation of 278.4. Additionally, we manually categorize charts into three complexity levels: easy (1,100), medium (868), and hard (432). The detailed categorization criteria and assessment methodology are thoroughly documented in Appendix B.
We further divide the 4, 800 examples of ChartMimic into two subsets: test and testmini set. The test set comprises 3, 600 examples, while the testmini set is composed of 200 examples. The testmini set is designed for rapid model development validation. Our partitioning strategy ensures each chart type is proportionally represented, preserving distribution in the testmini set that closely aligns with the test set. Detailed comparative experimental results, discussed in Appendix C demonstrate the consistency across two subsets. Unless otherwise stated, we report results on the testmini set.
2.4 EVALUATION METRICS
For tasks within ChartMimic, an appropriate evaluation necessitates comparing the visual similarity between the generated and ground-truth figures. To achieve this, we propose multi-level metrics (i.e., high-level and low-level) to assess the similarity at different granularities. Specifically, the high-level metric encompasses the GPT-4o Score, and the low-level metric encompasses Text, Layout, Type and
4

---

Published as a conference paper at ICLR 2025
Table 2: A comparison of our proposed ChartMimic to other benchmarks. "I" and "NL" indicate
"Image" and "Natural Language" respectively.
Benchmarks
Source
Types
Test Instances
Input Format
Ouiput Format
Evaluation Metric
Chort Understanding Benchmarrss
ChanQA (Maery et al. 2022
Human Carated
10K
IeNL
NI.
Accumey
Chart-to Text Kamihara
2022
Crawl
44K
INLL
NL
Match based
ChartSumm Rahaan el al 2023
Human Curated
84K
I+NL
NL
Match-based
CharArXiv Wang et 20246
Human Curated
18
93K
I+NL
NL
GPT-4 Score
Code Generation Benclmarks
HumanEEval
2
Human Curated
164
Code
Code
Rate
MBPP (Austin ot cUe X023
Human Carated
500
NL+Code
Code
Pass Rate
MMCoie (L) 2024b
Crawl
263
I+NL
Code
Pass Rate
MalPlotBennch Yang cl 2024
Human Curated
13
100
NL
Code
GPT-4 Score
Plot2Code (Wrrd ai 20/-0
Crawd
15
132
I+NL
Code
Multi-Level
Design2CodE (St
Crawd
HTMI.
Code
Multi-L
ChartMimic (Our)
Human Curated
22
4,800
I+NL
Code
Multi-Level
Color Score. We compute the average score between the high-level and low-level scores, ranging from 0 to 100, as the overall score. The illustration of the multi-level metrics is depicted in Fig.
GPT-40 Score. Following the successful use of large foundation models for evaluation in both natural language processing (Zheng et al. 2024 [i et al. [2033] Dubois et al., 2024) and computer vision (Zhang et al. [023 Yang et al. [022 Wu et al. (024b), we adopt GPT-40 score as our high- level metric. Specifically, we input both the ground-truth figure and the generated figure into GPT-4o, and instruct it to output a high-level similarity score ranging to 100. Although CLIP Score (Radford et al. (222)) is widely used for assessing image similarity, in our preliminary experiments. it has struggled to distinguish variations in types and other critical elements in charts, resulting in a low correlation with human evaluation results. Therefore, we use only GPT-4o Score as our high-level evaluation metric. The detailed description of GPT-4o Score can be found in Appendix
In addition high-level similarity, evaluating the similarity among low-level elements between generated and ground-truth figures can provide a more fine-grained analysis of LMMs. Therefore, we propose to evaluate four key low-level elements in charts (Savva al. 2011 Poco & Heer 2017):
text, layout, type and color. Extracting them from figures is challenging task, as existing extraction models often fall short in terms of accuracy (Meng et al.) 2024). Considering that figures are rendered based on the code, we design code tracer to monitor the execution processes of the ground-trum code and generated code. The code tracer records the text, layout, type and color information. We calculate the FI score of these elements as their corresponding score. And we average text, layout, type and color scores to get the low-level score. The methodology for obtaining these elements is briefly introduced as follows, with detailed descriptions provided in Appendix E
Text Score. During the code execution process, for the function responsible for adding text elements to the rendered figures, the code tracer monitors it and records each text element parameter.
Layout Score. Layout refers to the arrangement of subplots in the figure. At the end of the code execution, the code tracer traverses all subplots in the figure and obtains their layout information.
Type Score. For each plot function, which serves the purpose of adding a specific chart type instance to the figure, the code tracer monitors its invocation status and records every invoked plot function.
Color Score. For each plot function, it will return the chart type instance at the end of the function invocation. The code tracer accesses the color attributes of these chart type instances.
is important to note that code execution success rate is a standard metric for code generation tasks (Sun al., 2024). We have implicitly incorporated this aspect into our high-level and low-level scores. Specifically, ly, if the code fails to execute successfully, both the low-level and high-level scores are assigned a value of O. Therefore, we do not separately weight it into the overall score.
2.5 COMPARISONS WITH EXISTING BENCHMARKS
To further distinguish the difference between ChartMimic and other existing ones, we elaborate the benchmark details in Tab. 2 From the chart understanding perspective, the prior benchmarks (Masry
et al. 2022 Kantharaj et al. 2022 Rahman et al.) 2023: Wang et al.) 2024b) are only focused on questions about the data in the charts without necessitating the advanced cognitive capabilities of LMMs, which include visual understanding, code generation, and cross- moos- reasoning. In the code generation aspect, HumanEval (Chen et al., 2021), MBPP (Austin et al.) 202T) and MatPlot- Bench (Yang et al. 2024) only consider tasks with text inputs, which may not meet requirements
5

---

Published as a conference paper at ICLR 2025
Table 3: The ChartMimic leaderboard with Direct Mimic task. The best scores are in bold. We also include the code execution success rate (Exec. Rate) and model size (Params).
Exec. Rate
Low-Leve
High-Leve
Model
Params
Overall
Text
Layout
Type
Color
Avg.
GPT-4o
Proprietary
GeminiPro Vision
68.2
52.6
64.2
51.3
47.1
53.8
53.3
53.6
Claude-3-opus
83.3
66.8
83.1
49.9
42.1
60.5
60.1
60.3
GPT-4o
93.2
81.5
89.8
77.3
67.2
79.0
83.5
81.2
Open- Weight
IDEFICS2-8B
7.6B
49.0
6.2
33.1
9.2
9.0
14.4
17.6
16.0
DeepSeck- VL-7B
7.3B
41.3
15.3
26.6
19.7
14.5
19.0
20.4
19.7
-laVA-Next-Yi-34f)
34.8B
50.2
15.9
29.6
17.6
15.2
19.6
20.6
20.1
LLaVA- WWWV-Next-Mistral--
7.6B
59.7
14.0
31.1
19.8
17.8
20.7
21.3
21.0
Qwen2-VL-2B
2.6B
47.0
20.1
29.5
21.3
17.9
23.4
Cogvlm2-Haama--hhtt
19.2B
50.5
21.3
31.8
18.4
17.0
22.1
24.5
VL2-2B
2.2B
52.5
23.6
35.8
16.0
15.4
22.7
24.2
Qwen2-VL- VL-7B
8.2B
26.4
51.0
31.0
23.3
32.9
35.0
ILtem4
4.2B
66.2
34.7
51.7
25.2
23.6
33.8
38.4
InternVL2-8B
8.1B
61.8
31.5
51.1
28.6
26.2
38.9
MinICPM-Llama3---25
8.4B
80.3
30.7
49.6
38.6
27.6
36.6
42.1
39,4
Phi-3-Vision-128k 128K
4.2B
37.5
49.6
37.4
29.8
41.0
39.8
InternVL2-26B
26.0B
58.7
35.9
76.0B
74.5
49.2
62.2
58.5
attempted to create a benchmark for multimodal code generation, but the vision inputs for their task still overly simple and only have single pass rate evaluation metric, Design2Code (Si et al. 2024) and Plot2Code (Wu et al. (022a) the most similar ones ours. Although they lelei- evaluation metrics like ours, their test is crawled directly from the internet or existing datasets, which may pose a risk of data leakage. Our ChartMimic benchmark gives complex scientific charts inputs, demanding capabilities grounding visual understanding into code generation, and provides multi-level evaluation metrics.
the era LMMs. Recently, MMCode (Li et al., 2024b)
EXPERIMENT
3.1 BASELINE SETUP
benchmark 17 widely utilized proprietary and open-source models currently available in the field. For proprietary models, consider representative models: GPT-4o (OpenAI) 2024), Claude- 3-opus (Anthropic) 2024) and GeminiPro Vision (Google 2023). For the open-weight models, we choose 14 competitive models with total parameter size from 2.2B 76.0B: Intern 4B, 8B, 26B, 76B) (Chen et al.] (203)), Qwen2-VIL(BB, 7B) (Wang et al. 2024a), Phi-3-Vision (phi) 2024a). IDEFICS2- (Xu et al. 2024) and Seevim2-llama--ee 19B (Wang et al. 2023a). We start with direct prompting, which provides the reference chart with direct instructions. The specific instructions and model configurations can be found in Appendix F
2024). DeepSeck- PoopSeck-VL-BBB (Lu al. 2024), LLaVA-Next(7B. 34B) (Li
8B (Laurençon et al. 2024), MiniCPM-Llama3-V22
3.2 MAIN RESULTS
We present the main results of 17 LMMS on ChartMimic. Tab. Band Tab, 4 show the results on the Direct Mimic and Customized Mimic task, respectively. The key findings are as follows:
GPT--0 performs best among proprietary models, while open-weight models. In proprietary models, GPT-4o achieves an overall score of 81.2 in Direct Mimic and 83.2 in Customized Mimic. Within open-weight models, Intern Vn.ernVL.2-L... L.2.--Lmmmm reaches the overall score of 58.5 in Direct Mimic and 64.7 Customized Mimic, which have compareble per- fromance with proprietary model like Claude-3-opus (60.3 in Direct Mimic and 65.4 in Customized Mimic). Meanwhile, Phi-3-Visiie Vision- 128K despite its 4.2B parameters, reaches the overall score of 39.8 in Direct Mimic and 42.1 in Customized Mimic, which outperforms LLaVA-Next S-A-Neet-Yi---A Yi-34B (20.1 1 in Direct Mimic and 35.3 in Customized Mimic). This indicates that even models with fewer parameters achieve decent performance through a refined training process.
excels among

---

Published as a conference paper at ICLR 2025
Table 4: The ChartMimic leaderboard with Customized Mimic task. The best scores are in bold. We also include the code execution success rate (Exec. Rate) and model size (Params).
Exec. Rate
Low-Level
High-Level
Model
Params
Overall
Text
Layout
Type
Color
Avg.
GPT-4o
Proprietary
GeminiPra Vision
76.2
52.2
70.9
56.0
49.4
57.1
59.6
58.4
Claude-3-opus GPT-4o
88.2
75.2
86.8
54.1
44.3
65.1
65.7
65.4
96.5
88.5
79.2
67.6
82.1
84.3
83.2
Open- Weight
Qwen2-VI.-28
2.6B
35.8
17.4
23.9
19.7
16.5
19.4
21.4
20.4
Cogvim2-llama--chtt-
19.2B
38.7
19.0
27.9
15.7
19.8
21.6
20.7
LLaVA-Nex--Mittal-77
7.6B
49.0
20.0
32.0
19.9
23.6
24.7
24.2
IDEFICS2-8B
7.6B
49.2
21.6
32.2
18.1
12.2
21.0
27.3
24.2
1nternV1.2-2B
2.2B
49.3
22.2
35.4
20.0
18.1
23.9
27.8
25.9
I.L.VANeet-VV-34FP
34.8B
64.2
28.7
44.8
32,9
27.7
33.5
37.1
35.3
DeepSeck-VL-7B
7.3B
59.3
27.5
47.5
36.8
31.5
35.8
39.3
37.6
Phi-3-Vision-128K
4.2B
67.8
29.7
52.5
42.3
36.5
40.3
44.0
42.1
IntemVVL2-4B
4.2B
74.0
41.3
55.6
39.6
33.1
42.4
47.8
45,1
Qwen2-VL-7B
8.2B
73.3
41.0
56.3
43.5
34.2
43.8
47.8
45.8
InternVL.2-8B
8.1B
73.0
43.1
54.4
39.9
35.4
43.2
48.9
46.1
MiniCPM--lamu3---2.55
8.4B
78.7
40.8
58.0
44.8
33.2
44.2
51.5
47,9
InternVL.2-2BB
26.0B
73.7
43.9
62.3
43.5
34.3
46.0
51.1
48.6
InternVL2-Llama3-77
76.0B
89.8
57.8
79.0
63,5
50,5
62.7
66.7
64.7
A performance gap persists between open-weight LMMs and proprietary ones. Although several open-weight LMMs can exhibit performance comparable GPT-4o across various bench- marks (OpenCoomass) 2023), even the best-performing Intern V1.22 falls short of achiev- ing the performance GPT-4o both Direct Mimic and Customized Mimic. This apparent performance disparity demonstrates the challenging nature of our ChartMimic benchmark for current open- weight LMMs. Moreover, open-weight LMMs even exhibit notable deficiencies in generating executable code, with majority of them showing execution rate below 75%. These findings highlight that there is still considerable scope for the open-source community to enhance LMMs capabilities regarding complex visual understanding, code generation and cross-modal reasoning.
When given additional user-customized data, LMMs exhibit performance improvements. the Customized Mimic task, most LMMs get performance improvement compared to the Direct Mimic task, particularly in terms of Text Score. This enhancement can be attributed to the provision of user-customized data, which alleviates the burden on LMMs to recognize textual and data information within the charts. Notably, Cogvim2-llamas -aat22llaa3aach-a---- 19B and Qwen2- VL-2B experienced decrease in their overall score and execution rate. This decline may arise from the reason that eser-customized data increases the burden on the models to process different modalities of information. Overall, except for GPT--0,, the performance
the remaining LMMS still below 65.0 the Customized Mimic task. This indicates that. in addition to the capability to recognize data in the chart, understanding the layout the chart and code generation are also key factors for LMMs to better accomplish the task.
4 DISCUSSION
4.1
DIFFERENT COMPLEXITY LEVELS
GPT4o
We report the performance of the top-6 models across different complexity levels for ChartMimic in Fig. 3 There is a consistent decline in performance across all tasks as the difficulty increases. For example, in Di- rect Mimic task, the performance of GPT-4o at easy, medium, and hard levels are 86.5, 77.7, and 74.8, re- spectively. These results demonstrate that ChartMimic are inherently challenging and confirm the efficacy of the established difficulty levels. Additionally, for weak LMMs such as MiniCPM-Llama's V-2.5, providing cus- tomized data at the Easy Level help it perform tasks better (from 46.8 to 59.7). However, at the Hard Level, when given customized data, their performance
J-pps
80
Overall Score
60
20
Easy (a) Direct Mimic (b) Customized Minic
Medium Hard Easy Medium Hard
Figure Overall scores of models at differ- ent complexity levels.

---

Published as a conference paper at ICLR 2025
Table 5: Results for different prompting methods for GPT-44o and different size of Intern VL2 LMMs on Direct Mimic task. The best scores for each model are in bold.
Model
Params
Method
Exec. Rate
Low-l LeveL
Hege
Overall
Text
Layout
Type
Color
Avg.
GPT-4o
Direct
93.2
81.5
89.8
77.3
67.2
79.0
83.5
81.2
GPT-4o
HimtEahanced
92.0
80.9
67.9
79.2
83.0
81.1
SeliReflection
95.2
86.6
93.7
81.1
68.2
82.4
87.1
84.8
Scaffold
91.0
81.3
90.6
75.1
63.9
77.7
80.3
79.0
Direct
83.2
54.1
74,5
49.2
41.5
54.8
62.2
58.5
76.0B
HimtEnhanced
78.0
54.3
71.7
41.6
54.5
60.3
57.4
SeliRefleztion
82.5
57.0
76.2
56.4
45.0
58.7
63.6
61.1
Scaffold
76.7
52.1
71.4
49.1
40.5
53.3
60.1
56.7
Direct
69.3
39.2
58.7
35.9
31.8
41.4
47.4
44.4
InternVL-2220 L22-26B
26.0B
HimtEnhanceed
66.8
40.4
33.0
41.2
47.0
44.1
Self
69.0
39.5
57.5
35.6
30.5
40.8
47.1
43.9
Scaffold
64.3
36.0
53.5
35.4
30.3
38.8
44.0
41.4
Direct
61.8
31.5
51.1
28.6
26.2
34.4
38.9
36.6
InternVL2-44 VL2-8B
8.
HintEnhanced
58.7
30.1
38.3
27.5
31.5
35.9
33.7
SelfReflection
56.8
27.4
39.8
25,1
22.9
28.8
33.4
31.1
Scaffold
61.8
22.8
43.5
24.8
20.2
27.8
32.0
29.9
Direct
52,5
23.6
35.8
16.0
15.4
22.7
24.2
23,5
Intern VL.22BB
2.1B
HintFnhanced
12.9
SelfReflection
35.5
16.3
22.2
12.8
14.8
16.5
17.3
16.9
Scaffold
38.0
6.5
17.8
3.2
2.2
7.4
15.6
11.5
do show improvement (from 31.9 to 31.5). This indicates that for Easy Level, complex reasoning abilities are not required, and providing data is sufficient to complete the task well. However, for Hard Level charts, LMMs need complex code generation capabilities grounded on visual understanding. In this case, even when given data, if the LMMs cannot perform the corresponding code reasoning, they still cannot complete the task. Moreover, when given addtional data, if the data too long and complex, the LLM may still become confused, leading to a certain decrease in performance.
4.2 DIFFERENT PROMPTING METHODS
We further examine the impact of different prompting methods the performance of ChartMimic benchmark. Specifically, we choose the GPT-4o and Intern series LMMs (2B, 8B, 26B, 76B) to study their performance on the Direct Mimic task. We select three representative prompting methods: HintEnhanced. SelfReflection, and Scaffold Prompting. HintEnhanced uses prompt with (Wei et al. 2022), explicitly prompting the LMMs to pay attention important details (e.g., layout, type, text, etc). SelfReflection (Shinn et al., 2024) involves inputting the LMMs' own output and the corresponding rendered chart as additional information, instructing the LMM' to self-reflect their output. Scaffold Prompting (Lei et al. (024) overlays dot matrix within the figure as visual information anchors and leverages mulli-dimmensioal coordinates as textual positional references. We detail the experimental setups in Appendix
As shown in Tab. 5 SelfReflection enables GPT-4o and InternVL2--Lann correct their outputs, demonstrating notable improvements over Direct Prompting. This underscore self-reflection as a key manifestation of System reasoning (Sloman 1996 Kumar et al. 2024). For GPT-4o, SelfReflection enhances Text, Layout, and Type Scores, though Color Score remains unchanged due to persistent challenges in fine color discrimination. However, LMMs with less developed reasoning capabilities (Intern VL2-26B, 8B, and 2B) show improvement even decline with self-reflection. This indicates that only LMMs with substantial reasoning capabilities can effectively engage in self-reflection and result optimization, underscoring the critical role of System 2 reasoning in addressing ChartMimic's 's challenges. Regarding HintEnhanced and Scaffold methods, they do not enhance the performance of models and reduce performance varying degrees. Upon examining the cases, we find that the HintEnhanced method, which involves generating captions first, can introduce hallucinations that lead to errors in the subsequently generated code. As for the Scaffold method, introduction of an additional dot matrix can interfere with existing coordinate axis information in charts with high information density, thereby negatively impacting performance. These negative effects intensify LMMs' reasoning capabilities decrease, further highlighting the importance of advanced reasoning capabilities in handling complex prompting strategies. We provide case studies of three prompting methods in Appendix
to reflect on and
8

---

Published as a conference paper at ICLR 2025
Dimersion
Parameter
Extransous
Missing
Extranious
Missing
Simiar
Diforentt
12.2%
19.0% (89)
13.5% (30)
33.1%
42.1%
41.3%
(1322)
(131)
38 9% (121)
45.3% (101)
66. .%
(261)
Code
(b) Text
(c) Type
(d) Color
Figure 4: Error analysis of GPT-4o across four eтrоr types on the Direct Mimic task. The number in
brackets indicates the count of error cases. Error examples can be found in Appendix Appendix [
4.3 CORRELATION WITH HUMAN EVALUATION
To evaluate the reliability of the proposed multi-level met- rics, we calculate their correlation with human evaluations. Specifically, we collect 1, 200 charts collected from GPT- 4o using four different prompting methods (Sec. (42) on Direct Mimic. Each charts is evaluated by three individ- ual evaluators, who assign scores ranging from 0 to 100 based on the similarity with the ground-truth chart. Details about the human evaluation process are provided in Ap-
Table 6: Pearson correlation coefficient between multi-level evaluation metric and human evaluation.
Metric
Coefficient
p-value
High-Level 0.7041
0.0001
Low-Level
0.7681
0.0001
pendix G As shown in Tab. 6 the Pearson corrllation coefficient (r) indicates that both the high-level (r = 0.7041) and lowelevel metric (r = 0.7681) 1) have a high correlation with human judgment, demonstrating the reliability of our multi-level metrics. Moreover, the low-level metric demonstrates a higher correlation with human judgments compared to the high-level metric. This attributed to its dependence on code execution logic, enabling capture detailed elements.
4.4 ERROR ANALYSIS
We conduct detailed error analysis of the state-of-the-an GPT-4o model to elucidate the current limitations of I.MMs on ChartMimic, thereby identifying potential areas for further improvement. Our analysis identifies and category four main types of errors, detailed below.
Code-related Errors. (1) Dimension: Errors associated with data dimension, e.g., data dimension does not satisfy required conditions for operations: (2) Access: Errors occurring when accessing iterable elements out of bounds or accessing undeclared variables; (3) Parameter: Errors related to passing incorrect parameters when invoking functions. As observed in Fig, (a), the majority of errors stem from Dimension, suggesting analysis and operation on data poses a challenge for GPT-4o.
Text-related Errors. (1) Detail: The text in the generated chart is largely consistent with the ground- truth chart text but contains minor discrepancies, such as mixing up "-" and "_"; (2) Missing: The generated chart omits text present in the ground-truth chart; (3) Extraneous: The generated chart contains text not found in the ground-truth chart, e.g., adding text for titles, reflecting the model's s own interpretation of the chart; As depicted in Fig. 4 (b). the majority of errors are Missing and Extrancous, which indicates that for GPT-4o, due to intensive information in charts, it is challenging for it to comprehend the whole scope of the charts, even the basic text recognition task.
Type-related Errors. (1) Confusion: GPT-4o misinterprets the chart type as one that appears or functions similarly, such as mistaking violin plots for box plots, and so on; (2) Missing: The generated chart omits chart types present in the ground-trutt chart; (3) Extraneous: The generated chart includes chart types not present in the ground-truth chart. As demonstrated Fig. 4 (c). although GPT-4o exhibits remarkable capability in object recognition for natural images, it still struggles with scientific charts, which contain more nuanced semantic meanings through visual logic.
Color-related Errors. (1) Similar: The colors are not the same as the cround-truth colors but appear analogous. (2) Different: The colors are entirely dissimilar to the ground-tue -orun colors. As shown in Fig. 4 (d), though GPT-4o can not exactly recognize the accurate colors, it can identify similar ones.
To sum up. GPT-4o still faces challenges in code generation and exhibits notable visual understanding deficiencies ChartMimic. It has difficulty accurately recognizing visual elements in figures or may hallucinate incorrect elements and struggle with complex data analysis. Combining these insights, there is need for further improvement in both visual understanding and code generation.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Large Multimodal Models. The proprietary LMMs such as GPT-4o (OpenAl
2024). Gem-
ini (Google, 2023), and Claude-3 (Anthropic) 2024) have enabled complex multimodal interac- tions. Similarly, emerging open-weight LMMs such as LLaVA (Xu et al. 2024 Li et al., 2024a).
InternVL (Chen et al., 2023), Qwen-Vi VL (Bai et al. 2023). DeepSeek- VL (Lu et al., 2024) have contributed to the community. Despite these advancements, the effective evaluation of LMMs remains a major challenge. However, effectively evaluating LMMs remains a challenge, with open-source
models performing well in benchmarks (L.iu et al.] [023 Chen et al. 2024 Yue et al. 2023 Lu et al.
2023 Luo et al. 2025) yet falling short in practical applications (Xie et al. 2024] Koh et al. [024]] Si et al., 2024 Ji et al. 2024). This gap emphasizes the need for real- baredd evaluations that
reflect authentic use cases (Ma et al., 2024a b). ChartMimic addresses this by requiring LMMs to translate complex visual information into code, testing their visual and coding capabilities.
Code Generation.
Tasks such as HumanEval (Chen et al. 2021). MBPP (Austin et al., 2021). )
APPS (Hendrycks et al. 202I). and DS- (Lai et al. 2023) are important benchmarks in natural
language processing, but the inputs for these tasks are single-modal, consisting only of text, limiting their scope. With the emergence of LMMs, the evaluation of multimodal code generation has become increasingly critical for assessing real-world capabilities (Li et al. 2024bb Si et al.) [024]] Wu et al., 2024a| Ge et al., 2025). For example, MMCode (Li et al. 2024b) attempts to address this issue, but their visual inputs may be simplistic and face the problem of a single evaluation method. Design2Code (Si et al. 2024) evaluates LMMs' code generation abilities through HTML web page generation. However, their test data comes from the C4 (Raffel et al. (020) dataset, which may pose a risk of data leakage. Meanwhile, they just imitate the HTML and not take the customized instructions; we take this into consideration. Recently, Plor2Code (Wu et al.) (024a) undertakes work similar to ours, aiming to measure models' code generation abilities through chart-to-code generation. Similarly, their approach of directly scraping data from the matplotlib gallery poses a risk of data leakage. Our ChartMimic provides a new set of manually curated 4, 800 data pairs, ensuring chart diversity and offering more fine-graince evaluation methods.
Chart Understanding, ChartMimic evaluates the capabilities of LMMs in grounding chart under- standing into code generation, bridging visual and programmatic domains. Previous works focus
on chart question answering (Masry et al., 2022 Methani et al. 2020; Xu et al. 2023 Wang et al.
2024b
Li al. 2024c; Zeng al. (022) and chart captioning (Rahman et 2023 Kantharaj et al.
2022). They assess the LMMs' ability to understand specific data characteristics or summarize key in-
formation into text. ChartMimic advances the field by introducing a chart-to-code task, transforming the LMMs' understanding of charts into code, which is neglected before but realistic scenario for practical, real-world usage. This approach enables a comprehensive evaluation of the LMMs' overall comprehension of charts and their ability to express this understanding in code form. Leveraging the linguistic properties of code, our benchmark introduces fine-grained metrics to assess LMMs' chart understanding capabilities across multiple dimensions, including text, chart type, layout and color.
6 CONCLUSIONS
In this study, we develop the ChartMimic benchmark to evaluate LMMs' proficiency capability via chart-to-code generation. ChartMimic focuses on real-world applications for data visualization, aiming to assess LMMs' ability to harmonize a blend of cognitive capabilities, including visual understanding, code generation, and cross-modal reasoning. We propose two distinct levels of evaluation metries (low and high level) to provide comprehensive assessment. ChartMimic directly contributes to the understanding of progress towards artificial general intelligence, reflecting the expertise and reasoning abilities expected of skilled adults in various professional fields. Despite its comprehensive nature, ChartMimic, like any benchmark, has limitations. The manual curation process, although thorough, may introduce biases. Additionally, using scientific charts as information- intensive visual inputs to measure LMMs' multimodal code generation capabilities, while effective, still encounters domain-specifi, challenges. Our evaluation metric, despite considering most elements' similarity, does not uniformly score details of sub-icons, such markers. We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence. Future research could explore various aspects, such as multimodal reasoning prompt strategies, to further reduce the gap between open-weight LMMs and proprietary ones.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partly supported by the National Key Research and Development Program of China (No. 2024YFB2808903) and the Shenzhen Science and Technology Program ISGC20220811100000007)
REFERENCES
New
models
added to the
phi-3 family, available on microsoft azure.
May 2024.
URL https: https://azure.nrr
comm/nnus/bble
new models added to the the--hi-- 1111-3-0 f-- allllaaaallllllllllaaaa on wwteeeoooomiiiooieeeeeeeeeeeeeoe microsoftt azure/
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXii:2303.088774 2023.
AI@ Meta. Ilama 3 model card.
11ama3/
1lbbmmain/0003L1 CARE___AARmmm
Anthropic. Introducing the next generation of claude. 2024. URL wttptttp:///m htttps::/www..nthrrett.
[oom/news/claude-233 family
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprini arXiv:2308. /2966, 2023.
Hannah K Bako, Xinyi Liu, Leilani Battle, and Zhicheng Liu. Understanding how designers find and use data visualization examples. IEEE Transactions on Visualization and Computer Graphics, 29 (1):1048-1058, 2022.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language 2024.
models? arXiv preprint
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Zhe Chen, Jiannan WU, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint 1rXiv:2312.14238. 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-comtrolled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 2rXiv:2404.04475 2024.
Stephanie DH Evergreen. Effective data visualization: The right chart for the right data. SAGE publications, 2019.
Li Fan, Lee Ching- Hung, Han Su, Feng Shanshan, Jiang Zhuoxuan, and Sun Zhu, new era in human factors engineering: A survey of the applications and prospects of large multimodal models. arXiv preprint arXiv:2405.13422 2024.
Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng. Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, et al. Autopresent: Designing structured visuals from scratch. arXiv preprint arXiv:2501.009/2. 2025.
Google. Gemini: A family of highly capable multimodal models. models, ArXiv, abs/2312.1 2023. URL https://apt semanticsche ar. wtt.o///ooopss1::263335.

---

Published as a conference paper at ICLR 2025
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang. Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal Ilm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 2), 2021.
Yatai Ji, Shilong Zhang, Jie Wu, Peize Sun, Weifeng Chen, Xuefeng Xiao, Sidi Yang, Yujiu Yang. and Ping Luo. Ida-vlm: Towards movie understanding via id-aware large vision-language model. arXiv preprint ar arXiv:2407.07577 2024.
Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, et al. Chart-to-text: A large-scale henchmark for chart summarization. arXiv preprint arXiv:2203. 06486, 2022.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop. Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint 2ttp::2499.19917,,.. 2024.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zeutlemoye, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319- 18345, PMLR, 2023.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models?, 2024.
Xuanyu Lei, Zonghan Yang, Xinrui Chen, Peng Li, and Yang Liu. Scaffolding coordinates to promote vision-language coordination in large multi-modal models. arXiv preprint arXiv:2402. 12058, 2024.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li. Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal ca- pabilities in the wild, May 2024a. URL. https:://llvaaa github. lo/blog////
2024-05-10- setevaa-neet-tsrrgeee-eeeee 11ms/
Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, and Jing Ma. Mmcode: Evaluating multi- modal code large language models with visually rich programming problems. arXiv preprint 200.2400449986, 2024b.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooinng models. http://:/ggggaaaaooooa/aaaaaaaaa
aawaaaaaapaaaaaaaaaaa eval 2023.
Zekun Li, Xianjun Yang. Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, et al. Mmsci: A multimodal multi-discipline dataset for phd-level scientific comprehension. arXiv preprint arXiv:2407.04990 04400, 2024c.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman. Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 9493 9500. IEEE, 2023.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint ar ar0iv 00281/ 2023.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseck-vl: Towards real-worle vasion-language understanding, 2024.
12

---

Published as a conference paper at ICLR 2025
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
M Ronnier Luo, Guihua Cui, and Bryan Rigg. The development of the cie 2000 colour-difference formula: Ciede2000. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, 26(5):340-350 2001.
Ruilin Luo, Zhuofan Zheng, Yifan Wang, Yiyao Yu, Xinzhe Ni, Zicheng Lin, Jin Zeng, and Yujiu Yang. Ursa: Understanding and verifying chain-of-thougee reasoning multimodal mathematics. 2025.
arXiv preprint arXiv:2501.
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yeue Yeung Shum, Wei Liu, et al. Follow-your-click: Open-domain regional image animation via short prompts. arXiv preprint 2.Xix:24033082655 2024a.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. Follow- eellow-your-emoji:: Fine-controllable and expressive freestyle portrait animation. arXiv preprint MXXXi::24060110000 2024b.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Fanging Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre- training and multitask instruction tuning. arXiv preprint AXiv:2401.02368 2024.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In CVPR, pp. 1527-1536, 2020.
OpenAl. Gpt-4o, 2024. URL http::://ooeeaa..c./mmn/ee////// gpt 40 2024-05-13.
Accessed:
OpenCompass. Opencompass: A universal evaluation platform for foundation models, https //ggghhh.coom/o/oo cwwpas/ cttp:m/cppppppcccooppppppppaaaaa 2023.
Paul Parsons. Understanding data visualization design practice. IEEE Transactions on Visualization
and Computer Graphics, 2811)66656675, 2021.
Jorge Poco and Jeffrey Heer. Reverse-engincering visualizations: Recovering visual encodings from chart images. In Computer graphics forum, volume 36, pp. 353-363, Wiley Online Library, 2017.
Xuedi Qin, Yuyu Luo, Nan Tang, and Guoliang Li. Making data visualization more efficient and effective: survey. The VLDB Journal, 29(1) (9)3331-17 2020.
Annemarie Quispel, Alfons Maes, and Joost Schilperoord. Aesthetics and clarity in information visualization: The designer's perspective. In Arts, volume 7, pp. 72. MDPI, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, et al. Chartsumm: A comprehensive benchmark for automatic chart summarization long and short summaries. arXiv preprint arXiv:2304. 13620, 2023.
13

---

Published as a conference paper at ICLR 2025
Manolis Savva, Nicholas Kong, Arti Chhajta, Li Fei-Fei, Maneesh Agrawala, and Jeffrey Heer. Revision: Automated classification, analysis and redesign of chart images. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pp. 393-4033 2011.
Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujuu Yang, and Wai Lam. A thorough examination of decoding methods in the era of llms. arXiv preprint wrXiv24422.06922 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are from automating front-end engineering? arXiv preprint arXiv:2403.03163, 2024.
Steven A Sloman. The empirical case for two systems of reasoning. Psychological bulletin, 119(1):3, 1996.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengeheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint 2403:22003.14334, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2023a.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 747766 2023b.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406. 18521, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Sfain-of-thoughhe prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang. Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-moda large language models in code generation from scientific plots. arXiv preprint vxXiv:240..09900 2024a.
Tianhe Wu, Kede Ma, Jie Liang, Yujiu Yang, and Lei Zhang. A comprehensive study of multimodal large language models for image quality assessment. arXiv preprint urXiv:2403. 10854, 2024b.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07977 2024.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. LLaVA-UHD; an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint arXiv:2403./ /703, 2024.
Zhengzhuo Xu, Sinan Du Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint 13X2:23322159155 2023.
14

---

Published as a conference paper at ICLR 2025
Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, et al. Matplotagent: Method and evaluation for Ilm-based agentic scientific data visualization. arXiv preprint arXhv:2402.11453, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502 2023.
Xingchen Zeng, Haichuan Lin, Yilin Ye, and Wei Zeng. Advancing multimodal large language models in chart question answering with visualization-refereneee instruction tuning. IEEE Transactions on Visualization and Computer Gruphics, 2024.
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v (ision) generalist evaluator for vision- language tasks. arXiv preprint arXiv:2311.013/1 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623,, 2023.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zheng Zhu, Xiaofeng Wang, Wangho Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. Is sora world simulator? a comprehensive survey on general world models and beyond. arXiv preprint arX2v:2405.03520, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A Author Contributions
17
Data Annotation
17
B.1 Data Annotation Principles
17
B.2 Data Annotation Pipeline
18
B.3
Complexity Levels
19
Instruction Examples
20
Correlation Between Test Set and Testmini set
25
D Chart Taxonomy
26
Details of Evaluation Metrics
35
GPT-4o Score
35
Text Score
40
E,3 Layout Score
40
Type Score
Color Score
41
Model Configurations and Prompting Methods
43
Generation Configurations
43
Prompts
43
F3
Cases of Different Prompting Methods
46
Details of Models
50
Correlation with Human Evaluation
50
H
Correlation with Chart Understanding and Code Generation
52
Cases Error Analysis
53
Error Analysis of Weig-Weight Models
56
Ethics, Societal Impact and Scalability of ChartMinic
57
16

---

Published as a conference paper at ICLR 2025
A AUTHOR CONTRIBUTIONS
Yujuu Yang served as the principal advisor for this project. Cheng Yang and Chufan Shi acted as the student project leads.
Data Annotation. Yaxin Liu and Bo Shui co-led the data annotation process (Appendix B). Chufan Shi, Cheng Yang. Mohan Jing and Linran Xu helped to contribute to various aspects of this process. Xiaomei Nie provided expert guidance on data visualization and conducted quality control.
Code Implementation. Cheng Yang and Chufan Shi co-implemented the codebase for ChartMimic.
Experiments. Cheng Yang and Chufan Shi jointly conducted the evaluation of LMMs on ChartMimic. Yaxin Liu and Bo Shui jointly led the human evaluation.
Paper Writing. Cheng Yang and Chufan Shi finished the main manuscript. Junjie Wang and Yujiu Yang provided in-depth reviews and revisions. Yaxin Liu and Bo Shui created all visualizations presented in the paper and contributed to the Data Annotation (Appendix (B) and Chart Taxonomy (Ap- pendix D). Deng Cai, Xinyu Zhu, Siheng Li, Yuxiang Zhang and Gongye Liu contributed through proofreading, discussions and feedback.
B DATA ANNOTATION
B.1 DATA ANNOTATION PRINCIPLES
Diversity of Chart Types. Data visualization has become an essential tool for conveying informa- tion in various fields, and the design practices and requirements for different types of charts vary significanll(Parsonn
202T). Most of previous work (Han et al., 2023 Xu et al. 2023 Masry et al.. 2022 Kantharaj et al. 2022 Rahman et al.] (023) focus only on line, bar, pie charts, etc, which are commonly used within the field of computer science. However, with the increasing integration of LMMs into everyday tasks, people from all fields are starting to use generative techniques as daily assistants to enhance their design and creative processes when creating visualizations. In this light, enriching the spectrum of chart diversity is crucial for evaluating LMMs' proficiency in multimodal chart-to-code generation.
Balance of Charts Complexity. Charts serve visual aids for data presentation, enabling users promptly grasp the underlying patterns and significance within the data. It is essential to adopt the appropriate chart type and complexity level to effectively convey the information (Evergreen) 2019). Previous works have focused mainly on charts with only a single data format and low information density, which are rarely encountered in practical settings such as academic writing. Noting that LMMs like GPT-4o have already demonstrated outstanding data visualization capabilities, our focus on using charts that are actually employed in practice, such as those from research papers, and on selecting charts with varying levels of complexity when constructing our benchmark.
Reduction of Data Leakage. Recognizing that augmenting training data is a primary method for enhancing the performance of L.Mss has led researchers to more comprehensively exploit all accessible data during pre-training. However, this approach introduces the possibility of data leakage. especially when pre-training data might already include resources such the matplotlib gallery] or other pre-existing datasets, potentially resulting in inaccurate evaluations. To mitigate this issue, deliberately avoid the use of code that can be readily found online or code that could be auto- generated by large language models for chart creation in constructing our dataset, thereby reducing the probability of data leakage. To further illustrate the potential data leakage in matplotlib gallery, we select the code for 20 charts from the matplotlib gallery. Specifically, we provide the first half of the code as a prefix for Llama3-8B (AI@ Metal 2024) and let the model complete the remaining code. Then we calculate the edit distance between the generated complete code block and the ground truth. Similarly, we apply the same process the code in Direct Mimic, calculating the edit distance between the generated results and the ground truth. We find that the edit distance for the code in the matplotlib gallery is 22.1, while the edit distance for Direct Mimic's code is 39.8. This indicates that the code in Direct Mimic have larger edit distance, which further reduces the risk of data leakage compared to the matplotlib gallery ones.
https://matplott11 ib.org/
17

---

Published as a conference paper at ICLR 2025
Integration of Authentic User Requirements. Users from diverse domains such as finance, health- care, education, and engineering demonstrate unique needs and preferences for data visualization. These requirements extend beyond mere diversity; they demand the incorporation of charts capable of articulating complex and nulti-dimensional data, and conforming to domain-specifie aesthetic preferences(iin al., 2020; Evergreen 2019). By aligning our dataset construction process with these real-world demands, we enable a more relevant and precise evaluation of LMMs. Adopting this approach not only reflects actual user patterns but also steers the research community in the iterative improvement of LMMs. This focused development will also lead to an enhanced user experience and increased user satisfaction, the models become more proficient in meeting the sophisticated and varied needs of users.
B.2 DATA ANNOTATION PIPELINE
Select
Customize
Diversity and Information
Manual Manual Selection
General Filtering
Density Filtering
and Filling
and Instruction Writing
5 Augmentation
Figure 5: An illustration of the data annotation pipeline, which encompasses five key steps.
We present the detailed description about the five-step data annotation pipeline in this section. Fig. 5 demonstrates an illustration of the data annotation pipeline.
General Filtering. To obtain a high-quality dataset aligned with real-world use cases and to avoid data leakage, we initially scrape figures from source files of publications on arXiv that hold a CC BY 4.0 license and have a publication date after February 2024. We then extract figures in PDF format, yielding approximately 174, 100 figures across various domains (such as Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, Economics). We filter these figures based on file format and generation method, retaining only caaplotlib-geeerate PDFs, indicating that these figures can be reproduced using Python. This process results in a refined collection of 15,800 figures.
Diversity and Information Density Filtering. This stage involves two-phase process conducted by five domain experts from information visualization, digital media, industrial design, visual communication design, and computer science. The information visualization expert has three years of research experience in scientific visualization and visual analytics. The digital media specialist focuses human-computer interaction and multimedia design. The industrial design expert brings perspectives from user experience and product visualization. The visual communication design expert specializes in graphic design principles and information aesthetics. The computer science expert has extensive experience in data visualization programming and scientific computing.
In the first phase, these experts conduct a 7-day manual review of the 15,800 figures, focusing on visual diversity and information communication effectiveness. They reference the Matplotlib gallery, gradually identifying and finalizing chart type while reviewing the figures, and build corresponding type pools. For each new figure, they assess its visual elements-sun as layout, axes, line styles, marker styles, and against existing figures in their corresponding type pool. If the figure as long exhibits distinctive difference in at least one of these aspects, it is retained; otherwise. it is excluded. This process results in 1,295 figures being selected for the second phase.
In the second phase, these experts independently review the 1,295 figures and further select those figures they deem to exhibit significant distinctions and diversity. Figures selected unanimously by all experts directly included, while the remaining figures are subjected to majority voting system requiring at least 3/5 votes for inclusion. This rigorous process, which takes less than 3 days to complete, results in a final set of 279 figures.
https://arxiv.orgg
18

---

Published as a conference paper at ICLR 2025
Manual Selection and Filling, In addition to sourcing from arXiv, curate chart figures from diverse platforms such as the matplotlib gallery, Stack Overflow, and plotting-relate forums on Twitter and Reddit. These charts deliberately chosen for their distinctive styles, which are not present in our arXiv dataset. Consequently, we obtain 600 prototype charts for ChartMimic. This stage took us less than week to complete the data selection.
Code and Instruction Writing. We propose to manually write codes and instructions for ChartMimic based on the collected 600 prototype charts. To ensure annotation quality, team of skilled Python students in computer science with 6+ years of Python and matplotlib experience reproducee 600 prototype charts using Python 3.9.0 and matplotlib v3.8.4. Since the unannotated data in the figures cannot be fully restored, they can only be approximated when writing the code. This process generates 600 (figure, code, instruction) triplets for the Direct Mimic task and another 600 triplets for the Customized Mimic task by integrating data from various domains into the corresponding code and instructions, comprising 1,200 high-quality seed data.
users-Python annotators-mastee
Data Augmentation. After developing the seed triplets, proceed with manual data augmentation process. Python Annotators modify various elements each seed triplet, including data, color schemes, and mark styles, to create augmented triplets. For each seed triplet, we generate three additional augmented triplets. This process enhances our dataset, resulting total of 4, 800 triplets. The "Code and Instruction Writing" and "Data Augmentation" stages together take the data annotators approximately 1.5 months to complete.
B.3 COMPLEXITY LEVELS
Easy: bar_ 8
Medium: bar_
Eye Movement
Brainwave
Benettx-
Benedit
Rolobiity
Jum
20-25
13.%%
27.85
Rolability
55%
26.2%
45.2%
23.5%
14.2%
Privacy
24.1%
kmm
Privacy
2234
60
45
20
Strong Disugree
Disegree
Neutral
Agree
Strong Agree
(c) Hard: bar_21
Figure 6: Representative examples of chart complexity ChartMimic (Easy, Medium, Hard).
Our complexity assessment framework is established upon two fundamental criteria, designed to comprehensively evaluate both the visual and technical aspects of chart generation:
Visual Elements Complexity We systematically evaluate the sophistication of each chart through a comprehensive scoring mechanism applied to its constituent visual components. This encompasses the chart typology, data grouping structures, marker configurations, textual elements, chromatic schemes, compositional layouts, and coordinate systems. Each visual component quantitatively assessed
19

---

Published as a conference paper at ICLR 2025
based both its frequency and sophistication level (designated as 1-3 points for low/medium/high complexity, respectively). This multi-dimensional scoring approach ensures a thorough evaluation of the visual complexity inherent in each chart.
Implementation Complexity We incorporate code complexity as a quantitative metric, measured primarily through code length and structural intricacy. This parameter effectively captures the technical sophistication required for accurate chart reproduction, including the complexity of data preprocessing, visualization logic, and stylistic customizations. The implementation complexity provides insights into the programming challenges associated with each chart type.
Consequently, charts in ChartMimic systematically categorized into three distinct complexity levels, each representing specific combination of visual and implementation challenges:
Easy: Fundamental chart configurations featuring minimal visual complexity and straightforward implementation requirements (e.g., in Fig. 6(a) monochromatic bar charts with sparse textual elements and simplified data representation).
Medium: Charts exhibiting intermediate complexity in visual element composition or implementation requirements (e.g., in Fig. 6[b] dual-subplot bar charts incorporating grouped data structures, diverse chromatic schemes, and moderate textual annotations).
Hard: Charts demonstrating sophisticated visual elements or advanced implementation (e-g.. in Fig. [6(]] complex dual-subplot bar charts featuring divergent data patterns, extensive color schemes, comprehensive textual annotations, and substantial code complexity).
B.4 INSTRUCTION EXAMPLES
To illustrate instruction format and task requirements, we present examples (bar 28 and CB 29) with their corresponding (figure, instruction, code) triplets for both Direct Mimic and Customized Mimic tasks. Fig. 7 and Fig. g show the Direct Mimic and Customized Mimic tasks for bar chart (bar 28), respectively. The instruction provides guidance on creating the visualization. Similarly, Fig. 9 and Fig. 10 demonstratte the tasks for a more complex combination chart (CB 29).
20

---

Published as a conference paper at ICLR 2025
Task 1: Direct Mimic
bar_28 28
Instruction
Figure
You are expert Pytboe developer who specinlizes writing motpletiib code based given picture. found nice picure in correspondngg source available. your help generate Python code can reproduce the picture based on the picture I provid..
STEM paper,
image match the original Now, please give me the matplotibb oode that reproduces the picture below.
recessary figsize-(6,
Code
2/3
Labels and Title 2/3
Part 1: Importing Libraries
Part 3: Plot Configuration and
wox.set.ylim(yling (liimmmmmmmm ax set,
Rendering
ax syt_ylabeel(laael)
import matplotlib, pyplot as plt import numpy np
ax se._st__ ww.set_xiabelxxlabee
Plotting
sx.set_xticks(x)
fig. ax = pir.subblots((iggszoee 5))
ax ax. www.eetetiicoloolllaettttt......
ax wow.ledd left")
Part 2: Date Preparation
Adjusting figure size match the original inage's dimensions ax bar width, jpeg, width, label=labels(O]] hatch="//", edgecolor="blese bbackk,, color="white")
Data
Part Saving Output
datasets (*VOC 2012", 10O0 2017*]
plt. tight plt.fsvefig("bbs: ("bpp="2.__..fffe bbox_inch es="tight")
tit.tight_layouu(( laao__yya)))))
ipea [3.1, 1.6]
deepiscc [1.5. 2.5] ours [0.5, 1.0]
bx-bar( x, deepjscc, width,
Ground-truth Figure (Rendered from Code)
X-axis positions (o-ozeegg(Len(ssttsett)))
label=labeis(1,,
Bar width
hatchs*.."
width 8.2
edgecalor: black",
["JPEG", ofdm", "]URS*]
w/
310
ylim [0, 4.3]
ylabel "Tranentesion Delay (ma)"
ax bar width, ours, width, Label=1sbele[2), hatch="*x"" edgecoloz="bled color="white")
xlabel "Datasets"
Figure 7: An Example of the Direct Mimic task (bar 28), showing the (figure, instruction, code) triplet. Additionally, also display the ground truth figure rendered from the code for illustration.
21

---

Published as a conference paper at ICLR 2025
Task 2: Customized Mimic bar 28
Instruction
Figure
There a reforence image for you and I also have set of data about fuel efficiency of different types of traneport urban and rural unaas,
reforence image genantt Python maiplotlib that clurt ecadly style the reforence image and recurately represents my dam.
datasets ("Ubbun Trmmport" "Rural Transpont"]
electric 3.4]
hyhnd 2111
diesel [3.5, 2.8]
X-nxis positions mp.arangs(endtanees))
Bur width
labels Gibelss("leerree ("Electric, "Hybrid", "Diesel")
[0.4.5]
ylabel FFadl Effiesency (amily'
xlabel "hanspott Types"
Now. ouppui Python mappotiib code thai chart that exactly minrrs the style zeference and accurately represcuts my data.
Note: You must copy the data I provide completely into your code, and ensure that the code generate contains the data and is executable.
Nutc that it necessary use ligsioe-(6, S) set the imager sies.
Code
Labels swt._yymymmmlin)
Part 1: Importing Libraries
Part Plot Configuration and Rondering
set_stt_lee het.yylabel(yllal
import intelotlib.pypleer as plt Import nunpy ee np;
ax. ax.set ex.set_xlabbllxllbbe ax se__icctccks(
Plotting
fig. plt.euuplotta(figiizee 5))
aett...tticeeeeellaaattttttt lpppe left")
Part 2: Data Preparation
Adjusting figure size to match original image's dimensions bar width, electric, width, label=lebols(0). hatch="//* bbback" color="white") ex. ber
["Ubban Transport", "Rural Transport*]
Saving Output
electric [2.7, 3.4]
plt wet.tight_llytrrtt plt.sevefigh bbar_ bor_28..pff Abox__nnhees= "tightt
hybrid [1.9, 2.1]
diesel [3.5, 2.8]
X-axis positions
hybrid,
Ground-truth Figure (Rendered from Code)
(p..raang(lem((sttasett)))
width,
label=labeis(1),
hatch="..
RC3 mabel
width 8.2
odgecollo==bbaack"
lsbels ["Electric", "Hybrid",
color="white",
"Diesel"]
ylim [0, 4.5]
8x. bar(
ylabel Efficiency (kn/1)"
width, diesel,
xlabel "Transport Types"
label label_1sbels(2]. hatch edgecalor="ble "baack", color="shite"
Figure 8: An Example of the Customized Mimic task (bar 28), showing the (figure, instruction, code) triplet. Additionally, we also display the ground truth figure rendered from the code for illustration.
22

---

Published as a conference paper at ICLR 2025
Task 1: Direct Mimic CB_ 29
Instruction
Figure
expert Python developer who specializes writing maqploeiih based given picture. found a very picture a STEM paper, but there is commoponding source onde available. need your help to generate the Pythom code picture provid:
reproduse the picture
Note that it neceseary use figsize((10, 5) set the image size mutch the original size.
Now. please me the muppltiib code reproduces the picture below.
Code
Importing Libreries
Configuration and
Part Saving Output
Rendering
import natplotlib. pyplot as plt Import numpy as np
Adjust lsyout to prevent clipping and overlap pit.tight_layour(( plt.sovefig("Ca_22
Create figure with two subplots (1x2) shared y-axis fig. put.lubplotss1, 2, figsize=(10, 6), sharey=True) colors = ["skyblus", "salmon"]
Part 2: Data Preparation
Technology fields tech_fields
Plotting for i, (patents, color) in enumecate(zip(ppatent__.. patents_2022] colors)): (ere(i).barhttochh.FF patents, color=color, odgecolor="grayy
graphe
"AI",
"Blackchain", "Claud Computing",
"IoT",
"Rbbttccs,
"Biotech".
*VR",
"Cybersecuritt"!
Adding 1ine graph the
"Quantum Computing", "Nanotechnoloog""
axee{1).plot(psstent,
tech_fields, "o-", "87622"77)667")
Patent filings 2013
patents_2013 = [120, 50, 80, 70, 60, 30, 40, 10, 20]
Adding data labels for patents in zip(axes, (patents_2013, patents_2022]): for index, value in enumerate(patents): textvvvuue index, f" [value)*, va="center")
Ground-truth Figure (Rendered from Code)
Patent filings 2022 patents 400, 250, 180, 100, 150, 50, 90)
[500, 200,
x_label = "Number of Patents"
y_label "Technology Field"
axl_title "Patent Filings
titles setell)).et.iittle(xxttttt [ee[1)),ee_rritle(xx2.tttl beee0)).eet.ylbbelll.eeeee
2013"
ax2_title *Pteett Filings in 2022"
Figure 9: An Example of the Direct Mimic task (CB (CB_29) (1B__)), showing the (figure, instruction, code) triplet. Additionally, we also display the ground truth figure rendered from the code for illustration.
23

---

Published as a conference paper at ICLR 2025
Task 2: Customized Mimic CB
Instruction
Figure
you and also have of dats about the number registered vehicles different transporaaton modes 2013 and 2022. Please the reference image and genense Python matplotlib code that can create chart that exactly minors the style of the reference image and accurately represents my datn.
There reforence
Transportation modes
transport modes
"Automebiiles"
"Bieycles".
"Buses".
"Timnss".
"Planes".
"Ships".
"Mooooyylles""
"Scootens";
"Trmm".
"Helicoptens.
Number of registered vehicles in 2013 (in thousinds)
vehicles 2013 - 150000
15000, 20000, 3000, 7000, 5000, Number of registered vehicles in 2022 (in thousands)
vehicles, 2022 - [55000, 12000, 16000, 22000, 9000, 3500, 7500, 6000, 1200] label "Number of (Registrred Vebicles thoueandss"
Iabel "Iansportation Mode"
"Registered
in2 title 'Registened Vehicles in 2022*
Now, planse output the Python matplotib code that can create a chart that exactly mirroes the style of the reference image and accurasely represents my datn. Nute: You must copy that prvide completely into your cude, and ensure that vode you genurate contains the calls and is direcily excentable.
necessary use figsize 10, 5) image size.
Code
Lebeel "Numbex Registered
a/s
Adding data 1abels for vehicles in zip(axes, [vehicles_ 2013, vehicles_2022)): for index, enunerato(vehicles):
Part 1: Importing Libraries
Vehicles (in thousands)"
y_label "Transportation Mode"
import natplotlib, pyplot plt import numpy
til_title "Registezed Vehicles
2013"
ax2_title "Registered Vehicles in 2022"
tx.text(value + 50, index, f" [value}", vo="center")
Part 2: Data Preparation
labels titles
Transportation modes transport_modes
Part Plot Configuration and Rendering
[xes[0].eet title(sxl, title)
[[fl]]_sst__ttteee title) axes(0). setyyle)e__ylbbelll (y. (y_lsbel)
"Automabiles",
"Bicycles",
figure
"Buses",
subplots (1x2) and shared y-axis fig. axes subpllttl,,,,,,, 2, figsize=(10, 5), sharey=lzue) colors ["skyblue", "salmon"]
'Trains",
# Part 4: Saving Output
"Planes",
"Ships",
Adjust leyout prevent
"Motorcycles",
clipping overlap
Plotting the ber graphs for i, (vehicles, color) in mumerate(zip(vveiieee vehicles_ 2022)), colors)): (are(i))aar((ttaassor.. Number of registered vehicles in vehicles, colorscolor, 2013 (in thousands)
"Scooters",
(it.ou))t_ssyou( plt. tight lsyout()
'Trams",
pat. 'C3_ 29_ bbox, Aboc_inches "ight )
"Helicopters",
Ground-truth Figure (Rendered from Code)
edgecolor==gren set__l).ll_.ll Adding greph the
vehicles_2013 [50000, 10000, 15000,
8000,
7000,
5000, 2000, 1000]
Number of registered vehicles in 2022 (in thousands)
axee[ii plotveiices tzansport_noode todess. "o-", color= #f77667))
vehicles_2022 [55000. 16000, 22000, 9000, 3500, 7500,
6000. 2500, 1200]
Figure 10: An Example of the Customized Mimic task (CB 29), showing the (figure, instruction. code) triplet. Additionally, we also display the ground truth figure rendered from the code for illustration.
24

---

Published as a conference paper at ICLR 2025
Table 7: The ChartMimic benchmark with Direct Mimic task. We report the results of both Testmini and Test set.
Exec. Rate
Low-Level
High-Level GPT-4o
Model
Params
Test set
Overall
Text
Layout
Type
Color
Avg.
Proprietary
Testmini Test
93.2
81.5
89.8
77.3
67.2
79.0
83.5
81.2
GPT-4o
93.0
90.2
76.7
66.0
79.1
84.1
81.6
Open- Weight
Open-Weight
Testmini
52.5
23.6
35.8
16.0
15.4
22.7
24.2
23.5
InternVL2-2B
2:2B
Test
51.8
23.3
34.8
15.9
15.9
22.5
25.6
24.0
Intern InternVL.2-88
8.1B
Testmini
61.8
31.5
26.2
38,9
36.6
Test
61.6
33.4
47.4
25.7
24.2
32.7
39,9
36.3
Testmini
69.3
39.2
58.7
35.9
31.8
41.4
47.4
44.4
InternVL2-2GBB
26.0B
Test
69.9
41.7
58.0
35.6
31.0
41.6
48.1
44.9
Testmini
83.2
54.1
74.5
49.2
41.5
54.8
58.5
Intern wla.22/aaa.a.37770
76.08
Test
83.3
55.6
73.5
62.7
58.9
Table 8: The ChartMimic benchmark with Customized Mimic task. We report the results of both Testmini and Test set.
Model
Params
Test set
Exec. Rate
Low-Level
High-Level
Overall
Text
Layout
Type
Color
Avg.
GPT-4o
Pmoprietoy
Testmini Test
96.5
88.5
92.9
79.2
67.6
82.1
84.3
83.2
GPT-4o
96.2
87.2
91.7
80.1
66.4
81.4
84.8
83.1
Open- Weight
Testmini
49.3
22.2
35.4
20.0
18.1
23.9
27.8
25.9
InternV1.2-28
2.2B
Test
49.6
22.4
19.2
19.6
23.8
28.4
26.1
Testmini
73.0
43.1
54.4
35.4
43.2
48.9
46.1
InternVL2-881
8.1B
Test
73.5
43.7
54.1
41.1
34.1
43.3
49.8
46.5
Testmini
73.7
43.9
62.3
43.5
34.3
46.0
51.1
48.6
Intern InternVLL2-266 VL.2-66B
26.0B
Test
74.7
44.7
66.3
46.8
35.1
48.2
50.8
49.5
Testmini
89.8
50.5
62.7
66.7
64.7
Intern WL2--VLL22--33
76.0B
Test
88.1
57.9
79.6
65.6
51.7
63.7
68.2
66.0
C CORRELATION BETWEEN TEST SET AND TESTMINI SET
Tab. 7 and & reports the performance of GPT-4o and Intern series LMMs (2B, 8B, 26B, Llama3- 76B) on Direct Mimic and Customized Mimic task. The minor differences between scores on the test subset and the testmini subset suggest that testmini effectively mirrors the test subset, serving as valuable evaluation subset for model development, especially for those who have limited computing resources.
25

---

Published as a conference paper at ICLR 2025
D CHART TAXONOMY
This section presents the chart taxonomy in ChartMimic. It encompasses structure of 22 categories according to chart type characteristics and data composition. The categories comprise of:
regular types, ordered as follows: Bar, Heatmap, Scatter, Box, Errorbar, Errorpoint, Line, Violin, Radar, Pie, Density, Graph, Quiver, Contour, Histogram, Tree, Area, and 3D charts.
advanced types: PIP (Plot-in-Plot), Multidiff (Multiple Differences), Combination, and HR (Hard-to-Recoognize))
The regular types are further divided into subcategories according chart feature data charac- teristics, whereas each chart of the advanced types represents a unique subcategory. The taxonomy showcases examples to showcase diversity of each category.
Bar: Bar chart uses rectangular bars to represent data and can be distinguished by its ori- entation, horizontal or vertical, with its nuanced data attributes. There are subcategories of the data attributes for each orientation, consisting of 16 subcategories in total, as shown
in Figure
1. Base (single positive data set, unordered)
2. Sorted (data in ascending or descending sequence)
3. Grouped (multiple positive data sets, adjacent)
4. Stacked (multiple positive data sets laying atop one another)
5. Normalized (proportioned stacks of positive data summing to one, a Stacked variant)
6. Diverging (multiple stacked data sets expanding from central axis)
7. With-Negative (data sets including negative values)
8. Reverse (exclusively negative data sets)
Probubiiiyy Inprovemert over VUM image Encoder Baso lnd Retums
Swar Shepp
5.38
0.0
0.2
(1) bar 32
(2) bar $2
(3) bar 63
(4) bar 39
(5) bar 19
(6) bar 62
(7) bar 40
(8) bar_ 65
(9) bar 4
(10) bar_67 67
(11) bar
(12) bar
(13) bar
(14) bar
(15) bar
(16) bar
Figure 11: Examples Bar chart subcategories.
26

---

Published as a conference paper at ICLR 2025
Heatmap: Based the layout and visual representations, Heatmap is categorized into subcategories, each reflecting unique aspects of data presentation, as shown in Figure
1. Base (the general layout of typical heatmap)
2. Missing-Dare Data (visualization indicating the absence of data points)
3. Triangle-Layout (heatmap configured in a triangular layout)
4. Other-Shaped (heatmap comprising elements in non-rectangular forms, such circles)
(1) heatmap_3
(2) heatmap_4 6
(3) heatmap
(4) heatmap_10
Figure 12: Examples of Heatmap subcategories.
Scatter plot is classified based on the dot characteristics and data distribution. The taxonomy divided into subcategories shown in Figure [3
Scatter:
1. Base (basic scatter plot, uniform dot size, color may vary)
Diff-Shape (different dot shapes)
Diff-Size (different dot sizes, such as bubble chart)
4. Clustered (scatter plot with clear clustering)
(I) scatters 7
(2) scatters 4 (3) scatters 13
(4) seatters_ 12
Figure 13: Examples of Scatter plot subcategories.
Box plot characterized by the orientation of boxes and data characteristics. The orientation be horizontal vertical, while data characteristics include the grouping of data and the presence of missing lines. The taxonomy is divided into subcategories for each orientation, consisting of 6 subcategories in total, shown in Figure 14
Box:
1. Base (single group of data, complete box shape)
2. Grouped (multiple groups of data, complete box shape)
3. Missed-Line (missing parts below the first quartile line and above the third quartile line)
An Error bar chart is an enhanced variant of the basic bar chart, augmented with error margins to represent the variability or uncertainty within the data. Unlike Bar chart, Error Bar chart typically does not include the categories "Normalized" and "Sorted" the dimension of data attributes, as these are less common. Therefore, Error Bar chart classified into data attributes subcategories for each orientation, resulting in 12 subcategories in total, as shown in Figure 15
Error Bar:
1. Base (single positive data set, unordered)
2. Grouped (multiple positive data sets, adjacent)
3. Stacked (multiple positive data sets laying atop one another)
4. Diverging (multiple stacked data sets expanding from central axis)
5. With-Negative (data sets including negative values)
6. Reverse (exclusively negative data sets)
27

---

Published as a conference paper at ICLR 2025
(1) box
box_ 6
box _2_
box
(5) box 25
(6) box 16
Figure 14: Examples of Box chart subcategories.
(1) errorbar 7
(2) errorbar 3
(3) errorbar 16
(4) errorbar 26
(5) errorbar 28
(6) errorbar
Ne0 1994
(7) errorbar_
errorbar_ 8
errorbar 24
(10) errorbar
(11) errorbar
(12) errorbar 30
Figure 15: Examples of Error Bar chart subcategories.
Error Point: Error Point chart enhances the classic Scatter chart by introducing error bars to each data point, conveying the inherent variability or uncertainty of the data, When classifying Error Point chart, pivotal consideration lies in the orientation and symmetry of the error bars, leading us to define key characteristics that govern their taxonomy. These Syyartterissicss-ymmeetyyy and asymmetry in both horizontal and vertical orientations, coupled with composite category encompassing both
into 5 comprehensive subcategories, as illustrated in Figure 16 Here we contend the following delineations:
Orientation: Distinguishing the direction of error bars, which can 1. profoundly affect the interpretation of the data.
2. Symmetry-Asymmeerr: Acknowledging whether error bars exhibit mirrored consistency or an uneven distribution across data points.
(1) errorpoint
(2) errorpoint
errorpoint
(4) errorpoint
(5) enrorpoint 8
Figure 16: Examples of Error Point chart subcategories.
28

---

Published as a conference paper at ICLR 2025
Line: Line chart is classified based three primary attributes: data grouping, error visualization methods, and dot-line characteristics for grouped data, as shown in Figure T7
1. Data grouping: Single (individual dataset) Grouped (multiple datasets).
2. Error visualization methods: Base (no error), Striped-Eree Error (striped fill patterns), and Marker-Error (markers above and below data points).
Dot-Line Characteristics: For single dataset, No-Marker (data points without markers), Marker (data points with markers). For grouped datasets, Diff-Color (different colors for each group), Diff-Marker (different marker shapes for each group), and Diff-Line (different line styles for each group).
(1) line_22 _2
line 46
(3) line 24
(4) line 37
line
line 16
(7) line _3
(8) line_6n _1
Figure 17: Examples of Line chart subcategories.
Violin chart a combination the Box chart and kernel Density chart. It provides deep insight into the distribution of the data, indicating where individual data points fall within the overall data range. Based on the number of data groups and the shape and distribution of the violin form, the taxonomy is divided into 3 subcategories, as shown in Figure I8
Violin:
1. Base (standard shape, single data group)
2.
(standard shape, multiple data groups)
3. Grouped-Departee (half shapes joined, multiple data groups)
(1) violin_6
(2) violin
(3) violin _
Figure 18: Examples of Violin chart subcategories.
Radar: Radar chart, also known as spider chart or star plot, is a graphical method of displaying multivariate data
two-dimensional plane. It is particularly useful for showing performance metrics or skill assessments across multiple areas. Radar chart is classified based on three primary attributes: the number of data grouping, area fill, and dot-line characteristics for grouped data, as shown in Figure 19
1. Data Grouping: Base single dataset) or Grouped (multiple datasets).
2. Area Fill: FillArea (areas within the radar chart filled) or NoFillArea (areas without fill emphasize the outline).
3. Dot-Line Characteristics: For single dataset, NoMarker (dots without markers), Marker (dots with markers). For grouped datasets, Diff-Color (different colors for each group), Diff-Line (different line styles for each group), and Diff-Marker (different marker shapes for each group).
29

---

Published as a conference paper at ICLR 2025
radar
(2) radar 20
(3) radar
(4) radar 13
(5) radar
(6) radar
Figure 19; Examples of Radar chart subcategories.
Pie: Pie chart is circular statistical graphic that divides circle into slices to illustrate numerical proportion, whereas ring charts, also known as donut charts, utilize a hollow circle to serve similar purpose. The presence of ring, the number of layers, and the highlighted segment are the primary attributes in the classification of pie and ring charts, as shown Figure 20
1. Hollowness: Pie (no ring) Ring (with a ring).
2. Layering: SingleLayer (single data series) MultiLayer (multiple series categories).
3. Highlighting: Base (without highlighted segments) or Explode (with one or more segments emphasized capture viewer attention).
(1) pie_6
(2) pie_
(3) pie_ 2
(4) pie 9
(5) pie 14
(6) pie 12
(7) pie 15
(8) pie
Figure 20: Examples Pie and Ring chart subcategories.
Density chart conveys the concentration and distribution of data within space, often used to depict the magnitude or frequency across different areas or intervals. The orientation and data grouping are the primary attributes of the taxonomy density plots.
Density:
as shown in Figure 21
Orientation: Vertical (y-axis as density) or Horizontal (x-axis as density).
2. Data Grouping: Base (single dataset) or Grouped (multiple datasets).
(1) density
(2) density
(3) density
(4) density
Figure 21: Examples of Density chart subcategories.
Graph: Graph chart commonly represents relationships and interconnected data through nodes (also known as vertices) and edges. It is widely used to depict networks, path- ways, and complex inter-dependencies. The taxonomy of graph charts based on the directionality and weight of the edges, resulting in 4 subcategories, as shown in Figure 22
Directionality: Directed (edges have direction) or Undirected (edges have no direction).
2. Weight: Weighted (edges have weight) or Unweighted (edges have no weight).
30

---

Published as a conference paper at ICLR 2025
graph_ 1
(2) graph_ 5
(3) graph_? 3
(4) graph
Figure 22: Examples of Graph chart subcategories.
Quiver chart, also known as vector field plot, is used to display the magnitude and direction of vectors across two-dimensional plane. This type of chart is particularly useful physics and engineering to represent velocity fields, gradients, or other vector-based data spatially. The taxonomy of quiver charts is based the vector quantity and data grouping, resulting subcategories, as shown in Figure 23
Quiver:
Vector Quantity: Simple (limited vectors) Field (vector field).
2. Data Groups: Single (single group) Grouped (multiple groups).
(1) quiver 2
(2) quiver
(3) quiver_ 4
(4) quiver_ 3
Figure 23: Examples of Quiver chart subcategories.
Contour: Contour chart, also called contour map or isoline graph, is used to represent three-dimensional data on two-dimensional plane by plotting contour lines that connect points of equal value. This method is especially useful in fields like meteorology and geography, where visually communicates variations terrain elevation or changes meteorological elements like temperature and pressure. The taxonomy of Contour chart is based on the representation of the contour lines, resulting in 3 subcategories, as shown in Figure 24
1. Line (line representation)
2. Fill-Area (color-fille) filled representation)
3. Combination (both line and color-filled)
(1) contour
(2) contour_2
contour 5
Figure 24: Examples of Contour chart subcategories.
Histogram: Histogram chart, often referred as hist chart, is a representation of data distribution where the data is grouped into ranges or "bins" and illustrated bars show the frequency of data points within each bin. It is particularly useful for identifying patterns or anomalies in the data, such as skewness, peaks, or gaps in the distribution. The taxonomy of histogram charts is based the data grouping and positioning, resulting in 3 subcategories, as shown Figure
1. Base (single dataset)
2. Overlaid (overlapping multiple datasets)
3. Stacked (stacked multiple datasets)
31

---

Published as a conference paper at ICLR 2025
(1) hist
hist 4
(3) hist
Figure 25: Examples of Histogram chart subcategories.
Treemap: Treemap displays hierarchical data through nested rectangles, where each branch of the tree is represented by rectangle that contains smaller rectangles corresponding to sub-branches. This method allows for efficient use of space, enabling the viewer to quickly compare sizes and proportions within the hierarchy, and is especially useful for analyzing large datasets to reveal relationships and patterns. The taxonomy of treemap charts based on the compactness and edge presence, resulting 4 subcategories, shown in Figure 26
1. Tight-Edge (compact with border)
2. Tight-NoEdde (compact without border)
3. Loose-Edge (loose with border)
4. Goose-NoEdge (loose without border)
sagges
(1) treemap
(2) treemap_
treemap
treemap_2
Figure 26: Examples of Treemap chart subcategories.
Area chart is graphical representation where data points are connected by line segments and the area between the line and the axis is filled with color patterns, providing a sense of volume. It is particularly useful for visualizing the cumulative magnitude of values time, allowing for clear perception of trends and changes in the data series. The laxonomy of area charts is based on the presence of markers, resulting in 2 subcategories as shown in Figure 27
Area:
1. Base (without markers)
2. Marker (with markers)
(1) area
(2) area
Figure 27: Examples of Area chart subcategories.
3D charts: 3D chart extends two-dimensional charting into three dimensions with spatial representations, offering an added layer of depth to represent additional data variables or to enhance visual appeal. Based on the above chart types and the additional surface representation in 3D space, we classify chart into 5 subcategories, as shown Figure 28
hatl
1. Scatter (3D scatter chart)
2. Surface (3D surface chart)
3. Line (3D line chart)
32

---

Published as a conference paper at ICLR 2025
4. Bar (3D bar chart)
5. Density (3D density plot)
(1) 3D_8
(2) 3D33
(3) (D_
(4) 3D_7
(5) 3D> 14
Figure 28: Examples of 3D chart subcategories.
PIP chart insets magnified reduced portion of the main plot highlight key data features, as in Figs. (1) (1) and (2). They enhance readability for complex datasets and facilitate comparative analyses, demonstrated in Figs. 29 (3) and (4).
PIP:
likf....
(1) PIP
(2) PIP 2
(3) PIP
(4) PIP 9
Figure 29: Examples of PIP charts.
Multidiff: A Multidiff chart combines least two different chart types across multiple subplots, with each subplot presenting one type. Derived from the categories above and using diverse layouts, Multidiff charts offer numerous configurations, shown in Fig. 30
ntl MI!!
(I) multidiff 10
(2) multidiff 20
(3) multidiff 21
(4) multidiff 24
Figure 30: Examples of Multidiff charts.
Combination chart merges features from different chart types into one plot, offering multilayered presentation. Unlike Multidiff chart with multiple subplots for different categories, Combination chart displays multiple categories in single plot. As in Fig. 31 (2), a scatter plot illustrates the data distribution while adjacent density plots detail the axis-specific spread. Additional examples are shown in Fig. 31
Combination:
MMM
Combination 2
Combination 6 (3) Combination 11 (4) Combination 26
Figure 31: Examples of Combination charts.
HR: An HR chart is that defies the above 21 categories is challenging to identify. HR chart typically modified from common charts with distinctive features like custom visual arrangement atypical markers, as shown in Fig 32
Tnn
33

---

Published as a conference paper at ICLR 2025
(1) HR_3
(2) HR 4
(3) HR_100 10
(4) HR 20
Figure 32: Examples of HR charts.
34

---

Published as a conference paper at ICLR 2025
E DETAILS OF EVALUATION METRICS
In this section, we present the details of evaluation metrics, including GPT-4o score for high-level metries and Text Score, Layout Score, Type Score and Color Score for low-level metries, For more information, please refer to our source code.
E. GPT-40 SCORE
We employ GPT-4o (OpenAl 2024) to assess the extend to which the generated figure corresponds to the ground-truth figure. The specific content of the prompt is presented in Fig. 33 Specifically, input both the generated and the ground-truth figures into the GPT-4o simultaneously. Then, GPT-4o instructed evaluate the similarity between the two figures, taking into account six dimensions: text, layout, type, data, style, and clarity. Subsequently, GPT-4o outputs a score ranging from 0 to 100 to represent the degree of similarity between the figures.
Prompt for GPT-4o Score
You are an excellent judge at evaluating visualization chart plots. The first image (reference image) created using ground truth matplotlib code, and the second image (AI-generated image) is created using matplotlib code generated by an AI assistant. Your task is to score how well the Al-generated plot matches the ground truth plot.
### Scoring Methodology:
The Al-generated image's score is based on the following criteria, totaling score out of 100 points:
Chart Types (20 points): Does the Al-generated image include all chart types present in the reference image (e.g., line charts, bar charts, etc.)?
2. Layout (10 points): Does the arrangement of subplots in the Al-generated image match the reference image (e.g., number of rows and columns)?
3. Text Content (20 points): Does the Al-generated image include all text from the reference image (e.g., titles, annotations, axis labels), excluding axis tick labels?
4. Data (20 points): How accurately do the data trends in the Al-generated image resemble those in the original image and the number of data groups the same in the reference image?
Style (20 points): Does the Al-generated image match the original terms of colors (line colors, fill colors, etc.), marker types (point shapes, line styles, etc.), legends, grids, and other stylistic details?
6. Clarity (10 points): Is the Al-generated image clear and free of overlapping elements?
### Evaluation:
Compare the two images head to head and provide a detailed assessment. Use the following format for your response:
Comments:
Chart Types: S(your comment and subscore)
Layout: $[your comment and subscore)
Text Content: $[your comment and subscore)
Data: yoyour comment and subscore] }
Style: S(your comment and subscore) Clarity: $ (yyuur comment and subscore)
Score: S(your final score out of 100)
Please use the above format to ensure the evaluation is clear and comprehensive.
Figure 33: Prompt for GPT-4o Score.
35

---

Published as a conference paper at ICLR 2025
Task 1: Direct Mimic bar 28
Instruction
Figure
Pythom developer who specializes writing mappoolib code based picture. found STEM piper, but there no conespending source code avaiable. need your help to generate Python code that can reprodace the picture based on the pioture I provide
You
image march the original Now. please give me the matplotibb code that reproduccs the picture below.
necessary
figsizert6,
00
LMM
GPT-4o Scorer
Ground-truth Figure (Rendered from Code)
Figure
High-leve level Scores
Comments:
"Ciurt Types (20 " The Allgenerated imnge includes the same bur chart type as the reference image.
Subscore: 20:20
**Lyyuut (10 points)** The layout of Al Allgenerated image matches the reference image, with subplat.
Subssure: 10/10
T*xxxt
points)**; Al-generuted imaae includes
image, includdng legend
Subscore: 20/20
Dain (20 *e The data trends the Al-generated image accurately resemble those in the reference image, with the same number data groups and similar values.
Subssure: 20/20
*SStye points)**; The Al-ganernted image not the original tenms generated image uses different oolors for each har group. However, the marker types and patterns are cerrectly replicated.
reference image in graysoale, AL-
Subseure:
Claarity * The Al-generated clear and free of overlapping elements, similar the reference image.
Subscore: 10/10
Score: 85/100
Figure 34: Example of GPT-4o's scoring results for Direct Mimic example (bar_20
Stability of Evaluation with GPT-4o Score. We conduct the high-level evaluation for GPT-4o on the Direct Mimic task for times to assess the stability of GPT-4o Score. The result indicate mean GPT-4o Score of 83.4 with a standard deviation of 0.08, demonstrating the stability of GPT-4o Score.
Cost of Evaluation with GPT-4o Score. single-round evaluation with GPT-4o Score the Direct Mimic task approximately costs $5.25. Utilizing OpenAl's batch services further reduces this cost to $2.63 per round.
Longevity of Evaluation with GPT-4o Score. The GPT-4o Score metric is designed with long-term viability in mind. Our approach provides a meta-evaluation framework that can adapt to evolving language models. While currently leveraging GPT-4o, the method is model-agnostic and can be implemented with other advanced LLMs such Claude or Gemini. This adaptability ensures that as more capable models emerge, the evaluation process can seamlessly transition to utilize these superior systems. Importantly, this approach has demonstrated increasing correlation with human assessments as model capabilities improve, as evidenced by studies like AlpacaEval (Li et al. 2023) and MT-Bench (Zheng et al. 2023).
GPT-4o Score Examples. To demonstrate evaluation framework, we present GPT-4o's scoring examples for both Direct Mimic and Customized Mimic tasks. Figs, 34 to 37 show the evaluation
results for representative charts of different complexity levels (bar 28 and CB CB_29). These examples demonstrate how GPT-4o systematically evaluates various aspects of chart reproduction across different chart complexities and task types.
Correlation Coefficient Comparison with CLIP Score. CLIP Score (Radford et al. 2021 is widely used for assessing figure similarity. However, our preliminary experiments indicate that it struggles
36

---

Published as a conference paper at ICLR 2025
Task 2: Customized Mimic bar___
Instruction
Figure
There reference image for and also have set of data about fuel etficiency of differear types of tmneport urban and nural arcas,
reforence image and genenne Python maplotiih the referencee imzge and accurately represents my data.
chart exactly minrrs style
daracess = ("Lirh T'mnggerr" "Rural Trmepoot")
electii 3.47.
hybrid 2.11
dicsel - 2.8]
X anis positions npp.rangsllendhasssss)
Bar width
width -0.2
labods ("Eletric", "Hybrid", "Diesel))
ylim (.5)
ylabel - "Fael Efficiency (imily'
xlubel "Tranport Types"
New, ouppt Python marpkotlih
chart that exactly of reference image
accurately represents my daia.
Note: You copy the data provide complesely into codk, and that the code contains data and
direetly excemtable.
Note that necessary use figsizertti, 5) set the image size.
EQ
00
LMM
GPT-4o Scorer
Ground-truth Figure (Rendered from Code)
Generated Figure
High-level Scores Scores
Comments:
**Chat Types (20 points)**; The Al-genematd image includes chart type the reference image.
Suhseare: 20/20
**Layout (10 " The amngeemnn the subplots in Al-geuerated image matehes the reference inngo, with a single subplot in both. Subssore: 10/10
**Tuxl Content (20 "o The Al-gemerated imaaee includes all the text from the reference image, such as the litle, axis labels, and logend. Subseaoe:: 20/20
**Duta (20 * The data trends in the Al Al-generated sccurately resemble those in reference and the sumber data groups
the same.
Subseore: 20/20
**Style (20 points)**: The Al-generated image does not match the original in terms of colors. The reference image in black and white, while Al-generated image uses colors (bluc, orange, greea) The marker types patterns) correctly replicntdd, the abseuce of color signifieant deviation.
Subszore: 5/20
**Clarity (10 points)**; : The A Al-generated image is clear and free of overlapping elements, The bars and text well-defined. Subseore:
Figure 35: Example of GPT-4o's scoring results for a Customized Mimic example (bar_28).
37

---

Published as a conference paper at ICLR 2025
1: Direct Mimic CB
Instruction
Figure
You are expen. Pythom developer who specializes in writing mulplotib code hased on given picture. I fiund a vary nice eocrespundnng source code avuilbllc. your help geuante the Python that can reproduce the picture based on the picture provide. figsizer 10, the image match the original size.
STEM puper,
Note that
Now. please give me the cutplotiib codk that reproduces the picture below.
EQ
LMM
GPT-4o Scorer
Ground-truth Figure (Rendered Code)
Generated Figure
High-leve Scores
Comments:
**Chart Types (20 * The Al-generated image includes the same chart types as reference image, with harizomial bur charts and lime plots.
Subbsooee: 20/20
**Layout points)** amangementt subplots Al-generated image
imuge, with two now.
SSubscore: 10/10
TTxxx Content (20 * The Al- Al-genertted image includes all the text from the reference image. inclading titles and axis labels. However. the order the technology fielés the y-axis is reversed.
SSubsoore: 15/20
generated image accurately those image, Gills on the y-axis is invertd. The number of data groups is the same.
**Data points)**:
technology
Subseore: 10:20
bers and lines, as well as the markers, are consistent with the reference image. colors, marker types, other stylistic details. The colors the
**Slyle (20 "o The Al-generated image matches the original
Suubsore:: 20:20
*Clariity (10 points)**: The Al Al-generated image is clear and free of overlapping elements. The text and data points are well well-space and legible. Subseore: 10/10
Score: 85/100
Figure 36: Example of GPT-4o's scoring results for Direct Mimic example (CB_29).
38

---

Published as a conference paper at ICLR 2025
Task 2: Customized Mimic CB_29
Instruction
Figure
There rofoenoe image for you and alse have a set of data about the number of registered vehicles for differest reference image and genense Python mutplotiib code
tran-porttion modes 2013 and 2022. Please create chart that exardy minors the style of the reference image and accurately regeesents my data.
innepporaaion modes
transport_ modes [[ "Autormdiiles",
"Bicycles".
"Buses".
"Trains'.
"Plames".
"Ships",
"Motoneyoles",
"Scootens",
"Trams".
"Helicoptens",
Number registored vebicles in 2013 thousends)
vehicles, 2013 (50000, 10000, 15000, 30000, 8000, 3000, 7000, 5000, 2000, 1000) registered vehicless 2022 - [55000, 12000, 16000, 22000, 9000, 7500, 6000, 2500, 1200] Registered
thousands)
lobel
y Jabell "Iaanppottiion Mode
tide "Registered Vehicles 2013"
ax2 tikc
"Registered chicles 2022"
New, pleuse output Python marplecihh and accurately represents my data. Note: You copy data provide complotely your ensure you gonernte and is direetly esceutable.
cratte exactly mirress style image
Note that
figsize (0, the image size.
oo
GPT-4o Scorer
Ground-truth Figure (Rendered from Code)
Generated Figure
High-level Scores
Comments::
**Chart Types points)**; The Al-genented image includes chart types present image, with overlaid line plocs.
Subscore: 20/20
**1 **yyt (10 ** The amangement sabplots the Allgenerated image matches the reference image, with subplots arranged side side.
Subscore: 10/10
**Text Content (20 points)**: The Al Al-generated image includes all the text from the reference image, such titles and axis labels. However, the image includes additional Subscare: 10/20
not present Al generated image.
**Data (20 * The data trends in Al-generated image resemble those in the original image, and number of cata groups the sume.
Subscore: 20:20
**Style (20 pointsj**; The Al-generated image matches the original in terms of oolors, marker types, and other stllisicc details. However, the reference image includes additional labels buns, Subseore: 18:20
present generated image.
**Clarity (10 points)**; The Al-gonerated image is alear and free of overlapping elements. The additional data lahels do not obscure the visualication. Subseare: 10/10
Figure 37: Example of GPT-44oss scoring results for a Customized Mimic example (CB_29).
39

---

Published as a conference paper at ICLR 2025
to differentiate variations in types and other critical elements in charts, resulting in low correlation
coefficient of 0.53 with human evaluation. In contrast, as stated in Sec. [4.] GPT-4o Score achieves correlation coefficient of 0.70 with human evaluation. Therefore, adopt GPT-4o Score the high-level evaluation metric.
E.2 TEXT SCORE
Listing I: An exemplary Python code for logging text information.
from matplot ib. backends backend_pdff import RendererPdf
@rawed_texts = ||
def log_function (func):
def wrapper wrapper (Aargs, *<kwargs); global trxsed_texts
text_string args [4]
trawed_texts..ppaat text_string )
return func (targs, **wargs))
return wrapper
Gendereerrdf.draw..e oe
In order to accurately capture the textual content presented in the rendered figures, our code tracer monitors the function used to add text the rendered PDF, logging each textual element. exemplary Python code is provided in Listing I where we record the text elements by adding log wrapper to the "draw_text () function in the matplotlib package. Subsequently, we employ this approach to extract the text information from both the generated code and the ground-truth code.
Based on these two groups of texts, precision is defined as the ratio of the number of correctly captured ground-truth text to the total number of text in the generated figure. Recall is defined as the ratio of the number of correctly captured ground-truth text to the total number text in the ground-truth figure. The Fi-score, calculated using precision and recall, serves as Text Score.
E.3 LAYOUT SCORE
Listing 2: An exemplary Python code for logging layout information.
def get_griddpe__aaott__to info (fig):
***
Get the layout information of given figure.
Args:
fig (matplatlib. (igureeFFgur)) The figure to extract layout information from.
Returns:
layout info (list): A list of dictionaries, each containing layout information for each subplot in the figure.
layout_info ()
for ax in fig. axes:
spec ax. gee.e_ssbblottpeecccc ()
if spec None:
continue
gs = spec. gpocc.ge__gridppe
nrows, ncels = gs. (ee___eommtry))
row_start, row_end spec. rowspan.ttt spec. rowspan. stop
col_start, col_end coeccn spec. col span. stop
40

---

Published as a conference paper at ICLR 2025
dict (nrows=nrows, ncols=ncols, row_start-roo_start, zow_endFrow_end, col_start-oolsstrrt, col_end=col_end)
layout_info 1ist return laycut_info
layout_infs get_
The layout refers to the organization of subplots within a chart figure. In each figure implemented using matplotlib, multiple axes objects present, each containing layout information that specifies its position within the figure. To analyze the layout, we iterate through each axis object in the figure and obtain their respective layout information. An exemplary Python code accomplish this process provided in Listing 2 where the position information of each axis recorded. Subsequently, we gather the layout information from both the ground-truth code and the generated code.
Based on these two groups of layout information, precision is defined as the ratio of the number of correctly identified ground-truth layouts the total number of layouts in the generated figure. Recall is defined as the ratio of the number of correctly identified ground-trus trrth layouts to the total number of layouts in the ground truth figure. The Fi-score, calculated using precision and recall, serves as Layout Score.
E.4 TYPE SCORE
Listing 3: exemplary Python code for logging type information.
from matplctlib axes import Axes
import inspect
called_ = {)
def fog_function (func)
def wrapper (rapper<*args, **kwargs):
file_name = serrect.gefilleetuuune name file_name "/" +
called_functi [name] calleed_unctionss (name, 0)
result func (*args, **kwargs)
return func (vargs, **warggs)
return wrapper
Axos. bar = log_function (Axes bar)
The matplotlib package provides a variety of functions for easily generating diverse chart types, such as using "axes. .bar " to create bar charts. By monitoring the invocations of these functions, we can identify the types of charts being utilized. A successful invocation of plot functions indicates the incorporation of a specific chart type in the final visualization. Listing presents an exemplary Python code that demonstrates this approach, where we introduce logger to the "bar () function. Subsequently, we gather the chart types from both the generated and ground truth code.
Based on these two groups of chart types, precision is defined as the ratio of the number of correctly identified ground-truth chart types to the total number of chart types the generated figure. Recall defined as the ratio of the number of correctly identified ground-truth chart types to the total number of chart types in the ground- ttunt figure. The Fl-score, calculated using precision and recall, serves as Type Score.
E.5 COLOR SCORE
Listing An exemplary Python code for logging color information.
from matplot ib. import
41

---

Published as a conference paper at ICLR 2025
import inspect
drawed_colors []
def log_Eunction (
def wrapper (*apser(*args, **kwargs):
func_ name = gespect.getfile + "/" -unc... nare
result = func (+args, **kwargs)
for item result:
color item. fet_facccoler(
drawed_colors. append ( func_name - + color
Axes.be funetion (Axes.bar)
In the matplotlib package, each plot function returns chart type instance at the end of the function invocation. These instances contain various attributes, including those related to color properties, such as facecolor, edgecolor and colormap. To assess the color attributes, we employ the code tracer that captures the color information of each chart type instance. An example Python code demonstrating this approach provided in Listing 4 Subsequently, we gather the color information from both ground-truth code and the generated code and calculate the similarity between them.
noteworthy that through preliminary experiments, we find that using exact color matching resulted in very low color similarity, even slight variations would result in similarity score of zero. To address this issue, we employ the CIEDE2000 color difference formula (Luo al. (001), which converts the matching value between two colors from discrete [0, 1] scale continuous range between and 1. Finally, we calculate the maximum color similarity between the sets of colors in the ground-truie truth code and the generated code. Precision is defined the ratio of maximum color similarity to the total number of color in the generated figure. Recall is defined as the ratio of maximum color similarity the total number of color in the ground-truth figure. The Fl-score, calculated using precision and recall, serves as Color Score.
42

---

Published as a conference paper at ICLR 2025
F MODEL CONFIGURATIONS AND PROMPTING METHODS
F.1 GENERATION CONFIGURATIONSS
Following previous setups Wang et al. 2023b; Shi et al. 2024), for open-weight models, we set the temperature = 0.1 to achieve optimal results, while for proprietary models, set the lemperature т 0 for greedy decoding. For all models, we set the maximum generation length 4096. Additionally, inferred on A 100 80G GPU.
BF16 for model inference for open-weight models. All models
F.2 PROMPTS
We provide prompts for Direct, HintEnhanced, SelfReflection and Scaffold Prompting in Figs. 38 to 42 for Direct Prompting, we meticulously design separate prompt for open-source models to achieve optimal results, shown in Fig. 38 while the prompt for proprietary Models is shown Fig.
Prompt for Direct Prompting (Open-Weight Models)
You are an expert Python developer who specializes in writing matplotlib code based on given picture. found a very nice picture in STEM paper, but there is no corresponding source code available. need your help generate the Python code that can reproduce the picture based on the picture provide.
Please note that it is necessary size to match the original size. Additionally, will not provide you with the actual data the image, you have extract the actual data by yourself and based on the extracted data to reproduce the image. Ensure that the code you provide can be executed directly without requiring me to add additional variables.
(gsizc=((wwdth)) (height)) to set the image
Now, please give me the matplotlib code that reproduces the picture below.
Figure 38: Prompt for Direct Prompting Weight Models). (text) in blue font represents placeholders, which varies according to different test examples.
Prompt for Direct Prompting (Proprietary Models)
You are an expert Python developer who specializes in writing matplotlib code based on given picture. found a very nice picture in STEM paper, but there is no corresponding source code available. need your help to generate the Python code that can reproduce picture based on the picture provide.
Note that it is necessary match the original size. Now, please give me the matplotlib code that reproduces the picture below.
figsize=(widthh,, (height)) to set the image size to
Figure 39: Prompt for Direct Prompting (Proprictary Models). [text] in blue font represents placeholders, which varies according to different test examples.
43

---

Published as a conference paper at ICLR 2025
Prompt for HintEnhanced Prompting
You are an expert Python developer who specializes in writing matplotlib code based on given picture. found a very nice picture in STEM paper, but there is no corresponding source code available. need your help to generate the Python code that can reproduce the picture based on the picture provide.
To ensure accuracy and detail in your recreation, begin with comprehensive analy- sis of the figure to develop an elaborate caption. This caption should cover, but not be limited to, the following aspects:
Layout Analysis: e.g., identify the picture's composition, noting the presence and arrangement of any subplots.
2. Chart Type Identification: e.g. determine how many charts within subplot. Are they independent, or do they share a common axis?
3. Data Analysis: e.g., summarize the data trend or pattern.
Additional Features: e.g..identify any supplementary elements such as legends, col- ormaps, tick labels, or text annotations that contribute to the figure's clarity or aesthetic appeal.
Now, given the picture below, please first output your comprehensive caption and then use the caption to assist yourself to generate matplotlib code that reproduces the picture. Note that it is necessary to use (essize(((widi))) (height)) to set the image size match the original size.
Figure 40: Prompt for HintEnhanced Prompting. [text) in blue font represents placeholders, which varies according to different test examples.
Prompt for Scaffold Prompting
You are an expert Python developer who specializes in writing matplotlib code based on a given picture. found very nice picture in STEM paper, but there is no corresponding source code available. need your help to generate the Python code that can reproduce the picture based on the picture provide.
I will provide you with two images. The first image is the original picture. The second image is the picture overlaid with dot matrix of shape of (dot_matrix height) {dot_maa matrix_ width) help you with your task, and each dot is labeled with two-dimensional coordinates (x.y). Within each column, the x-corrdinate increases from top to bottom, and within each row, the y-coordinate increases from left right.
Please first use this dot matrix as reference anchors to generate the description of the picture (e.g., between dot A and dot B is something) and generate matplotlib code that reproduces the picture.
Note that it is necessary to use figsize=((width), (height)) to set the image size to match the original size.
Figure 41: Prompt for Scaffold Prompting. textt in blue font represents placeholders, which varies
according to different test examples.
44

---

Published as a conference paper at ICLR 2025
Prompt for SelfReflection Prompting
You are an expert Python developer who specializes in writing matplotlib code based on given picture. have code for implementing the reference picture as follows:
Now, I have the Python matplotlib code for implementing the reference picture as follows:
python
(python_ccom code]
The rendered picture of the code is:
Now, please compare whether the renderer picture is the same as the reference pic- ture. The difference may cover, but not be limited to, the following aspects:
1. Chart Types: Does the Al-generated image include all chart types present in the reference image (e-g.. line charts, bar charts, etc.)?
2. Layout: Does the arrangement of subplots in the Al-generated image match the reference image (e.g.. number rows and columns)?
3. Text Content: Does the Al-generated image include all text from the reference image (e.g.. titles, annotations, axis labels), excluding axis tick labels?
4. Data: How accurately do the data trends in the Al-generated image resemble those in the original image and the number of data groups the same as in the reference image?
5. Style: Does the Al-generated image match the original in terms of colors (line colors, fill colors, etc.), marker types (point shapes, line styles, etc.), legends, grids, and other stylistic details?
6. Clarity: Is the Al-generated image clear and free of overlapping elements?
If the generated picture matches the reference, please output the original imple- mentation code.
If there are discrepancies, first list the specific differences between the two pictures. Then, modify the existing code to address these differences, ensuring the revised code capable of reproducing the reference picture. Finally, output the revised code.
Note that it is necessary figsize=((width), (height)) to set the image size to match the original size.
Prompt for SelfReflection Prompting. (text) in blue font represents placeholders, which varies according to different test examples.
Figure 42:
45

---

Published as a conference paper at ICLR 2025
F.3 CASES OF DIFFERENT PROMPTING METHODS
We provide cases of HintEnhanced, SelfReflection and Scaffold Prompting in Figs. analysis of error cases related to Direct Prompting, please refer to Appendix]
43
46 For
<HintEnhanced Prompting
Human Labeling and Agreement Rias Checking
30
Do
Human
Response
Comprehensive Caption: The figure presented is bar chart featuring six grouped sets of vertical bars, each representing different human annotators labeled from 1 to along the x-axis. Within each group, there are five bars corresponding categories, making it grouped bar chart. The bars color-coded based on scores ranging from 1 to 5, allowing for visualization the distribution scores each annotator.
various
The height of each bar reflects count annotations assigned to specific score. Notably, the distribution of scores varies among the annotators. For instance, some annotators tend to have a higher frequency lower scores (1 and 2), while others display more balanced trends or different patterns altogether.
Humen Lebel Dngingg and Agroement Blas Chesking
25-
O
Figure 43: An error case HintEnhanced Prompting. The text highlighted in red is LMM's incorrect understanding regarding provided figures. The remaining part of response is omitted with "......
46

---

Published as a conference paper at ICLR 2025
<SelfRefiection Prompting
Morr Near 1re
Do
Response
Comparison of Images:
Chart Types:
Both images include line charts for
temperature and
precipitation.
Layout:
Both images have single chart layout without subplots.
Text Content:
The "Weekly Weather Data" appears generated images
should changed to "Weekly Weather Report" match the reference image.
Axis labels for temperature and precipitation are included in both images.
4. Data:
The data points for temperature and precipitation trends appear consistent across both images. The number of data groups is the same.
5. Style:
Color differences: The generated image appears to have some variations in colors. The generated image should use green for temperature and orange for precipitation.
Wessly Report
125
15.3
Figure 44: An error case of SelfReflection Prompting. The text highlighted in red is LMM's incorrect understanding regarding provided figures. The remaining part of response omitted with ".....". "
47

---

Published as a conference paper at ICLR 2025
<SelfRefiection Prompting
Do
Response
Here the specific differences between the two images:
Chart Types:
The reference image uses separate bar charts (pending and solved cases), while the generated image combines both into one chart.
Layout:
The layout of the reference image displays two distinct sections for pending and solved cases, while the generated image merges them into single horizontal bar chart.
3. Text Content:
The titles differ; the reference image has "Pending Cases by Country" and "Solved Cases by Country," whereas the generated image has "Pending and Solved Cases by Country."
The x-axis label in the reference image specifies "Pending Cases" and "Solved Cases" separately, while the generated image only uses " "Number of Cases."
Pencing Zaen Carllyy
Saleed Canis Cearty
-1500
Figure 45: An good case of SelfReflection Prompting. The text highlighted in blue is LMM's correct
reflection regarding provided figures. The remaining part of response is omitted with
48

---

Published as a conference paper at ICLR 2025
<Scaffold Prompting
106
Do
Response
Analysis of Images with Dot Coordinates
Axes and Labels:
The x-axis represents "Pre-training GPU Hours" and the y-axis represents "Accuracy [%]".
The axes are labeled with grid lines indicating increments in
GPU hours and accuracy percentages.
Data Points:
iBOT (dot at (3,1)): Approximately 81% accuracy and hours.
(dot (5,2)): Approximately 79% accuracy and 1000 GPU hours.
MoCo v3 (dot at (2,4)): Above 81% accuracy and 2000 GPU hours.
D2V2 (dot at (1,6)): Approximately 83% accuracy and 3000 GPU hours.
D2V22Refinee (dot at (1,7)): Approximately 85% accuracy and 4000 GPU hours.
3. Lines and Arrows:
ImageNet JK Linear Probing
07V2- (3.7)
covg
MaCo y c2.1
1000
ho- CPU Hours
Figure 46: An error case Scaffold Prompting. The text highlighted in red is LMM's incorrect understanding regarding provided figures. The remaining part of response is omitted with ".....".
49

---

Published as a conference paper at ICLR 2025
F.4 DETAILS OF MODELS
We list the evaluated models in Tab. 9
Model
Purams
Languuge Model
Vision Model
Model Code/API
GPT-4o
gpl-4o 2024
Cloude-3-opus
GeminiPProVision
gemini-pro-vission
76.0R
Alama-3-700m
200-3---OBIIntrrie
IntemViT--6B
InlemVL2-268
26.08
M2-208
IntemViI-6B
opensWWaab/.nterrnV222222
Cog-inn-llama-chht--B
19.28
Laaa-3-98-Instroas
EVA2-CLIP-E
MIDDX/sog2:2221Lanaa
Phi-3-Vision- 288--Instrcc
4.28
Phi-3
CLIP VIT-1/14
IDEFICS2-80 SB
7.6B
Mistral-TI 7B
Siyl.ip
LLaVA-NesttMissee Mistrall-B
7.68
Mistral-BB
CLIP VIT-L/14
DoepScck-VL-7B
7.38
DeckSock-78
SAM-B & SigLIP-L
MiniCPM-Llama3-222
8.48
Llama3-SB IB--t-ttetttttet
Sp
Qwen2.V1.-7
8.20
Qwen2 7B
VIT-600M
IniemL.
openskLab//.tteevvii--2
VL. 4LB
4.2B
Phi.3
Opon:Viaab/"nterrVVi2---
Qwen2-
2.6B
Qwen2-1.5B
Intem VL2-BB
2.28
InternL 12-1888
Intem
Open3VLab/Laterr/V2222
Table 9: Model code/API of the evaluated models.
G CORRELATION WITH HUMAN EVALUATION
Sammen
8.000
24
Chart Layout
Cierity
Data Trend
Check whother the sub- charts the shart the are erranged some way as original chert.
Whother the generated charts are clear and data read, without overlspping, blocking, blumiig.
Whether the cists trends in the generated
orgiinll
20
Same (10)
10
Chart Type
Text Content
Chart Style
Whether
generated contains in the original picture (auch as title, annotation,,
Compane the line/til color, mark (shape line style, etc). legend, grid and other style details in the genwnsted chart.
types in the original chart.
ett.)) you can ignore the acale text of the coordinate
Different
Enany dinirent
Parte same
Exacly
Partly the same (10)
20
Quachhy
Exectly
D
B
Figure 47: A screenshot of the human evaluation questionnaire.
50

---

Published as a conference paper at ICLR 2025
In order to evaluate the reliability of the proposed automatic metrics, calculate their correlation with human evaluations. Specifically, a selection of 300 test examples from the Direct Mimic task is utilized. Subsequently, gather the results generated by GPT-4o using four different prompting methods (Sec. 4.2) on these test examples. This process yields 1, 1,200 figures to be assessed.
Each figure is independently assessed by three evaluators through questionnaire, who assign scores on a scale of O to 100 based on the similarity between the generated figures and the ground -trutth figures. The final score for each figure is calculated the mean of scores given the three evaluators. Our evaluators comprise volunteer graduate students holding bachelor's 's degrees in computer science. To facilitate their assessments, we provide scoring rubric in the questionnaire, which closely aligns with the low-level and high-level metric evaluations. The scoring criteria encompass the following dimensions: chart type, layout, textual content, data, style, and clarity. A screenshot of the final questionnaire is available in Fig. 47
Upon collecting the human evaluation results, we calculate the Pearson correlation coefficient to ascertain the correlation between our automatic metrics and human evaluation.
51

---

Published as a conference paper at ICLR 2025
H CORRELATION WITH CHART UNDERSTANDING AND CODE GENERATION
Correlation Understanding
(b) Correlation with Code Generation
0. 177887
100
--0.7744 value - 4.92 05
Paaalee 6.56g-03
80
HumsnEval
CharQQA
40
60
50
ChartMimic
ChartMimii
Figure 48: Performance correlation of ChartMimic with benchmarks assessing chart understanding and code generation capabilities.
In an effort to shed light on the factors enhancing performance on ChartMimic, we explore the performance correlation between ChartMimic and existing benchmarks that evaluate chart under- standing and code generation capabilities. Specifically, select ChartQA (Masry et al. 2022) the benchmark for chart understanding and HumanEval (Chen et al. 2021) as the benchmark for code generation. We then calculate two Pearson correlation coefficients quantify these relationships: The correlation coefficient between the performance LMMs in the Direct Mimic task and their performance in the ChartQA task, denoted rcharts and the correlation coefficient between the performance LMMs in the Direct Mimic task and the performance of their corresponding LLMs in the HumanEval task, denoted Foode
As depicted in Fig. 48 the calculated values are Yakars = 0.6877 and Foode = 0.8714. These results indicate that both chart understanding and code generation abilities influence the performance in the Direct Mimic task, with code generation having more significant impact. This implies that for future model development aimed at enhancing multimodal code generation capabilities, it is crucial to focus on the foundational code generation abilities of LMMs.
52

---

Published as a conference paper at ICLR 2025
I CASES OF ERROR ANALYSIS
We provide cases of text-related, type-related and color-related enrors in Figs. 49 to 54 These cases encompass various error types mentioned in Sec. 4.4
Subgroups Calibration
1.00
Suberoups
Calbration curve
0.75
Fracian of postbres
0.5
0.25
-0.2
0.00
0.25
0.50 probability
0.75
0.0
0.2
redicted probubility
0.4
0.6
0.8
1.0
(a) Ground-truth figure (Combination 20)
Generated figure
Figure 49: Error Case 1. In this case, the errors include text-related errors of the Missing type and type-related errors of the Missing type.
1M
Enonciin Bitaros
Oboene
325
31
33
-103
Caligeras
(a) Ground-truth figure (bar 7)
(b) Generated figure
Figure 50: Error Case 2. In this case, the emoors include type-related errors of the Extraneous type. and color-related errors of the Different type.
53

---

Published as a conference paper at ICLR 2025
Clude_2.1 Buyer
-18
-18
-20
-20
GRC
Caude Gxaek
18
C74 Cluceed:
Clouted
CPT-3
OFF..5
(a) Ground-truth figure (errorbar_ie 12)
(b) Generated figure
Figure 51: Error Case In this case, the errors include text-related errors of the Detail and Missing type, and color-related errors of the Different type.
Kite
20
Agripra
(a) Ground-truth figure (multidiff' 9)
(b) Generated figure
Figure 52: Error Case 4. In this case, the errors include text-related errors of the Missing type, repatrelated errors of the Missing type, and color-related errors of the Different type.
Comparative Groups
Disribution Groups
-6
Contro Group
Tradmmnt Group
Group
Number Conercatirr
(a) Ground-truth figure (HR_20
(b) Generated figure
Figure 53; Error Case 5, this case, the errors include type-related errors of the Confusion type, and color-related errors of the Similar type.
54

---

Published as a conference paper at ICLR 2025
1.0
0.8
Accuracy
Accuracy
0.6
0.4
0.0
0.2
iTrans
PalmTree Method
CLAP
[Trans
PalmTree
CLAP
(a) Ground-truth figure (violin_4)
(b) Generated figure
Figure 54: Error Case 6. In this case, the errors include text-related errors of the Extraneous type, rype-related errors of the Confusion type and color-related errors of the Similar type.
55

---

Published as a conference paper at ICLR 2025
Siniar
Differant
Extraneous
21, .4%
13. 13.8% (9)
15.9% (10)
30.2% (19)
36. 9% (24)
44.4% (32)
14.3%
55. 6%
64.3% (9]
(40)
49 2% (92)
64.0% (34)
(a) Code
(b) Text
(c) Type
(d) Color
Figure 55: Error analysis of Intern WL2-LLamaa-733000 across four error types on the Direct Mimic task. The number in brackets indicates the count of error case.
Dimenaion Accees
Peremeter
Detail Extranecus
Missing
Confusion Extraneous
Micsing
Siillar
Different
17.2% (5)
13.3%
17. 17.2% (10)
36. 7% (22)
(15)
45,2% (28)
20.7%
(94)
(18)
56 9%
Code
(b) Text
(c) Type
(d) Color
Figure 56: Error analysis of Intern VL2-26B across four error types on the Direct Mimic task. The number in brackets indicates the count of error case.
J ERROR ANALYSIS OF WEIGHT MODELS
We conduct detailed error analysis on representative open- open-weight models of different scales, sampling 100 cases for each model. For large-scale models such as
and Intern VI.2- 26B (Fig. 55 Fig. 56). the error patterns align closely with those observed in GPT-4o. Specifically, Code-related Errors predominantly arise from dimensional inconsistencies, while Text-related Errors primarily manifest as missing elements. Type-related Errors mainly comprise missing components cases, and Color recogmition demonstrates the capability identify similar, if not identical, chromatic properties. In contrast, smaller-scat models like DeepSeek- -eo-eeek-VL-BB VL-7B (Fig. (57) exhibit markedly different error distributions. Code-related issues show significant increase Access and Parameter Errors (approximately 53.7%), while Text-related errors demonstrate a higher prevalence of missing elements (approximately 66.7%). Type-related cases reveal a substantial increase in missing errors (approximately 78.4%). Moreover, the degradation in color recognition is more severe, with the model failing to identify even similar colors. These findings indicate that while larger models encounter challenges similar GPT-4o, smaller models exhibit more fundamental difficulties in both code generation and visual comprehension.
56

---

Published as a conference paper at ICLR 2025
Siiier
15.9% (10)
17,5 (11)
15.7% 5.9% (8)
(3]
62 2%
66.7% (42)
(40)
(a) Code
(b) Text
(c) Type
(d) Color
Figure 57: Error analysis of DeepSeek-VL-7B across four error types on the Direct Mimic task. The
number in brackets indicates the count of error case.
K ETHICS, SOCIETAL IMPACT AND SCALABILITY OF CHARTMIMIC
ChartMimic designed as a comprehensive evaluation framework for advancing LMMs, with a strong emphasis ethical considerations, societal impact and scalability. Its primary objective is to provide researchers with a rigorous, ethically sound tool for assessing LMMs' capabilities across critical domains, including visual understanding, code generation and cross-modal reasoning.
The ethical integrity of ChartMimic is foundational to its design and implementation. Our dataset is derived from scientific domains, specifically using figures from arXiv papers distributed under the CC BY 4.0 license. This focus scientific content significantly mitigates potential ethical concerns related bias, representation, or sensitive personal information that often arise in datasets derived from social media general web content. The scientific nature of our data ensures: Minimal risk of perpetuating societal biases or stereotypes; Absence of personally identifiable information; Content that generally neutral and objective, focusing factual scientific representations. Our manual annotation process for code generation incorporates stringent ethical controls. Annotators are trained to ensure that the information conveyed in the code does not contain any biased content and strictly adheres to the factual representation of the scientific charts. This process further reinforces the ethical robustness of our dataset.
Ethical Considerations.
Societal Impact. By providing a benchmark rooted in scientific data, ChartMimic contributes to the development LMMs that can better understand and interact with complex, data-driven visualizations. This capability has far-reaching positive implications for scientific communication. data analysis, and knowledge dissemination across various fields.
Scalability. The architecture of ChartMimic is inherently extensible, featuring modular code- base that facilitates the seamless integration of additional chart types and evaluation metrics. Our data collection methodology leverages the continuous update cycle of arXiv. enabling sustainable expansion of the dataset while maintaining its focus on ethically sound, scientific content. The evaluation framework also demonstrates notable scalability: The concept of low-level metric can be readily adapted to other programming environments such as JavaScript and R. The high-level metric incorporates a meta-evaluation approach using LMMs, which allows for sustainable alignment with human preferences as stronger models emerge.
57


Reviews:
[{"review_id": "29uUHXwuUI", "reviewer": "Reviewer_LHtP", "summary": "The paper proposes a new benchmark, ChartMimic, consisting of (figure, instruction, Python code) triplets covering 4,800 charts that are manually annotated by  the authors after being extracted from arxiv papers (along with other manually-curated sources). The authors propose five new automated metrics for evaluating proposed generations. The authors conduct an empirical study of existing models on the benchmark, demonstrating (among other findings) that existing SOTA GPT-4o attains a score of ~82.2%; that open-source models lag behind proprietary models; etc.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Overall assessment\n\nThis is a relevant benchmark for a task likely of high interest and impact for both AI developers and AI users alike (chart generation). In general, the benchmark construction appears reasonable, but at times it is difficult to understand the details of the process and the authors do not provide quantitative details on the highly manual filtering process (these details seem important to assess the filtering procedure, since it relies extensively on manual curation by the authors). The empirical study is useful and shows some interesting, if unsurprising, results.\n\n# Major comments\n\n* In many cases, it would not be possible to fully reconstruct the original plot, as even at the pixel-level it would not be possible to fully recover exact data values. How is this handled in the \"direct mimic\" metric?\n\n* Does \"customized mimic\" always *only* involve changing the data of a plot? If so, please state this explicitly. Furthermore, I would suggest changing the name to something more specific and clear (to me, this is really \"direct mimic with new data\").\n\n* Related to the above point, it seems that there are two, completely independent, tasks being performed here. One is generating code to create a plot that exactly matches the aesthetic style of a given image. The other is recovering the *data* to be plotted, and applying the generated code to this data. I think it would be useful for the authors to also discuss this distinction in the paper, and to clearly motivate in which cases recovering the data is actually useful (in many cases, it seems we can simply take the data as given, and we would relaly just like to evaluate the correctness of the plotting code irrespective of the data it is applied to).\n\n* In the filtering stage, the authors state that in the first phase \"Charts with significant differences in complexity or information density are added to ensure diversity and effective communication\". How exactly is this process conducted, given that the pool consists of \"about 15,800 figures\" prior to the first phase?\n\n* There are many places in the dataset description where more detail would be helpful; for example \"We preserve the intersection of their selections and finalize the set through a voting process.\" What voting process is used? How many charts were filtered at this stage? In general, it would be helpful to know precisely how many data points were considered, and how many filtered out, at each stage of the pipeline. Please add this data to the paper.\n\n* Did the authors assess the impact of few-shot learning, chain-of-though prompting, or other methods on the benchmark tasks?\n\n# Minor comments\n\n* The abstract says the benchmark is comprised of \"(figure, instruction, code)\" triplets; but isn't data also a necessary component of the chart?\n\n* The abstract gives an \"average score\" but it isn't at all clear what this score represents. Given that the authors propse a set of new metrics in the benchmark, perhaps they could more clearly describe the composite score being reported.\n\n* Please include hyperlinks to relevant sections when referring to them (e.g. in L88-91).\n\n* Figure 2 isn't particularly illuminating; in fact I find it quite confusing. I would suggest to modify the layout substantially and add clear individual labels to the different boxes.\n\n# Typos etc.\n\n* L40: \", such real-life scenarios have yet to be fully explored.\" This should be a separate sentence.\n* L53: \"Based on this, we define two tasks, Direct Mimic and Customized Mimic (Sec. 2.1), which utilizes\" --> which utilize\n* L141: \"clarity and visual appeals\" --> clarity and visual appeal", "weaknesses": "see above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rEozffxZdq", "reviewer": "Reviewer_XbLj", "summary": "This paper introduces a new dataset ChartMimic, which tests LMM's ability to replicate a variety of different plots based on a set of carefully constructed set of examples and metrics. This paper evaluates multiple LMMs on ChartMimic including both proprietary and open models, finding that most models with the exception of GPT-4o perform quite poorly on the tasks in ChatMimic but also benefit from additional natural language information for customization tasks. In terms of prompting, the authors find that self-reflection is effective.", "questions": "Please address the specific questions raised in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Overall the writing is easy to follow and the paper is well-organized.\n- The work provides a clear dataset contribution that is not present in prior work as they overview in Table 2.\n- The evaluation is generally well done, covering many different models, prompting strategies, incorporating a comparison to human judgments, and an error analysis of the best-performing model.", "weaknesses": "While already did a good job with the presentation of dataset construction and evaluation, here are a few more suggestions that could further improve clarity:\n- Provide more examples of both direct and customized mimic tasks. In particular, I am curious how well-defined are instructions for customized mimic tasks are.\n- Provide clearer definitions of easy, medium, and hard tasks. This kind of naming obfuscates the actual nature of the tasks themselves. \n- Provide examples of GPT-4o scoring on different plots for sanity checks\n\nAnother suggestion is to conduct the same error analysis on open models, are there similar trends?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSEbDkdtHp", "reviewer": "Reviewer_XRf9", "summary": "This paper introduces ChartMimic, a new benchmark aimed at assessing the visually grounded code generation capabilities of large multimodal models (LMMs). ChartMimic uses visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. The benchmark includes 4,800 human-curated (figure, instruction, code) triplets across various domains like Physics, Computer Science, and Economics. Additionally, the paper proposes multi-level evaluation metrics for an automatic and comprehensive assessment of the output code and the rendered charts. The evaluation of three proprietary models and 14 open-weight models shows the substantial challenges posed by ChartMimic, with advanced models like GPT-4o and InternVL2-Llama3-76B achieving average scores of 82.2 and 61.6 respectively, indicating significant room for improvement. This benchmark is expected to inspire further development of LMMs and advance the pursuit of artificial general intelligence.\n\nThe real-world example in Figure 1 effectively illustrates the benchmark's motivation, addressing the common challenge of replicating well-designed figures computationally. This enriches the research significance by exploring new avenues in multimodal input.\n\nThe paper is overall well-crafted; however, it does necessitate further clarifications in several sections. I would like to highlight specific areas where additional details from the authors would be beneficial:\nFirstly, the manual annotation process described in section 2.2 ‘data curation process’ requires more comprehensive details. Given the significant involvement of human annotations, the quality of these annotations directly impacts the benchmark's effectiveness. It would be helpful if the authors could provide further information on the backgrounds of the annotators, the diversity within the team, the mechanisms for resolving any conflicts that arise during annotation, the duration of the annotation process, the versions of Python used, and approaches to managing incomplete figures. These details are crucial for assessing potential biases within the process.  The authors could include this information in an appendix or supplementary material if space is limited in the main paper.\n\n\nSecondly, the manual complexity assessment of charts mentioned in section 2.3 raises questions about the classification criteria.\n‘\nAdditionally, we conduct a manual complexity assessment for each chart, classifying them into 3 levels: easy, medium, and hard.\n’\nHow are charts categorized into easy, medium, and hard levels? What criteria are used, and can the authors provide examples of each level to enhance the transparency of this process? If the classification relies on the subjective judgment of the annotators, how is bias controlled? If objective indicators such as the number of elements is used, more detailed information would be valuable. I suggest authors include a table or figure showing examples of charts at each complexity level, along with the specific criteria used for classification.\n\n\nAdditionally, the use of GPT-4O scores in the evaluation metrics introduces concerns about the inherent instability of LLMs. It would be pertinent for the authors to explain how they address these variabilities to ensure reliable evaluation outcomes. The authors could consider conducting a stability analysis of this metric by executing the evaluation repeatedly and documenting the variability in the resultant scores.\n\nDo the authors plan to make all the data in the benchmark publicly available after publication? I recommend that during the review stage, the authors could use anonymous GitHub repositories or similar platforms to offer more detailed information about the benchmark, including data and usage tutorials. This would significantly enhance the thoroughness of the benchmark evaluation.\n\nLastly, I am curious if the authors have considered expanding this benchmark into a practical tool for figure-to-code tasks, possibly incorporating methodologies like Retrieval-Augmented Generation, to further its applicability and utility.", "questions": "Can you provide more details about the manual annotation process, including annotator backgrounds, diversity, conflict resolution, and handling of incomplete figures?\nWhat specific criteria are used to classify charts into easy, medium, and hard categories? Are these criteria subjective, or are they based on objective indicators?\nHow do you address the instability of LLM outputs in your evaluation metrics to ensure consistent and reliable results?\nHave you considered developing this benchmark into a tool for practical figure-to-code applications, possibly integrating methodologies like RAG?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Innovative benchmark;\nthe paper is good-writting;\nGood motivation.", "weaknesses": "Insufficient Detail on Annotation Process\nLack of Transparency in Complexity Assessment", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m40jmX3yzE", "reviewer": "Reviewer_gbor", "summary": "This paper presents a dataset named ChartMimic for evaluating LMM's ability to generate data visualization code based on (1) an image of a reference chart and (2) user instruction (either a reproduce-type of instruction or a custom instruction asking for creating a chart similar to the reference chart but for a new dataset). This dataset is curated from arxiv paper plots and online galleries following design guidelines to ensure both diversity of charts and high-quality ground-truth references (charts created by human). \n\nBased on this dataset, the paper presents an evaluation pipeline, leveraging GPT-4o's visual reasoning ability to compare chart generated by model and human. Besides a high-level similarity score, the paper introduces low-level metrics considering values logged from the visualization code (texts in the plot, layout, chart type func names, color attributes) to accompany evaluation. \n\nThis paper reports evaluation results of 3 closed-sourced models and 14 open source models with varying sizes. Results show the considerable gap between most powerful closed sourced model and open source model. The reports on how prompting strategy influence also point out interesting benefits of self-reflection. It's also interesting to learn that smaller model best benefits from direct prompting.", "questions": "Please checkout the previous section for questions. Please elaborate (1) comparison to related work (2) justification for the eval metric (3) discuss how individual ability relates to overall task performance.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper targets the practical problem of how chart authors reference existing visualizations to create their own visualizations, and the dataset/pipeline is fantastic for evaluating LMMs' ability jointly leverage visual and codegen abilities to solve coding task. In general this paper makes the following contributions:\n\n1. Chart reusing is a practical task studied in Visualization and HCI communities, this dataset bridges the gap of LMM development and practical chart authoring tasks.\n2. The dataset is sufficiently different from existing chart understanding (QA/summary etc) and codegen (HumanEval etc) that tests both visual reasoning and codegen together. It's a good addition for LMM evaluation tasks.\n3. The dataset coverages a diverse set of visualizations with quite convincing human curation process.\n4. The evaluate pipeline and results provide baseline for future LMM development.\n\nIn general, I find this paper interesting and would make good contribution to the field, pending responses to weakness and questions.", "weaknesses": "Here are some weakness of the paper.\n\n1. The comparison with related work is a little hand-wavy. Please (1) elaborate the comparison of chart type coverage, complexity and evaluation metric difference of this paper with close related work like Plot2Code and Design2Code, (2) add a few sentences to discuss the related work with computation notebook related work (e.g., DS1000 and following workstream) since many computation notebook completion tasks involve similar libraries and plotting. If possible, also add a few sentences on whether existing results on models' data science codegen ability aligns with the evaluation results found in this paper.\n\n2. The evaluation metric is not very convincing, especially high-level evaluation score, since existing work (e.g., Meng 2024, VisEval) has shown LMM's don't have impeccable visual reasoning abilities. I think the authors should elaborate why the metric here is convincing in the following ways:\n(1) does the high-level metric provide consistent results, with different runs, and with perturbation to images (e.g., manually change a small part of the chart to investigate if the metric is still reasonably conherent)\n(2) does the metric performs consistently across different types of charts? E.g., while the model may make precise assessment on popular chart types of bar chart, is the accessment consistent in less popular ones like contour chart?\n(3) how does low-level metric relate to high-level metric? \n(4) how does current evaluation approach compare to other LMM based evaluation approach?\n(5) ideally, show how the evaluation metric is coherent with human evaluation.\nIdeally, the authors should perform human evaluation to measure how good this current evaluation metric is. But if that is not the case, the authors should provide more rationales to convince the readers.\n\n3. As the chart mimic task involves both chart understanding and code generation tasks (as comopared in reproduction task versus custom data task), this paper missed an opportunity to identify how does models' chart understanding and code generation abilities individually contribute to the overall task. Some ideas of comparison: cross reference these models' performance on codegen task and chart reading task, to see how the model's scores or rankings in individual tasks relate to their overall performance in chart mimic. (there is no need to run additional experiment, but please include a discussion section about this topic.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)