ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation

Authors: Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran XU, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, Yujiu Yang
Primary area: datasets and benchmarks
Keywords: Dataset and Benchmark, Code generation, Chart Understand and Reasoning

Abstract:
We introduce a new benchmark, ChartMimic, aimed at assessing the visually-grounded code generation capabilities of large multimodal models (LMMs). ChartMimic utilizes information-intensive visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering.
ChartMimic includes $4,800$ human-curated (figure, instruction, code) triplets, which represent the authentic chart use cases found in scientific papers across various domains (e.g., Physics, Computer Science, Economics, etc). These charts span $18$ regular types and $4$ advanced types, diversifying into $201$ subcategories.
Furthermore, we propose multi-level evaluation metrics to provide an automatic and thorough assessment of the output code and the rendered charts.
Unlike existing code generation benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to harmonize a blend of cognitive capabilities, encompassing visual understanding, code generation, and cross-modal reasoning. The evaluation of $3$ proprietary models and $14$ open-weight models highlights the substantial challenges posed by ChartMimic. Even the advanced GPT-4o, InternVL2-Llama3-76B only achieved an average score across Direct Mimic and Customized Mimic tasks of $82.2$ and $61.6$, respectively, indicating significant room for improvement. 
We anticipate that ChartMimic will inspire the development of LMMs, advancing the pursuit of artificial general intelligence.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730488876462

thread_type:
official_review

<thread_object>
creator: reviewer_gbor

<content>
summary:
This paper presents a dataset named ChartMimic for evaluating LMM's ability to generate data visualization code based on (1) an image of a reference chart and (2) user instruction (either a reproduce-type of instruction or a custom instruction asking for creating a chart similar to the reference chart but for a new dataset). This dataset is curated from arxiv paper plots and online galleries following design guidelines to ensure both diversity of charts and high-quality ground-truth references (charts created by human). 

Based on this dataset, the paper presents an evaluation pipeline, leveraging GPT-4o's visual reasoning ability to compare chart generated by model and human. Besides a high-level similarity score, the paper introduces low-level metrics considering values logged from the visualization code (texts in the plot, layout, chart type func names, color attributes) to accompany evaluation. 

This paper reports evaluation results of 3 closed-sourced models and 14 open source models with varying sizes. Results show the considerable gap between most powerful closed sourced model and open source model. The reports on how prompting strategy influence also point out interesting benefits of self-reflection. It's also interesting to learn that smaller model best benefits from direct prompting.

strengths:
This paper targets the practical problem of how chart authors reference existing visualizations to create their own visualizations, and the dataset/pipeline is fantastic for evaluating LMMs' ability jointly leverage visual and codegen abilities to solve coding task. In general this paper makes the following contributions:

1. Chart reusing is a practical task studied in Visualization and HCI communities, this dataset bridges the gap of LMM development and practical chart authoring tasks.
2. The dataset is sufficiently different from existing chart understanding (QA/summary etc) and codegen (HumanEval etc) that tests both visual reasoning and codegen together. It's a good addition for LMM evaluation tasks.
3. The dataset coverages a diverse set of visualizations with quite convincing human curation process.
4. The evaluate pipeline and results provide baseline for future LMM development.

In general, I find this paper interesting and would make good contribution to the field, pending responses to weakness and questions.

weaknesses:
Here are some weakness of the paper.

1. The comparison with related work is a little hand-wavy. Please (1) elaborate the comparison of chart type coverage, complexity and evaluation metric difference of this paper with close related work like Plot2Code and Design2Code, (2) add a few sentences to discuss the related work with computation notebook related work (e.g., DS1000 and following workstream) since many computation notebook completion tasks involve similar libraries and plotting. If possible, also add a few sentences on whether existing results on models' data science codegen ability aligns with the evaluation results found in this paper.

2. The evaluation metric is not very convincing, especially high-level evaluation score, since existing work (e.g., Meng 2024, VisEval) has shown LMM's don't have impeccable visual reasoning abilities. I think the authors should elaborate why the metric here is convincing in the following ways:
(1) does the high-level metric provide consistent results, with different runs, and with perturbation to images (e.g., manually change a small part of the chart to investigate if the metric is still reasonably conherent)
(2) does the metric performs consistently across different types of charts? E.g., while the model may make precise assessment on popular chart types of bar chart, is the accessment consistent in less popular ones like contour chart?
(3) how does low-level metric relate to high-level metric? 
(4) how does current evaluation approach compare to other LMM based evaluation approach?
(5) ideally, show how the evaluation metric is coherent with human evaluation.
Ideally, the authors should perform human evaluation to measure how good this current evaluation metric is. But if that is not the case, the authors should provide more rationales to convince the readers.

3. As the chart mimic task involves both chart understanding and code generation tasks (as comopared in reproduction task versus custom data task), this paper missed an opportunity to identify how does models' chart understanding and code generation abilities individually contribute to the overall task. Some ideas of comparison: cross reference these models' performance on codegen task and chart reading task, to see how the model's scores or rankings in individual tasks relate to their overall performance in chart mimic. (there is no need to run additional experiment, but please include a discussion section about this topic.)

questions:
Please checkout the previous section for questions. Please elaborate (1) comparison to related work (2) justification for the eval metric (3) discuss how individual ability relates to overall task performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730563863554

thread_type:
official_review

<thread_object>
creator: reviewer_xrf9

<content>
summary:
This paper introduces ChartMimic, a new benchmark aimed at assessing the visually grounded code generation capabilities of large multimodal models (LMMs). ChartMimic uses visual charts and textual instructions as inputs, requiring LMMs to generate the corresponding code for chart rendering. The benchmark includes 4,800 human-curated (figure, instruction, code) triplets across various domains like Physics, Computer Science, and Economics. Additionally, the paper proposes multi-level evaluation metrics for an automatic and comprehensive assessment of the output code and the rendered charts. The evaluation of three proprietary models and 14 open-weight models shows the substantial challenges posed by ChartMimic, with advanced models like GPT-4o and InternVL2-Llama3-76B achieving average scores of 82.2 and 61.6 respectively, indicating significant room for improvement. This benchmark is expected to inspire further development of LMMs and advance the pursuit of artificial general intelligence.

The real-world example in Figure 1 effectively illustrates the benchmark's motivation, addressing the common challenge of replicating well-designed figures computationally. This enriches the research significance by exploring new avenues in multimodal input.

The paper is overall well-crafted; however, it does necessitate further clarifications in several sections. I would like to highlight specific areas where additional details from the authors would be beneficial:
Firstly, the manual annotation process described in section 2.2 ‘data curation process’ requires more comprehensive details. Given the significant involvement of human annotations, the quality of these annotations directly impacts the benchmark's effectiveness. It would be helpful if the authors could provide further information on the backgrounds of the annotators, the diversity within the team, the mechanisms for resolving any conflicts that arise during annotation, the duration of the annotation process, the versions of Python used, and approaches to managing incomplete figures. These details are crucial for assessing potential biases within the process.  The authors could include this information in an appendix or supplementary material if space is limited in the main paper.


Secondly, the manual complexity assessment of charts mentioned in section 2.3 raises questions about the classification criteria.
‘
Additionally, we conduct a manual complexity assessment for each chart, classifying them into 3 levels: easy, medium, and hard.
’
How are charts categorized into easy, medium, and hard levels? What criteria are used, and can the authors provide examples of each level to enhance the transparency of this process? If the classification relies on the subjective judgment of the annotators, how is bias controlled? If objective indicators such as the number of elements is used, more detailed information would be valuable. I suggest authors include a table or figure showing examples of charts at each complexity level, along with the specific criteria used for classification.


Additionally, the use of GPT-4O scores in the evaluation metrics introduces concerns about the inherent instability of LLMs. It would be pertinent for the authors to explain how they address these variabilities to ensure reliable evaluation outcomes. The authors could consider conducting a stability analysis of this metric by executing the evaluation repeatedly and documenting the variability in the resultant scores.

Do the authors plan to make all the data in the benchmark publicly available after publication? I recommend that during the review stage, the authors could use anonymous GitHub repositories or similar platforms to offer more detailed information about the benchmark, including data and usage tutorials. This would significantly enhance the thoroughness of the benchmark evaluation.

Lastly, I am curious if the authors have considered expanding this benchmark into a practical tool for figure-to-code tasks, possibly incorporating methodologies like Retrieval-Augmented Generation, to further its applicability and utility.

strengths:
Innovative benchmark;
the paper is good-writting;
Good motivation.

weaknesses:
Insufficient Detail on Annotation Process
Lack of Transparency in Complexity Assessment

questions:
Can you provide more details about the manual annotation process, including annotator backgrounds, diversity, conflict resolution, and handling of incomplete figures?
What specific criteria are used to classify charts into easy, medium, and hard categories? Are these criteria subjective, or are they based on objective indicators?
How do you address the instability of LLM outputs in your evaluation metrics to ensure consistent and reliable results?
Have you considered developing this benchmark into a tool for practical figure-to-code applications, possibly integrating methodologies like RAG?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660014164

thread_type:
official_review

<thread_object>
creator: reviewer_xblj

<content>
summary:
This paper introduces a new dataset ChartMimic, which tests LMM's ability to replicate a variety of different plots based on a set of carefully constructed set of examples and metrics. This paper evaluates multiple LMMs on ChartMimic including both proprietary and open models, finding that most models with the exception of GPT-4o perform quite poorly on the tasks in ChatMimic but also benefit from additional natural language information for customization tasks. In terms of prompting, the authors find that self-reflection is effective.

strengths:
- Overall the writing is easy to follow and the paper is well-organized.
- The work provides a clear dataset contribution that is not present in prior work as they overview in Table 2.
- The evaluation is generally well done, covering many different models, prompting strategies, incorporating a comparison to human judgments, and an error analysis of the best-performing model.

weaknesses:
While already did a good job with the presentation of dataset construction and evaluation, here are a few more suggestions that could further improve clarity:
- Provide more examples of both direct and customized mimic tasks. In particular, I am curious how well-defined are instructions for customized mimic tasks are.
- Provide clearer definitions of easy, medium, and hard tasks. This kind of naming obfuscates the actual nature of the tasks themselves. 
- Provide examples of GPT-4o scoring on different plots for sanity checks

Another suggestion is to conduct the same error analysis on open models, are there similar trends?

questions:
Please address the specific questions raised in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693508063

thread_type:
official_review

<thread_object>
creator: reviewer_lhtp

<content>
summary:
The paper proposes a new benchmark, ChartMimic, consisting of (figure, instruction, Python code) triplets covering 4,800 charts that are manually annotated by  the authors after being extracted from arxiv papers (along with other manually-curated sources). The authors propose five new automated metrics for evaluating proposed generations. The authors conduct an empirical study of existing models on the benchmark, demonstrating (among other findings) that existing SOTA GPT-4o attains a score of ~82.2%; that open-source models lag behind proprietary models; etc.

strengths:
# Overall assessment

This is a relevant benchmark for a task likely of high interest and impact for both AI developers and AI users alike (chart generation). In general, the benchmark construction appears reasonable, but at times it is difficult to understand the details of the process and the authors do not provide quantitative details on the highly manual filtering process (these details seem important to assess the filtering procedure, since it relies extensively on manual curation by the authors). The empirical study is useful and shows some interesting, if unsurprising, results.

# Major comments

* In many cases, it would not be possible to fully reconstruct the original plot, as even at the pixel-level it would not be possible to fully recover exact data values. How is this handled in the "direct mimic" metric?

* Does "customized mimic" always *only* involve changing the data of a plot? If so, please state this explicitly. Furthermore, I would suggest changing the name to something more specific and clear (to me, this is really "direct mimic with new data").

* Related to the above point, it seems that there are two, completely independent, tasks being performed here. One is generating code to create a plot that exactly matches the aesthetic style of a given image. The other is recovering the *data* to be plotted, and applying the generated code to this data. I think it would be useful for the authors to also discuss this distinction in the paper, and to clearly motivate in which cases recovering the data is actually useful (in many cases, it seems we can simply take the data as given, and we would relaly just like to evaluate the correctness of the plotting code irrespective of the data it is applied to).

* In the filtering stage, the authors state that in the first phase "Charts with significant differences in complexity or information density are added to ensure diversity and effective communication". How exactly is this process conducted, given that the pool consists of "about 15,800 figures" prior to the first phase?

* There are many places in the dataset description where more detail would be helpful; for example "We preserve the intersection of their selections and finalize the set through a voting process." What voting process is used? How many charts were filtered at this stage? In general, it would be helpful to know precisely how many data points were considered, and how many filtered out, at each stage of the pipeline. Please add this data to the paper.

* Did the authors assess the impact of few-shot learning, chain-of-though prompting, or other methods on the benchmark tasks?

# Minor comments

* The abstract says the benchmark is comprised of "(figure, instruction, code)" triplets; but isn't data also a necessary component of the chart?

* The abstract gives an "average score" but it isn't at all clear what this score represents. Given that the authors propse a set of new metrics in the benchmark, perhaps they could more clearly describe the composite score being reported.

* Please include hyperlinks to relevant sections when referring to them (e.g. in L88-91).

* Figure 2 isn't particularly illuminating; in fact I find it quite confusing. I would suggest to modify the layout substantially and add clear individual labels to the different boxes.

# Typos etc.

* L40: ", such real-life scenarios have yet to be fully explored." This should be a separate sentence.
* L53: "Based on this, we define two tasks, Direct Mimic and Customized Mimic (Sec. 2.1), which utilizes" --> which utilize
* L141: "clarity and visual appeals" --> clarity and visual appeal

weaknesses:
see above

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732234702090

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Manuscript and Response to All Reviewers

comment:
We sincerely thank all reviewers for their dedicated efforts during the review process and their valuable suggestions for improving our work. We have marked all revisions in **blue** in the updated manuscript. Below are the key revisions:

1. Abstract: Added clarification "... across Direct Mimic and Customized Mimic tasks..." as suggested by reviewer LHtP (lines 26-27)

2. Fixed grammar errors in lines 40, 53, and 142 following reviewer LHtP's comments

3. Added hyperlinks to relevant sections for better navigation as recommended by reviewer LHtP (lines 88-91)

4. Figure 2: Added labels for Direct Mimic and Customized Mimic in the Results & Evaluation part based on reviewer LHtP's suggestion

5. Provided clearer definitions of easy, medium, and hard tasks following reviewers XbLj and XRf9's comments (lines 197-199 and Appendix A.3)

6. Enhanced the data annotation pipeline description in Appendix A.2 as requested by reviewers LHtP and XRf9

7. Added task triplet examples in Appendix A.4 following reviewers LHtP and XbLj's suggestions 

8. Included GPT-4o scoring examples in Appendix D.1 as suggested by reviewer XbLj

9. Added error analysis of open-weight models in Appendix I based on reviewer XbLj's feedback

10. Provided performance correlation with chart understanding and code generation benchmarks in Appendix G as suggested by reviewer gbor

We greatly appreciate the reviewers' constructive feedback and their recognition of our contribution.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces ChartMimic, a new benchmark for evaluating large multimodal models (LMMs) on visually grounded code generation tasks. The reviewers are generally positive about the paper and argue for the acceptance. Specifically, the reviewers think the benchmark is novel and relevant, and agree that ChartMimic addresses an important and practical task relevant to both AI developers and users, and fills a gap in existing LMM evaluation datasets. The reviewers also commended the authors for conducting comprehensive evaluation, by proposing a multi-level evaluation metric that provides a holistic view of model performance. The reviewers also appreciated the thorough evaluation and analysis. The reviewers raised some concerns about clarity, which was sufficiently addressed during rebuttal.