Distill Visual Chart Reasoning Ability from LLMs to MLLMs

Authors: Wei He, Zhiheng Xi, Wanxu Zhao, Xiaoran Fan, Yiwen Ding, Zifei Shan, Tao Gui, Qi Zhang, Xuanjing Huang
Primary area: generative models
Keywords: multimodal large language models, synthetic data, chart question answering

Abstract:
Solving complex chart Q&A tasks requires advanced visual reasoning abilities in multimodal large language models (MLLMs). Recent studies highlight that these abilities consist of two main parts: recognizing key information from visual inputs and conducting reasoning over it. Thus, a promising approach to enhance MLLMs is to construct relevant training data focusing on the two aspects. However, collecting and annotating complex charts and questions is costly and time-consuming, and ensuring the quality of annotated answers remains a challenge. In this paper, we propose Code-as-Intermediary Translation (CIT), a cost-effective, efficient and easily scalable data synthesis method for distilling visual reasoning abilities from LLMs to MLLMs. The code serves as an intermediary that translates visual chart representations into textual representations, enabling LLMs to understand cross-modal information. Specifically, we employ text-based synthesizing techniques to construct chart-plotting code and produce ReachQA, a dataset containing 3k reasoning-intensive charts and 20k Q&A pairs to enhance both recognition and reasoning abilities. Experiments show that when fine-tuned with our data, models not only perform well on chart-related benchmarks, but also demonstrate improved multimodal reasoning abilities on general mathematical benchmarks such as MathVista.

Decision: reject
Reviewer scores: 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730543552408

thread_type:
official_review

<thread_object>
creator: reviewer_yw4l

<content>
summary:
This paper proposes an efficient method (CIT) that aims to use code as a bridge and incorporates certain characteristics of diagram research (such as visual complexity) to generate data (REACHQA). On this basis, the paper deconstructs the capabilities required for the Chart QA task into recognition and reasoning, and demonstrates through experiments that the proposed method is effective.

strengths:
1. CIT incorporates the characteristics of the chart tasks to generate high-quality data.
2. This paper attempts to deconstruct the capability of chart-based question answering into recognition and reasoning and provides thorough justification based on this approach.

weaknesses:
1. The authors should consider and attempt to provide data on the overlap between the generated data, the validation set, and relevant benchmark datasets to further substantiate the validity of the experiments.
2. I hope the authors can provide more evaluation results on benchmarks that are specific to recognition [1] and reasoning [2] to demonstrate that the method has strong generalization capabilities. If the results do not decline, that would be an even more interesting observation.

[1] OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
[2] We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?

questions:
Please refer to the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730594312288

thread_type:
official_review

<thread_object>
creator: reviewer_4der

<content>
summary:
This paper purpose CiT (Code-as-Intermediary Translation), a data synthesis method for distilling visual reaoning abilities from LLM to MLLMs. In CiT, code are rendered to charts and translated to instruction tuning by LLMs. Using this pipeline, this paper developed ReachQA, which is an effective benchmark and training dataset.

strengths:
- The overall writing is clear and easy to follow.
- The idea of CIT is straightforward, effective and easily scalable.
- CIT addressed an important gap in MLLM training in lacking accurate textual annotations of visual diagrams.
- The resulting dataset has high quality. Althought only with 3K images and 20K QAs, the perforamnce of MLLMs improves by at most 35 percent. In addition, when trained a mixture with general multimodal dataset, the model can effectively retain its general MM benchmarks.

weaknesses:
- As authors claimed, the MLLMss ability consist of two main parts, 1. recognizing key information from visual inputs 2. conducting reasoning over it. The design of ReachQA is rich in both parts, so it's unclear which part improves models' over performance the most, the reviewer is aware that similar analysis is conducted in section 5, but an error analysis (similar with figure 1) on before/after ReachQA training can make it more clear. 
- Dataset volume is a concern. As a training dataset, only 3K images is not sufficient, especially for cross-modal training. So it's likely that the improvement gained by training on ReachQA is that the model learns reasoning better rather than recogonition.

questions:
Please refer to above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692504802

thread_type:
official_review

<thread_object>
creator: reviewer_chni

<content>
summary:
This paper aims to distill the visual chart reasoning ability from LLMs to MLLMs through a code-as-intermediary translation (CIT) method. The motivation behind the work lines the concern of current MLLMs and then distill the LLMs knowledge about reasoning to MLLMs is a way. The method separates two steps, recognition and reasoning, then the authors try to generate more codes for chart generation, and then generate instruction-answer pairs. Besides, the authors also propose a ReachQA dataset for training and evaluation. Through multiple experiments, the authors demonstrate the effectiveness of the method, the dataset with strong performances.

strengths:
1. The inspiration of the method is interesting and leverages the translation principle to incorporate an intermediate language. By using code as an intermediate language, the method is interesting and novel. 
2. The steps behind the method are clear and reasonable, first use code to generate charts, and then generate QA pairs. The authors use multiple methods, such as Evol-instruct, self-instruct, llm-as-judge and so on to make the generated data to be high quality and diversity.
3. The experiments are extensive to show the effectiveness of the method and the study is also good.

weaknesses:
1. The main concern is about the motivation or the story. While the authors mention that the existing MLLMs are struggle in the recognition and reasoning abilities, the authors then want to use distill LLMs' reasoning ability to MLLMs. However, it is clear that Claude and GPT-4o are smart in these ways. Therefore, the problem is not about existing MLLMs but the open-sourced, or freely sourced MLLMs. I encourage the authors to rephrase the story, which then could introduce the method more clearly. 
2. Though the steps are great and the authors use multiple existing methods to generate and guarantee the data quality. The novelty of the method/framework is clearly not that big, but I like the code-as-intermediate translation method. 
3. In detail, the analysis of the question types should be reported, which could show the diversity and quality; from the experiments, strong improvements are achieved but still largely fail behind the close models; the self-repair method is good but the prompt is too simple, I am curious about other clear prompts; as the authors show figure 1 as a motivation, it is still necessary to better analyse the two type errors in the final trained model, not like figure 3, but like figure 1 to show the improvement.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731429107554

thread_type:
official_review

<thread_object>
creator: reviewer_u7ub

<content>
summary:
This paper proposes a novel pipeline for constructing multi-modality chart datasets. The central idea is the introduction of the “Code-as-Intermediary Translation” concept, which facilitates the creation of chart datasets for MLLMs (Multimodal Large Language Models). Experimental results demonstrate that the proposed datasets not only enhance chart-related QA capabilities but also improve overall reasoning performance.

strengths:
Utilizing code as a generative engine is a sound approach to creating new chart images.

Experimental results indicate that the new dataset significantly enhances specific performance metrics.

weaknesses:
The use of code as a tool for rendering the chart dataset appears similar to the concept presented in [1]. Furthermore, data augmentation through code is reminiscent of [2]. Could you elaborate on how your approach differentiates from these works?

The model fine-tunes general MLLM models. How would the performance change if an equivalent amount of chart data were selected from general datasets for fine-tuning the MLLMs? A comparative analysis might be insightful.

Why is the chart type described as ∞? Are there any specific chart types that cannot be generated using Python code? 

What is the accuracy or success rate of MLLMs when generating code?

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732441879023

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response from Authors.

comment:
Dear Reviewers,

We are very grateful for your recognition of our work (e.g., the **novelty** of our idea, the **effectiveness** of our method and dataset, and the **completeness** of our experiments), which is a great encouragement to us. 

We also appreciate the constructive suggestions, and have made updates to our paper based on your feedback. **All revisions are highlighted in red for your ease of reference**. In this overall response, we provide a summary of the key actions we have undertaken.

---

## Clarifications

1. **The comparison between our proposed CIT and other data synthesis methods, as well as our innovations and contributions.** (Reviewer U7uB & cHNi)
   - Our approach builds upon well-established practices in data synthesis, such as leveraging code for rendering charts and augmenting seed data. While these foundational aspects are shared with prior work, our key innovation lies in introducing the **“code-as-intermediary”** concept, which bridges visual and textual modalities to tackle unique challenges in chart-related tasks. By enabling the seamless application of existing text-based data synthesis techniques within a chart-specific context, our CIT method addresses the current scarcity of visually complex and reasoning-intensive chart Q&A datasets.
2. **The broader applicability of our work beyond open-source MLLMs.** (Reviewer cHNi)
   - While our experiments primarily target open-sourced MLLMs due to accessibility constraints, our evaluation results, along with recent studies like IsoBench, suggest that even advanced proprietary models face challenges in reasoning-intensive multimodal tasks. By leveraging our constructed dataset, we achieve consistent improvements across various base models. This highlights the potential of our approach to enhance the reasoning abilities of open-source or even proprietary MLLMs through effective cross-modal knowledge distillation.
3. **Dataset efficiency and potential for scaling.** (Reviewer 4dEr)
   - Although our current dataset contains only 3k images, our approach effectively leverages the learning potential through carefully designed samples, where each visually complex chart is paired with multiple diverse questions covering different aspects of understanding. Moreover, the automated synthesis pipeline is inherently scalable, making it straightforward to expand the dataset size when needed.

## Supplementary Experiments

1. **Adding a baseline using general (mixed) chart datasets for fine-tuning.** (Reviewer U7uB)
   - We conducted additional experiments using 20k chart-related samples from The Cauldron, a large-scale collection of 50 vision-language datasets chosen for its generality. The results, presented in **Table 5** of the revised paper, reveal that training with this general mixed dataset slightly outperforms using individual datasets. However, it still falls short of the performance achieved by fine-tuning with high-quality synthetic datasets such as our ReachQA.
2. **Adding experiments on two new benchmarks.** (Reviewer Yw4L)
   - We conducted additional evaluations on the OCRBench and We-Math benchmarks to further investigate the generalization of our method. The results, presented in **Figure 4** of the revised paper, align with the trends previously discussed. Specifically, the findings indicate that training on ReachQA enhances reasoning capabilities and incorporating general data yields a well-balanced improvement.
3. **Adding analysis of error types and model improvement.** (Reviewer cHNi & 4dEr)
   - We conducted additional error analyses before and after training with ReachQA to better illustrate the improvements in models. The results are presented in **Appendix C** of the revised paper. This analysis reveals that both recognition and reasoning errors decrease significantly across different models, confirming enhancements in related abilities.
4. **Adding analysis of question types.** (Reviewer cHNi)
   - To demonstrate the diversity and quality of our generated questions, we categorized them into several types using GPT-4o. Detailed statistics are provided in **Appendix D** of the revised paper.
5. **Adding analysis of data contamination.** (Reviewer Yw4L)
   - We conducted an in-depth analysis to address concerns about data overlap. Our comparison included the ReachQA train set against other chart-related benchmarks and its own test set. The results, detailed in **Appendix E** of the revised paper, show our dataset's distinct characteristics from existing benchmarks. To mitigate concerns about data leakage in the test set, we manually reviewed the 50 most similar chart pairs, ensuring no further data contamination was present.

---

Thank you for taking the time to read our response. We hope this addresses your concerns and welcome any further questions or feedback. If our revisions meet your expectations, we kindly ask for consideration in adjusting your scores accordingly.

Best regards,

Authors of Submission 5783
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539431599

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Friendly Reminder from Authors.

comment:
Dear Reviewers,

As the Author-Reviewer Discussion phase is drawing to a close, we are still here and welcome any further discussion or feedback you may have on our paper. We sincerely look forward to your feedback. Your insights are incredibly valuable to us.

Thank you sincerely for all the time and effort during the review process.

Best regards,

Authors of Submission 5783
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary:
The paper introduces Code-as-Intermediary Translation (CIT), a scalable method for synthesizing chart datasets to enhance reasoning abilities in Multimodal Large Language Models (MLLMs). It contributes the ReachQA dataset, demonstrating improved performance on chart-related tasks and general reasoning benchmarks.

(b) Strengths:
The paper addresses an important gap in MLLM training, proposing a novel and efficient data synthesis pipeline. Extensive experiments show measurable performance improvements, and the dataset exhibits high quality despite its small size.

(c) Weaknesses:
The novelty of the CIT framework is incremental, as it builds on existing techniques. Concerns include limited dataset volume, lack of sufficient qualitative examples, and questions regarding scalability with cheaper models.

(d) Decision:
While the paper is well-executed with valuable contributions, concerns about limited novelty and dataset sufficiency remain. I recommend rejection.