R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models

Authors: linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, Yingying Zhu, Xiang Bai
Primary area: foundation or frontier models, including LLMs
Keywords: Large Multimodal Models, Mathematical Reasoning, Geometric Reasoning

Abstract:
Existing Large Multimodal Models (LMMs) struggle with mathematical geometric reasoning due to a lack of high-quality image-text paired data. Current geometric data generation approaches, which apply preset templates to generate geometric data or use Large Language Models (LLMs) to rephrase questions and answers (Q&A), unavoidably limit data accuracy and diversity. To synthesize higher-quality data, we propose a two-stage Reverse Chain-of-Thought (R-CoT) geometry problem generation pipeline. First, we introduce GeoChain to produce high-fidelity geometric images and corresponding descriptions highlighting relations among geometric elements. We then design a Reverse A&Q method that reasons step-by-step based on the descriptions and generates questions in reverse from the reasoning results. Experiments demonstrate that the proposed method brings significant and consistent improvements on multiple LMM baselines, achieving new performance records in the 2B, 7B, and 8B settings. Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730569293982

thread_type:
official_review

<thread_object>
creator: reviewer_d7jj

<content>
summary:
The paper introduces R-CoT, a two-stage geometry problem generation pipeline aimed at enhancing geometric reasoning in large multimodal models. The approach involves generating high-fidelity geometric images and associated descriptions using GeoChain and subsequently creating questions through a reverse process using an LLM-based Reverse A&Q method. The authors claim substantial performance gains on MathVista and GeoQA benchmarks, with R-CoT-8B surpassing state-of-the-art performance on open-source and even closed-source baselines.

strengths:
1. The R-CoT approach is inventive in using a reverse process for generating questions.
2. The GeoChain component improves upon traditional synthetic data generation methods by increasing visual realism and complexity in the generated geometric problems.

weaknesses:
1.	**Unclear Necessity of the Vision Modality**:
The integration of a vision component within this framework raises concerns regarding its necessity and effectiveness. The core of the geometric problem-solving process, as presented, appears to rely on CoT reasoning based solely on textual descriptions. Given that the problems are fully described in the text, the role of the generated geometric images in augmenting reasoning is ambiguous. Specifically, the added visual modality does not clearly offer unique insights or support that could not otherwise be inferred through text alone. Without a compelling demonstration of how the visual component uniquely facilitates problem-solving, its inclusion seems unnecessary and potentially redundant.

2.	**Ambiguity in Difficulty Level Classification**:
The criteria used for categorizing the problems into three difficulty levels—easy, medium, and hard—are insufficiently defined and appear somewhat arbitrary. For instance:

- *Inconsistent Alignment with Difficulty Criteria*: The difficulty classification seems to focus on the number of reasoning steps, but without concrete justifications, this metric alone may not adequately capture the true complexity. The nature of geometric reasoning can vary widely, and simply having more reasoning steps does not necessarily equate to a harder problem. 

- *Spatial Reasoning as a Key Factor*: For certain problems that require interpreting spatial relationships or auxiliary constructs (e.g., adding segments for further reasoning), a visual component might be essential. These types of problems could then reasonably fall under higher difficulty levels if they require the vision modality to infer spatial arrangements. However, the paper does not fully clarify when and why visual input would be required in these cases.

Additionally, according to [Ref. 1], those data correctly generated by CoT of LLMs can be considered within a Completely Feasible Reasoning Boundary (CFRB), which characterizes the simple reasoning tasks. Therefore, the difficulty criteria should be re-evaluated to better suit the geometry task, accounting for situations where visual input may or may not be essential.

[Ref. 1] Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought. NeurIPS 2024.

questions:
1. Can you provide more analysis on the role of the vision part generated by GeoChain in solving the geometric problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626855173

thread_type:
official_review

<thread_object>
creator: reviewer_tf8u

<content>
summary:
The paper proposes Reverse Chain-of-Thought (R-CoT), a method to improve geometric reasoning in Large Multimodal Models by generating high-quality image-text data through a two-stage process. First, the GeoChain stage creates precise geometric images with descriptive relationships between elements. Then, the Reverse Answer & Question stage generates questions in reverse, based on step-by-step reasoning from these descriptions. The results show improved model performance on benchmarks like MathVista and GeoQA.

strengths:
1.	The R-CoT pipeline introduces a structured way to generate geometric problems by reversing the usual chain-of-thought, which enhances data accuracy and diversity. This approach aligns well with complex reasoning tasks in geometric settings.
2.	R-CoT achieves improvements in the 2B, 7B, and 8B model settings, outperforming existing open-source and closed-source models, such as GPT-4o, particularly on benchmarks like MathVista and GeoQA.

weaknesses:
1.	The reverse chain-of-thought concept is essentially a reconfiguration of existing Chain-of-Thought (CoT) techniques applied to geometric data generation. The novelty is thus limited, as R-CoT primarily adapts and combines established ideas without introducing a fundamentally new approach. 
2.	The effectiveness of R-CoT heavily relies on the fidelity of the generated images and descriptions. However, the process for validating the quality and accuracy of these generated descriptions seems insufficiently detailed, especially given the complexity of geometric reasoning.
3.	The R-CoT pipeline involves a multi-stage generation and reasoning process, which could increase computational cost. It is unclear whether this approach offers efficiency or if the computational demands outweigh its benefits.

questions:
1.	In Section 3.1, the paper states that GeoChain generates "high-fidelity" images. What criteria or metrics were used to define and verify this fidelity?
2.	The description generation for the images lacks detailed criteria for ensuring the accuracy. What specific metrics or validation methods were employed to assess the fidelity of these descriptions? In cases where the generated image descriptions were inaccurate or lacked detail, how were these issues addressed?
3.	How does line sampling affect image quality and the diversity of generated images? Could the authors explain the effect of varying the number of sampling rounds?
4.	Could the authors provide an analysis of common error types in generated Q&A pairs and any steps taken to minimize them?
5.	In Section 3.2, can the authors clarify how “single-step reasoning results” are progressively fused in Chain-of-Thought Fusion? Could examples or intermediate reasoning steps be provided to clarify this fusion process?
6.	Given the two-stage process of GeoChain and Reverse A&Q, can the authors provide an analysis of the computational cost and time efficiency of R-CoT? 
7.	Are there any specific geometric problem types where R-CoT struggles?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788338824

thread_type:
official_review

<thread_object>
creator: reviewer_bf5d

<content>
summary:
The authors present a two-stage Reverse Chain-of-Thought (R-CoT) pipeline to enhance mathematical geometric reasoning in large multimodal models (LMMs). First, they introduce GeoChain, which generates geometric images and descriptions that highlight relationships between geometric elements. This is followed by a Reverse A&Q method, which uses step-by-step reasoning based on these descriptions to generate questions in reverse from the reasoning outcomes.

strengths:
1. The multimodal mathematics direction is challenging, and dataset construction for geometric problems is meaningful. 
2. The paper is clearly written and easy to follow.

weaknesses:
1. Overclaim: The statement, "Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets," appears overclaim. Given that this work primarily focuses on data generation, fair assessment would require consistent model comparison settings. The claim in the abstract is misleading due to differences in model size and backbone. For instance, the authors use an 8B model as the backbone, whereas prior methods rely on a 7B model. As shown in Table 1, improvements under comparable settings are marginal.
2. Concern of Entire Data Quality: In Table 1, combining your data with Geo170K yields only marginal improvements, which raises questions about the efficacy of the GeoMM dataset. Can the authors provide results that reflect the performance of using only GeoMM?
3. Image Diversity: Clarification is needed on how images generated by GeoChain differ from those in existing datasets. For example, the images shown in Figure 12 (Examples of GeoMM dataset) appear similar to datasets like GeoQA and Geo3K. The marginal improvement observed further suggests potential alignment or overlap with existing data domains, calling into question the added diversity. Moreover, how is diversity measured?
4. Concern of QA Quality: The synthetic questions are annotated using ERNIE Bot 4.0. After generating the QA pairs, what steps were taken to ensure their correctness, and has this been verified?
5. Question about Method: When segmenting descriptions into patches for reasoning, it can be challenging to assess the difficulty level of each condition in relation to others. For any given description, how is the hierarchy of condition difficulty determined? Additionally, how do you ensure that each subsequent condition is logically derived from previous ones to yield varied difficulty levels in the answers?
6. Writing: In Table 4, it’s noted that the baseline models vary across R-CoT models of different sizes, yet comparisons mention only the model sizes, which creates an unfair comparison. Stating specific settings for both your models and baseline models (LLMs/VLMs) in the table is necessary to give readers an accurate understanding of performance.

questions:
Please refer to Weaknesses
</content>
</thread_object>

</thread>


Metareview: