Reason-and-Execute Prompting: Enhancing MultiModal Large Language Models for Solving Geometry Questions

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:20:26

Abstract: MultiModal Large Language Models (MM-LLMs)  have demonstrated exceptional reasoning abilities in various visual question-answering tasks. However, they encounter significant challenges when answering geometry questions. These challenges arise due to the need to engage in rigorous reasoning and executing precise arithmetic. To enhance the ability of LLMs to solve multimodal geometric questions, we propose Reason-and-Execute (RaE) prompting: a new prompting method specifically designed for enhancing MM-LLMs to solve geometric questions. Specifically, we first designed a rigorous reasoning process based on domain knowledge of geometry, using a reverse thinking approach, and obtained the precise arithmetic steps required for solving the question. Secondly, based on the analysis of the reasoning process, we designed code blocks in a programming language to implement the arithmetic functions. Finally, by executing the contents of the code blocks using an interpreter, we obtained the answers to the geometric questions. We evaluated the accuracy of 9 models in answering questions on 6 datasets (including four geometry datasets and two science datasets) using different prompting templates. Specifically, in the main experimental result, our RaE showed a maximum enhancement of 12.8% compared to other prompting methods, which proves strong reasoning and arithmetic abilities in solving geometric questions of our method. Moreover, we analyzed the impact of answering from the perspective of solving geometric problems by considering multiple factors, including domain knowledge, geometry shapes, understanding of the question text, and language. This once again emphasizes that our method has passed the comprehensive test of solving geometry questions. The source code and data will be published in a GitHub repository.

Main paper: 
 introduction: 
 Figure 2 : Error answer analysis of 700 geometry questions with GPT-4V (gpt-4-vision-preview ).Traditional methods for solving geometry questions typically focus on mathematical expression [28] , while the MultiModal Large Language Models (MM-LLMs) can better understand the relationship between natural language descriptions and geometry shapes [10] , as shown in Figure 1 . Enhancing the efficiency and accuracy of MM-LLM in solving geometry questions is of great significance for the development of education and intelligent aided systems [18] .The MM-LLMs have demonstrated exceptional reasoning abilities in various visual question-answering tasks [42] . However, there are obstacles in the task of automatically solving geometry questions [22, 28] , as shown in Figure 2 . The main obstacles include errors in arithmetic results (36%) and errors in logical reasoning processes (21%). At present, there are mainly two methods to overcome these obstacles. One is a fine-tuning method [34, 36, 44] based on specific downstream tasks. Although some MM LLMs are already open source and can be fine-tuned based on pre-trained models [2, 21, 35, 40] , they have extremely high requirements for datasets and devices, making it difficult to perform task fine-tuning. Especially in the task of solving geometry questions, firstly, the scale of publicly available high-quality geometry datasets is limited; Secondly, MM-LLMs typically have billions to tens of billions of parameters, which makes the fine-tuning process require a significant amount of computing resources and time. Compared to this method, another method that uses some examples as prompts [1, 17, 43] to solve new questions is easier to implement and achieves impressive results. Among them, the most representative works are Chain-of-Thought Prompt (CoT) [38] method and Program-Aided Language (PAL) [35] Models. While these methods have demonstrated remarkable performance in various tasks, such as the CoT and PAL prompt methods achieving accuracy rates of 94.7% and 99.2% in mathematical natural language reasoning tasks [35] , solving geometry questions remains a significant challenge. For example, Figure 3 (a) illustrates that the CoT prompting method, when applied to geometry questions, often misuses data in the reasoning process due to the complexity of domain knowledge, leading to incorrect answers. Similarly, Figure 3 (b) shows that the code block generated by the PAL prompting method contains excessive reasoning processes, rendering the program non-executable and unable to provide the answer. Thus, designing a prompting method that can facilitate rigorous reasoning and precise arithmetic for solving geometry questions remains a huge challenge.To address these challenges, we propose Reason-and-Execute (RaE) prompting: a new prompting method specifically designed for enhancing MM-LLMs to solve geometry questions. Specifically, we first designed a rigorous reasoning process based on domain knowledge of geometry, using reverse thinking [9] approach, and obtained the precise arithmetic steps required for solving the question. Secondly, based on the analysis of the reasoning process, we designed code blocks in a programming language to implement precise arithmetic functions. Finally, by executing the contents of the code blocks using an interpreter, we obtained the answers to the geometric questions.In the analysis of experimental results, we have demonstrated the ability of the RaE prompting method to perform rigorous reasoning and precise arithmetic operations. In addition, we also analyzed the impact of domain knowledge, geometry shapes, understanding of the question text, and language on our prompt templates for solving geometry questions. We conclude that rigorous reasoning and precise arithmetic processes are essential for accurately solving geometry questions. The contributions of this paper are as follows:• We propose Reason-and-Execute (RaE) prompting: the first prompting method specifically designed for enhancing MM-LLMs to solve geometric questions.  
related works 2.1 multimodal large language models: 
 With the multimodal large language models (MM-LLMs) showing a strong ability of image-text understanding [42] , the research of math reasoning using MM-LLM combined with images and texts began to appear [22, 24, 25] . Especially in the study of geometric question solutions that often appear in multimodal forms [28] ,there has been greater vitality [22] . Specifically, solving geometry questions requires a combination of image and text information to complete professional domain-knowledge reasoning and precise arithmetic operations. Although this is a huge challenge for LLMs [45] , the MM-LLMs can fully leverage its advantages [30] . For example, GPT-4(Vision) [29] uses a visual encoder with pre-trained components for visual perception, aligning the encoded visual features with the language model, thereby achieving a comprehensive understanding of geometric problem images and text information; Qwen VL [2] is a large-scale visual language model launched by Alibaba [41] Cloud that performs well in tasks such as image description, question answering, visual positioning, and flexible interaction, moreover the baseline model used in our experiment is its two important models: qwen-vl-chat and qwen-vl-plus; CogVLM [35] puts visual understanding as a higher priority to achieving the deep fusion of visual language features; mPLUG Owl [39] can learn the parameters of the visual encoder in the first stage of training, to achieve efficient image alignment with this article; InternLM-XComposer2 [8] proposes a new fine-tuning method of visual and text alignment, which enhances the visual understanding ability of the model; Yi Vision Language (Yi-VL) [40] demonstrates its strong capabilities in complex interdisciplinary tasks with its excellent ability to understand images and generate dialogue. DeepSeek-VL [21] is an innovative open-source visual language model that stands out for its ability to understand real-world scenarios in various applications such as logic diagrams, web pages, and natural images; Gemini [18] is a large language model released by Google [3] , and designed specifically for "general-purpose tasks", Gemini Pro, has performed well in various multimodal processing fields. They have the potential to solve geometry questions by combining images and text.LLMs have achieved tremendous success with the support of computing power and datasets. Computational power enables the model to be sufficiently large, while also possessing excellent comprehension, memory, reasoning, and generative abilities [36] . The dataset provides a learning foundation for the model. LLMs typically adopt a generative Transformer architecture [44] , and in the fine-tuning stage [34] , through the prompting method [1, 17, 43] , the model can be fine-tuned according to task requirements to make it more suitable for specific tasks and scenarios. The Chain of Thought (CoT) [38] method is a foundational approach in prompting, which involves appending multiple reasoning steps before providing the answer to a question. This simple few-shot prompting strategy [4] has significantly enhanced the performance of Large Language Models (LLMs) in complex reasoning tasks [20] . Few-shot prompting [4] is effective across various tasks and has notably improved mathematical reasoning tasks. Extensions of CoT [11, 31, 32, 37] have further expanded the range of reasoning tasks that LLMs can tackle, improving their performance on various benchmarks. However, previous approaches have struggled with accuracy in arithmetic calculations and reasoning errors [12, 19, 26, 27] . To address complex calculations and reasoning, advanced prompt strategies like Program of Thought [7] , Program-Aided Language (PAL) [35] , MathPrompter [13] , Least-to-Most Prompting [46] , and Plan-and-Solve [35] have been developed. Prompt engineering has evolved from static to dynamic strategies such as Active Prompt, RLPrompt, and PRewrite [16] . Additionally, LLMs have demonstrated effectiveness in providing automated error feedback, highlighting their utility in solving math word problems. Among numerous prompt methods, CoT and PAL are pioneering methods for reason and algorithm, respectively, and are also the most widely applicable methods. Therefore, our subsequent research will expand these two methods into prompt templates suitable for multimodal tasks as our baseline prompt methods. 
reason-and-execute prompting: 
 Overview. We introduce RaE prompting, a new prompting method specifically designed for MM-LLMs to solve geometric questions 𝑞, as shown in Figure 3 . It ensures MM-LLMs generate reasoning processes 𝑟 by utilizing domain knowledge, and generate executable code blocks 𝑏 to obtain answers. Compared with the few-shot CoT and PAL methods, the RaE prompting method, designed for more professional solving of geometric problems, features both rigorous reasoning processes and precise arithmetic operations. Specifically, RaE prompting leverages the advantages of MM-LLMs to solve tasks with a set of k examples, {(𝑞 𝑖 , 𝑟 𝑖 , 𝑏 𝑖 )} 𝑘 𝑖=1 ∥ 𝑞 𝑡𝑒𝑠𝑡 . Each example in the RaE prompt is a triplet of < 𝑞 𝑖 , 𝑟 𝑖 , 𝑏1 >, where 𝑞 𝑖 and 𝑏 𝑖 are input-output pairs, and 𝑟 𝑖 is an inference process that ensures the solvability of code block 𝑏 𝑖 . Note that the test question 𝑞 𝑡 𝑒𝑠𝑡 we input does not directly generate an answer, but is obtained through the execution of the generated code block 𝑏 𝑡 𝑒𝑠𝑡 operation by the interpreter.Overall, the RaE prompts involve two steps. In step 1, to solve geometric questions more professionally, we designed prompts for the inference process 𝑟 based on question-oriented thinking and domain knowledge, avoiding the interference of redundant information. In step 2, to obtain a precise arithmetic answer, a code block 𝑏 that can be executed by the interpreter is generated based To solve geometric questions more professionally, we adopted a Reverse thinking [9] approach, guiding MM-LLMs to start from the question being solved and gradually identify the known conditions necessary to solve the question, as shown in Figure 4 . The template constructed in this step needs to meet the following three conditions: Although the degree of ∠𝐶 here is not a known condition, it can be obtained through the triangle angle sum theorem that ∠𝐶 = 180 -∠𝐴 -∠𝐵. This example involves two domain knowledge: the property of complementary inner angles on the same side of parallel lines and the triangle angle sum theorem.Clarify available data The template needs to identify which data is needed to solve the question and obtain it from the graphic and textual content of the problem. For example, as shown in the figure, after analyzing domain knowledge, solving the problem requires the degrees of ∠𝐴=80 and ∠𝐵 = 60. Note that sometimes the required data is not in the text of the question and needs to be identified from the image of the question.Clarify the arithmetic process This template needs to integrate the domain knowledge and available data used in the question and clarify the operational process that needs to be transformed into program blocks. For example, as shown in this image, based on the analysis above, the process of performing precise arithmetic to answer this geometric question is: ∠𝐴 = 80, ∠𝐵 = 60, ∠𝐶 = 180 -∠𝐴 -∠𝐵, ∠𝐶𝐸𝐷 = 180 -∠𝐶.In summary, our reasoning prompting template is a process that starts from the question to obtain the required data and clarifies the need for precise arithmetic.It is easy to make mistakes when MM-LLMs rely solely on "memory" to obtain answers to the questions. To obtain precise answers to geometric questions, we use the idea of program-assisted problemsolving to guide MM-LLMs to understand the need for precise arithmetic from step 1 and generate executable code blocks, as shown in Figure 5 . Finally, the precise answer is obtained by running the code block through the interpreter. The template constructed in this step must meet two conditions: Meaningful variable naming This template requires meaningful variable names to ensure that the program block has high runtime quality. Meaningful variable naming can to some extent avoid the problem of invalid program syntax. This is also related to whether the parameters involved in the arithmetic process in step 1 are clear. For example, as shown in the figure, based on the required parameters in step 1, we have designed variable names: angle_A, angle_B, angle_C, angle_CED.Overall, to obtain an accurate answer to this geometric question, we must rely on the interpreter to smoothly execute the generated code block. 
experimental setup 4.1 benchmarks: 
 The proposed method is evaluated on the six benchmark datasets, as shown in Table 1 . Geometry question datasets: (1) the GEOS [33] dataset contains simple middle school geometry problems with geometric shapes, (2) the Geometry3K [23] dataset contains numerous geometry questions where semantic information is scarce and most values need to be obtained from images, (3) the GeoQA [6] dataset contains rich semantic information for middle and high school geometry questions, (4) the GeoQA+ [5] dataset is based on GeoQA, which adds more diverse types of geometry questions and forms an enhanced benchmark dataset. Other science question datasets: (1) the AI2D [14] dataset includes diagram questions for multiple natural science courses of the elementary school; (2) the TQA [15] dataset is drawn from middle school science curricula textbooks. The above datasets are all applicable and publicly available datasets for our work.Figure 6 illustrates the statistical distribution of question length in the six benchmark datasets. In the GEOS (a), GeoQA (c), GeoQA (d), AI2D (e), and TQA (f) datasets, the distribution aligns with expected patterns, with the majority of questions containing substantial textual content. Moreover, the textual content appears to reasonably correspond to the information depicted in the diagram. Conversely, in the Geometry3K dataset (b), approximately 18% of all questions contain 3 words or fewer. This indicates a lack of descriptive information from the text of the question and primarily provides specific queries, such as 'Find UT'.Table 1 : Details of datasets being evaluated. The "total" represents the question number of questions in an original dataset, the "sample" represents the number of questions randomly selected from a dataset in a test, "en" represents the questions in English, and the "zh" represents the questions in Chinese. Since geometry questions are mostly multimodal, we adopted seven multimodal large language models as the baseline: We evaluate the performance of various MM-LLMs on the six benchmarks, including both closed-source and open-source models. The closed-source models are evaluated by using their official API, while open-source models are evaluated by running inferences on 4-way RTX 4090GPU. For the closed-source models, we select state-of-the-art models GPT-4V (gpt-4-vision-preview), Gemini-Pro (gemini-pro-vision), and Qwen-VL-Plus (qwen-vl-plus).For the open-source models, model sizes vary from 6b to 7b, including CogVLM(cogvlm-7b), Qwen-VL-Chat (qwen-vl-chat-7b), Intern-XCompose (intern-xcomposer-7b), Yi-VL(yi-vl-6B), DeepSeek-VL(deep-seek-vl-7b), and MPLUG-Owl2(mplug-owl2-7b). The templates used in the experiment can be found in "Appendix A". In addition, We performed greedy decoding from the language model using a temperature of 0. Meanwhile, considering the real-world usage of the model, we simulated the use of different MM LLMs by users: randomly selecting the sample size shown in Table 1 from each dataset for testing. A total of 10 rounds were selected, and the average accuracy was taken as the accuracy of the final answer. 
main result: 
 The experimental results of applying various prompt methods to solve geometry and science questions in different MM-LLMs are shown in Table 2 . Due to the weak code generation ability of opensource models, we only used closed-source models when using PAL and RaE, while CoT and silent methods used all baseline models. Experimental results show that RaE outperforms all other prompting methods across the geometry datasets of GEOS (32.6%), Geometry3k (32.3%), GeoQA (31.1%), and GeoQA+ (29.3%). However, it performs less effectively than the COT method on the science datasets, A12D (79.6%) and the TQA (73.4%), indicating that our method significantly improves the accuracy of MM-LLMS in answering geometry questions. From Table 2 , we can also observe that the performance of PAL is relatively poor in these six datasets. This is because PAL generates code blocks instead of reasoning through natural language. However, when solving geometry questions, the "def solution()" generated code block contains reasoning steps, rendering the entire code block inoperable. Our method separates the reasoning of geometry questions from the code generation process and uses the generated rigorous reasoning process to guide MM-LLMs in generating executable code blocks, thereby achieving precise arithmetic. In summary, our proposed RaE is more suitable for solving geometric problems in closed-source MM-LLMs than other prompt methods and also enhances the ability of MM-LLMs to solve geometry questions.Solving geometry questions is a comprehensive test of the various abilities of the MM-LLMs, especially for our proposed RaE prompting method. For this, we have considered multiple factors from the perspective of problem-solving, including domain knowledge, geometry shapes, understanding of the question text, and language use. We tested the answer performance of GPT-4V models with RaE, PAL, and CoT prompting methods, and without prompting methods on these factors. Below are specific experimental analyses based on GPT-4V.Which is most important for RaE prompt templates, reasoning or executing? Our work proposes a prompting method called RaE, which mainly contributes to the design of a prompting template with rigorous reasoning and precise arithmetic parts. To explore the importance of these two parts for RaE prompts, we conducted ablation experiments using the same experimental setup as the Main Result. From the data in Table 3 , it can be seen that the rigorous reasoning part has a greater impact on the performance of RaE. Specifically, the lack of an inference section resulted in a maximum decrease of 7.6% in RaE's accuracy in solving geometric problems (on the Geometry3K dataset), while the maximum decrease without an execution section was only 4.2% (on the GEOS dataset). This indicates that using professional domain knowledge to guide MM LLMs in answering questions can fundamentally improve performance while using program solving can assist in improving limited performance. Does RaE work with Multi-domain knowledge? In the previous analysis, we found that rigorous reasoning is crucial in the RaE template. To achieve this rigorous reasoning, one needs to consider the domain knowledge required to solve the question. Therefore, we analyzed the domain knowledge involved in the GeoQA and GeoQA+ datasets and counted the number of questions. The detailed domain-knowledge statistics can be found in "Appendix B". In these statistics, we found that solving a large number of questions relies on mixed domain knowledge. That is, solving a geometric question may require mastering two domain knowledge points, or it may require mastering 𝑖 domain knowledge. Here, 𝑖 is taken as 1 to 7 in the GeoQA dataset and 1 to 5 in the GeoQA+dataset. As shown in Figure 7 , Approximately 42.3% of geometric problems can only be solved by combining two domain knowledge and 40.9% of the questions in GEOQA+ also contain more than one domain knowledge. To further analyze the impact of domain knowledge on solving geometry questions, we randomly selected 5 questions for each type of domain knowledge quantity, for a total of 100 rounds.The experimental results are shown in Figure 8 , when the number of domain knowledge increases, the accuracy of all methods decreases. According to Figure 8 (a), RaE performance remains high when solving questions involving less domain knowledge. PAL's performance is the worst. According to Figure 8 (b), when the number of domain knowledge increases, its performance is always superior to other prompting methods. The results show that RaE is more suitable than other methods for solving geometry questions with multi-domain knowledge.Is the error source of RaE prompting templates the reasoning or executing? Although our prompting method RaE performs well compared to other prompt methods in solving geometry questions, there is still room for improvement. Therefore, we analyzed the probabilities of reasoning errors and code execution errors using the same experimental setup as the Main Result in four geometry datasets. As shown in Figure 3 , we consider the program output " SyntaxError: invalid syntax " as a code execution error (𝑅 𝑒 ), and consider the generated answer not being numerically equal to the true answer as a reasoning process error (𝑅 𝑟 ). Since errors in COT prompt methods are all caused by 𝑅 𝑟 , we do not compare them here.From Table 4 , it can be analyzed that the main reason for PAL's error in answering questions is due to code execution errors. The error probability of our proposed RaE method in code execution is much lower than that of the PAL prompt method. Therefore, for PAL, to enhance the performance of solving geometric questions, it is necessary to optimize the generation of code blocks. For our work, we need to provide a more concise and rigorous reasoning process to guide MM-LLMs to achieve professional solutions. How does RaE work with different geometry shapes? The most important thing in solving geometry questions is to recognize geometric shapes. To analyze the impact of different geometry shapes on MM-LLms with different prompting methods, we analyzed the distribution of questions with different shapes in the geometry3k dataset. The analysis results are shown in Figure 9 , according to Figure 9 (a), questions containing only triangles account for 40% of the total number, and 17% of the questions contain more than one shape. Furthermore, as shown in Figure 9 (b), more than 82% of the questions in the dataset geometry3k contain only one kind of shape. To further analyze the impact of geometry shapes on solving geometry questions, we randomly selected 5 questions for each type of geometry shape, for a total of 100 rounds. The experimental results are shown in Figure 10 . The experimental results are explicit, Our RaE has the best performance in all kinds of geometric shapes, especially in quadrangles and triangles. The performance of CoT is only inferior to RaE, and the gap between CoT and RaE is the largest in the triangle. PAL and no prompt were the worst. To further analyze the number impact of geometry shapes on solving geometry questions, we randomly selected 40 questions for each type of geometry shape quantity, for a total of 50 rounds. The experimental results are shown in Figure 11 . According to Figure 11 , our RaE is the best in solving questions with one to three kinds of geometry shapes. When the number of shapes increases to four, COT performs better than all other prompting methods. This is caused by the fact that the reasoning process of RaE's prompt template did not fully consider the questions of mixing multiple shapes. Is RaE more suitable for "zh" or "en"? Due to the presence of both English and Chinese questions in our original geometry datasets, to minimize the interference factors of MM-LLMs in answering the questions, we unified the language of the four geometric data. Thus, we analyzed the impact of the language used in the question on the accuracy of the answer using the same experimental setup as the Main Result in four geometry datasets. The experimental results are shown in Table 5 . The results demonstrate that our RaE and other prompting methods are more suitable for English questions, and our answering accuracy reaches the highest of 32.7% on the GeoA_en dataset.  
conclusion: 
 In this paper, We introduce RaE prompting, a new prompting method specifically designed for MM-LLMs to solve geometric questions. It ensures MM-LLMs generate reasoning processes by utilizing domain knowledge, and generate executable code blocks to obtain answers. Compared with the few-shot CoT and PAL methods, the RaE prompting method, designed for more professional solving of geometric problems, features both rigorous reasoning processes and precise arithmetic operations. From the overall results of the experiment, our RaE showed impressive performance on four geometry question datasets. To analyze the influencing factors of solving geometry questions in more detail, we tested the answering performance of different prompting methods based on the GPT4V model, demonstrating rich experimental results. Our work provides a more comprehensive research approach to improving large language models for solving geometry questions and points the way for future research.  
reason-and-execute prompting: enhancing multimodal large: 
 Language Models for Solving Geometry Questions MultiModal Large Language Models (MM-LLMs) have demonstrated exceptional reasoning abilities in various visual question-answering tasks. However, they encounter significant challenges when answering geometry questions. These challenges arise due to the need to engage in rigorous reasoning and executing precise arithmetic. To enhance the ability of LLMs to solve multimodal geometric questions, we propose Reason-and-Execute (RaE) prompting: a new prompting method specifically designed for enhancing MM-LLMs to solve geometric questions. Specifically, we first designed a rigorous reasoning process based on domain knowledge of geometry, using a reverse thinking approach, and obtained the precise arithmetic steps required for solving the question. Secondly, based on the analysis of the reasoning process, we designed code blocks in a programming language to implement the arithmetic functions. Finally, by executing the contents of the code blocks using an interpreter, we obtained the answers to the geometric questions. We evaluated the accuracy of 9 models in answering questions on 6 datasets (including four geometry datasets and two science datasets) using different prompting templates. Specifically, in the main experimental result, our RaE showed a maximum enhancement of 12.8% compared to other prompting methods, which proves strong reasoning and arithmetic abilities in solving geometric questions of our method. Moreover, we analyzed the impact of answering from the perspective of solving geometric problems by considering multiple factors, including domain knowledge, geometry shapes, understanding of the question text, and language. This once again emphasizes that our method has passed the comprehensive test of solving geometry questions. The source code and data will be published in a GitHub repository. 
 
 Appendix:  


Decision: This is a good submission and all reviewers are quite positive. All of the concerns have been addressed in the rebuttal so reviewers voted for an acceptance. The final decision is accept.

Reviews: review: paper_summary: paper_summary: By combining domain knowledge and program-aid strategies in prompts with a method called RaE, this article alleviates the problem of generating unexecutable code in the PAL method and promotes the ability of multi-modal large models to solve geometric problems. RaE is evaluated on 9 models across 6 datasets, which nicely demonstrates the broad applicability of the method.

Overall, RaE is a simple but efficient prompt strategy to stimulate the performance of MM-LLMs.
strength_weakness: strength_weakness: 1. RaE is evaluated on 9 models across 6 datasets, which nicely demonstrates the broad applicability of the method.
2. RaE is simple but efficient.
3. This paper is expressed with clarity and fluidity.
limitations: limitations: 1. RaE is not as effective as CoT on some models and data sets (GeoQA+ and qwen-vl-plus, etc.), but the author does not provide an analysis.
2. The experimental part does not show case studies (including good case and bad case).
3. It is necessary to compare some methods of improving PAL, such as PoT[1]
4. It is recommended that the presentation form of Table 2 be adjusted so that the performance of different methods on the same model and dataset can be clearly compared.
5. Since LLMs are prone to hallucination when generating specific domain knowledge, it is recommended to evaluate the accuracy of the following "#reasoning" part separately.
paper_summary: paper_summary: This paper demonstrates the existing challenges that multi-modal large language models face in answering geometric questions, through an error analysis conducted on GPT-4V. Then, it proposes a Reason-and-Execute prompting method to enhance multi-modal LLMs to solve geometric questions. Experiments on six benchmarks show the effectiveness of its methods.
strength_weakness: strength_weakness: - This paper presents an interesting and meaningful problem: how to enhance multimodal large language models to better handle geometric questions. It also exposes the potential challenges through an error analysis conducted on GPT-4V, clearly articulating the motivation behind this work.

- This paper is well-written, and the figures are clear and easy to understand.

- The method proposed in this paper is simple and intuitive, yet appears to be reasonable.

- This paper conducts experiments on 6 benchmarks and adopts the latest multimodal large language models (e.g. GPT-4V, Gemini, and Qwen). The experimental results demonstrate significant improvements and quantitative analyses showcase the effectiveness of the proposed method.
limitations: limitations: The paper does not have any obvious shortcomings, but I still have the following concerns:
- Is the method proposed in this paper sensitive to the choice of prompts? The authors should provide some guidance on how to select prompts. Additionally, the authors should release their code and prompts after publication to ensure reproducibility.

- There are some related works that solve mathematical reasoning problems through code generation. The authors should mention these papers in the related work section, such as [1].

[1] PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning.
paper_summary: paper_summary: The work proposes RaE, a novel prompting method called Reason-and-Execute (RaE). The RaE is designed to improve the performance of MultiModal Large Language Models in solving geometry questions.

This approach involves two main steps: 
- [A] Designing a rigorous reasoning process based on domain knowledge
- [B] Generating executable code blocks for precise arithmetic.

The authors evaluate their method against existing prompting techniques like Chain-of-Thought (CoT) and Program-Aided Language (PAL) across various datasets, demonstrating that RaE significantly enhances accuracy in solving geometric problems.
strength_weakness: strength_weakness: - Approach wise: The RaE prompting method introduces an innovative way to enhance MM-LLMs' ability to solve geometry problems by combining domain-specific reasoning with precise arithmetic execution.
- Evaluation and analysis: The paper includes thorough experimental evaluations on six benchmark datasets, showcasing the effectiveness of RaE compared to CoT and PAL prompting methods. The authors also provide an in-depth analysis of error sources, distinguishing between reasoning and execution errors, which offers valuable insights for further improvements.
- Scalability Considerations: The paper acknowledges the computational demands and addresses the scalability of their approach by evaluating both closed-source and open-source models.
limitations: limitations: As for baselines, while the paper compares RaE with CoT and PAL, it lacks a discussion on why certain other relevant prompting methods were not included in the evaluation.

I also have a few questions.

- Scalability: What specific optimizations were employed to manage the computational load (e.g. LoRA), and are there any plans to improve the scalability of the RaE method for larger datasets?
- Practical Application: How does the proposed framework translate to real-world applications, particularly in educational contexts? Are there any ongoing pilot projects or collaborations with educational institutions?
- Feature Integration: How does the model handle potential conflicts or redundancies when integrating multiple domain-specific knowledge points in the reasoning process?