MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems

Authors: bin  lei
yi  zhang
shan  zuo
ali  payani
caiwen  ding
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:33:11

Abstract: Recent advancements in large language models, such as GPT-4, have demonstrated remarkable capabilities in processing standard queries. Despite these advancements, their performance substantially declines in advanced mathematical problems requiring complex, multi-step logical reasoning. To enhance their inferential capabilities, current research has delved into prompting engineering, exemplified by methodologies such as the Tree of Thought and Graph of Thought.
Nonetheless, these existing approaches encounter two significant limitations. Firstly, their effectiveness in tackling complex mathematical problems is somewhat constrained. Secondly, the necessity to design distinct prompts for individual problems hampers their generalizability.
In response to these limitations, this paper introduces the Multi-Agent System for conditional Mining (MACM) prompting method. It not only resolves intricate mathematical problems but also demonstrates strong generalization capabilities across various mathematical contexts.
With the assistance of MACM, the accuracy of GPT-4 Turbo on the most challenging level five mathematical problems in the MATH dataset increase from $\mathbf{54.68\\%}  \text{ to } \mathbf{76.73\\%}$.

Main paper: 
 introduction: 
 Large language models (LLMs) like GPT-4 [12] excel in various problem-solving tasks but still struggle with complex logical deduction, especially in mathematics involving abstract concepts and multi-step reasoning [11, 2] . This limitation hinders their accuracy and reliability in fields requiring precise mathematical reasoning, such as academic research, engineering, and theoretical physics.A contemporary and efficacious method for tackling this issue is the prompting engineering [19] . It enhances accuracy in complex problem-solving without necessitating further training of the model. By strategically crafting prompts, prompting engineering optimizes the utilization of large language models, guiding their processing pathways more efficiently and effectively [10] .Previous prompting methods mainly include the Chain of Thought (CoT) [18] , Self-consistency Chain of Thought (SC-CoT) [17] , Tree of Thought (ToT) [20] , and Graph of Thought (GoT) [4, 9] . CoT and SC-CoT show limited capabilities in complex logical reasoning, achieving only 4.0% and 9.0% accuracy in simple tasks like the 24-point game using GPT-4 [20] . While ToT and GoT have improved LLMs' problem-solving abilities, they lack generalizability, requiring specific prompts for each problem, as detailed in Appendix A. Universality We propose the Multi-Agent System for Condition Mining (MACM) prompting method. MACM has moved beyond being restricted by the specific contents of a problem. Instead, it first abstracts the conditions and the objective of the problem. Subsequently, through a Multi-Agent interactive system, it progressively mines new conditions conducive to achieving the objective, thereby ultimately fulfilling the goal of problem-solving.The comparison between MACM and current mainstream prompting methods in problem-solving is shown in Figure 1 . MACM extracts conditions and objectives from each math problem, iteratively adding new insights until enough information is gathered to find a solution. Performance-wise, MACM improves accuracy by over 10 percentage points on datasets like the 24-point game, matching the effectiveness of ToT and GoT. Moreover, MACM is versatile; it can apply the same set of prompts to various mathematical reasoning problems without manual modifications, unlike the tailored prompts needed for ToT and GoT.Through several experiments on the math related datasets, we verified MACM's generalizability and superior error correction compared to original prompting methods. With MACM, the GPT-4 turbo model's accuracy on the MATH dataset increased by 15.14%, and by 7.8% compared to SC-CoT. In the 24-point game, MACM achieved an accuracy rate 17% higher than ToT.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: The paper's contributions is included in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 In this section, we summarize several major current prompting methods.I-O Prompting: Input-Output (I-O) prompting is the most common method for interacting with large language models, where users specify the problem conditions directly to the model, which generates responses through a token-level, sequential decision-making process [22] .CoT Prompting: [18] : Chain of Thought (CoT) prompting refines the model's output into more structured and logically coherent text by methodically constructing and elaborating upon chains of reasoning. This approach enhances the model's ability to produce outputs rooted in logical deduction. There are several variants of CoT, including Zero-Shot CoT [8] , Few-Shot CoT [7] , and Auto-CoT [22] , each tailored to different prompting scenarios and requirements to improve logical reasoning in diverse contexts.Prompting: [17] : Self-Consistency Chain of Thought (SC-CoT) prompting improves upon CoT by introducing a voting mechanism, which emphasizes internal consistency and semantic interconnectedness. In this method, models evaluate (vote on) their own outputs to select the most coherent response, thereby reducing logical fallacies and inconsistencies.ToT Prompting: [20] : Tree of Thought (ToT) prompting uses a hierarchical, tree-like structure to organize and guide the model's text generation. This method improves precision and structure in responses, incorporating a voting mechanism to refine outcomes and reduce computational demands.GoT Prompting : [4, 9] : Graph of Thought (GoT) prompting enhances ToT by allowing interconnections between thoughts on different branches. It decomposes complex tasks into simpler sub-tasks, solves them independently, and merges the results, thus reducing computational costs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitation and discussion are included in the Appendix.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
macm overall structure: 
 The overall structure of MACM is shown in Figure 2 . We have designed an interactive system When a mathematical problem is input into our system, the Thinker initially sets up the Condition List and defines the final Objective based on the given problem. After initialization, the Thinker mines new conditions conducive to the objective from the current Condition List, i.e., the Known Conditions. The Judge then assesses these newly mined conditions. If deemed correct, the Judge incorporates the new condition into the Condition List. Otherwise, the new condition is discarded.Once all new Conditions have been reviewed, we obtain a revised Condition List. At this point, the Judge evaluates whether the current conditions are sufficient to achieve the objective. If the answer is False, the process reverts to step ➁ for further mining of new conditions. In our experiments, we set a limit of five iterations; if the objective is not met after five rounds of mining, we consider the problem unsolvable. This prevents the program from entering an infinite loop. If the answer is True, the Thinker designs steps based on the Known Conditions to achieve the Objective. Finally, the Executor performs calculations following these steps to produce the final result.MACM achieves a high level of generalizability by abstracting conditions and objectives from each specific mathematical problem. Through a multi-agent interactive system, where the Thinker is responsible for ideation and design, the Judge for inspection and decision-making, and the Executor for computation, most potential errors in reasoning and calculation are eliminated. By repeatedly mining for conditions and adding the correct ones to the Condition List, MACM ensures depth in thinking, making it suitable for analyzing complex mathematical problems.MACM moves away from the hierarchical dependencies of previous methods by introducing Conditions and Objectives. It continuously expands Known conditions to derive the final answer, eliminating the need for manual, problem-specific prompts. This method compresses information from various Thoughts into existing Condition List, capturing more connections than traditional prompting methods that rely on navigating a hierarchical structure.The Thinker initiates a thought set T 1 = {T 1 1 , T 2 1 , . . . , T m 1 } contains m new thoughts from question Q and generates subsequent thought setsT i = {T 1 i , T 2 i , . . . , T m i } based on the current condition list C i = {C 1 , C 2 , . . . , C i-1 }. Each condition C i-1 is derived from the most accurate thought T * i-1 in T i-1 . At each step i, the Judge selects the correct thought T * i in thought set T i such that T * i = arg max s P Judge i (T * i |T s i ∈ T i , s ∈ {1, • • • , m}), where P Judge is the probability that the Judge confirms the thought as correct. By using this method, we map the whole thoughts space T to the Condition List C. In an ideal situation where P Judge → 1, we have T → C. The probability of arriving at the correct answer A correct based on the final Condition list C is equal to that based on the entire thoughts space T. Thus we have:P MACM (A correct |C) = P MACM (A correct |T) > P GoT, ToT, CoT (A correct | {T ij | i = 1, 2, .. . , m and j = 1, 2, . . . , n} ⊆ T). In practice, where P Judge ̸ → 1, we performed the experiments to test its performance, the results are shown in Section 4.Our prompts and use cases are shown in Figure 3 . It demonstrates the specific process of MACM analyzing algebra and geometry problems. In these two examples, we have employed OpenAI's GPT-4 Turbo [1] as the intelligent agent, which is capable of performing calculations using code. It is endowed with three roles: Thinker, Judge, and Executor by using the following instructions:For Thinker: You take the role of a Thinker. I need you to help me gradually ponder over some problems following my instructions. You need to answer the question by using the following format: Based on Condition A and Condition B, we can get: C.For Judge: You take the role of a Judge. I need you to make judgments on some statements. You are only allowed to use the True or False as the final answer.For Executor: You take the role of a Executor. I need you to calculate the final result based on the given conditions and steps.Let S be the set of all real numbers α such that the function x 2 +5x+αx 2 +7x-44 can be expressed as a quotient of two linear functions. What is the sum of the elements of S? GPT-4 Turbo's raw response reached an incorrect conclusion: x 2 + 5x + α = k(x + 11)(x -4) , which then led to issues in the subsequent code design, ultimately resulting in an incorrect output.In the MACM analysis process, the Thinker initially identifies conditions and objectives from the problem statement and then uncovers new conditions. Although the Thinker initially identifies the same incorrect condition as GPT-4 Turbo, the Judge detects and rejects this error, preventing its addition to the Known Conditions. In the second round, the Thinker identifies two new conditions: (-11) 2 + 5 × (-11) + α 1 = 0) and (4) 2 + 5 × (4) + α 2 = 0 , which the Judge verifies and adds to the Known Conditions list. The Judge then confirms that the Known Conditions are sufficient to achieve the objective. The Thinker designs steps to reach the objectives based on these conditions, and finally, the Executor performs the necessary calculations to produce the result.Square ABCD has side lengths of 13 units. Point E lies in the interior of the square such that AE = 5 units and BE = 12 units. What is the distance from E to side AD? While GPT-4 Turbo's response had the correct theoretical approach, it failed to identify relationships between points in the problem, leading to incorrect expressions and an incorrect final result. The result is $20 or more is approximately 0.3214, or 32.14%.Conditions: An envelope contains a total of 8 bills. The envelope contains 2 one-dollar bills. The envelope contains 2 five-dollar bills. The envelope contains 2 ten-dollar bills. The envelope contains 2 twenty-dollar bills. Two bills are drawn at random without replacement. Objectives: Calculate the probability that the sum of the two drawn bills is $20 or more.New condition: There are a total of unique combinations of two bills that can be drawn. The answer is:Original Wrong Analysis Prompt 1equation is given by . we can apply the British Flag Theorem. This theorem states that in a rectangle, if a point is chosen inside it, then the sum of the squares of the distances from to two opposite corners is equal to the sum of the squares of the distances from to the other two opposite corners. Let be a square with side lengths of units, and let be a point inside the square such that units and units. We want to find the distance from to side . Let be the distance from to side . The distance from to side is also . Using the British Flag Theorem:We can solve this equation to find . To find the set of all real numbers such that the function can be expressed as a quotient of two linear functions, we need to find conditions under which the numerator and the denominator are proportional to each other. The denominator can be factored as follows:. Now, for the function to be expressible as a quotient of two linear functions, the numerator must be a scalar multiple of . This means there exists some constant such that . Expanding the right-hand side, we get . Equating the coefficients of like terms on both sides, we get . We can solve these equations to find the value is the set of all real numbers Objective: Calculate the sum of the elements of the set .New condition:Prompt 3Prompt 4The answer is True True During the MACM analysis, the Thinker first clarifies the conditions and objectives of the geometry problem and then uncovers new conditions to achieve the goal. Initially, it discovers that: △ABE is a right triangle . After verification by the Judge, this condition is added to the known conditions. The Judge then checks if the known conditions are sufficient. A False result means more conditions are needed, so the Thinker continues searching. In the second round, the Thinker deduces AE × EB = EF × AB , which the Judge verifies and adds to the Condition List.Code Analysis AreaUpon confirming sufficiency, the Thinker plans the steps to solve the problem, and the Executor performs the calculations to find the result.In analyzing these two problems, MACM first extracts the specific conditions and objectives from the questions. This allows MACM to directly use these conditions and objectives for prompt design in subsequent processes, enhancing our approach's generalizability. Previous methods like ToT and GoT lack this setup, resulting in poorer generalizability. For example, in the 24-point game experiment with ToT, the lack of this setting necessitated the manual configuration of the following prompt:Evaluate if given numbers can reach 24 \n (sure/likely/impossible) \n {input}In MACM, the if given numbers can reach 24 is obtained by the first step and the evaluation prompt is generalized to Evaluate {objective} {input} , ensuring higher generalizability.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The Theoretical Analysis is shown in Section 3.2. And we provide the corresponding assumptions for each conclusion.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
performance on math benchmark: 
 The MATH dataset [6] includes a variety of mathematical problems. It offers seven types of mathematical problems, including geometry, algebra, probability theory, etc., with difficulty levels ranging from 1 to 5. We first tested the overall performance of MACM on the MATH dataset without distinguishing difficulty levels. Afterward, we specifically selected the most difficult mathematical problems from the MATH dataset for testing. The detailed experimental setup is presented in the Appendix B. In Table 1 , we compared the accuracy of GPT-4 Turbo on the MATH dataset with various prompting methods. We found that compared to the original GPT-4 Turbo, MACM increased its accuracy by 20%. Compared to CoT, the increase was 13.56%, and compared to SC-CoT, it was 7.8%. Among these, MACM led to the greatest improvement in accuracy for the original GPT-4 Turbo model on number theory problems, at 23.53%. In geometry problems, although MACM has increased the accuracy of GPT-4 Turbo by 17.63%, the final accuracy rate is still only 62.74%. Upon analyzing the causes of errors, we found that many mistakes were due to GPT-4 Turbo's difficulty in accurately understanding the relationships between various geometric figures, thereby failing to design corresponding code to solve the problems. However, in algebra and number theory problems, MACM, by correcting the erroneous analysis of GPT-4 Turbo and helping it explore potential approaches, achieved accuracy rates of 96.07% and 98.04%, respectively. Moreover, compared to the previous SOTA method, CSV prompting, on the MATH dataset, MACM achieves a 3.6 percentage points higher accuracy rate on the same dataset. This demonstrates the effectiveness of MACM in solving mathematical problems.In Figure 4 , we tested the performance of MACM on the open-source LLaMA series models on the MATH dataset and compared it with other prompting methods. Since LLaMA models do not have a code interpreter like GPT-4 Turbo, we disabled the code-checking function of MACM in this group of tests. The rest of the experimental setup was consistent with GPT-Turbo. In zero-shot scenarios, the accuracy rates of LLaMA 7B and LLaMA 13B were both below 5% [14] . Majority voting could enhance their accuracies to 6.9% and 8.8% respectively [14] , while MACM further increased them to 9.5% and 10.2%. On LLaMA 2 [15] and LLaMA 3 [3] , compared to 4-shots, MACM could further improve the accuracy on the MATH dataset by 3-5 percentage points. Overall, We found that MACM can also be applied to LLaMA models, although the performance improvement was not as significant as with GPT-4 Turbo. This is because GPT-4 Turbo has a better understanding of MACM's intrinsic directive prompts, enabling it to find the correct results more effectively. 7 B 0 -s h o t 1 3 B 0 -s h o t 7 B M a j-V . 1 3 B M a j-V . 7 B M A C M 1 3 B M A C M 7 B 4 -s h o t 1 3 B 4 -s h o t 7 B M A C M 1 3 B M A C MDue to the lack of generalization of ToT and GoT prompting methods (See Appendix A for the reason), we were unable to test them on the MATH benchmark. To compare MACM with them, we selected two mathematical problems where their methods are applicable: the 24-point game and sequence sorting. Among these, ToT tested the 24-point game, while GoT studied the sequence sorting problem. The detailed experimental setup is presented in the Appendix B. When the model is GPT-4, MACM is 17% higher than ToT (b = 5). Note that here, to ensure a fair comparison, we used the standard GPT-4 without any code capabilities. Additionally, with the support of MACM, GPT-3.5 also achieved an accuracy of 67% in the 24-point game, which is higher than the GPT-4 model with ToT (b = 1) support. Upon analyzing the reasons for the improvement in accuracy, we found that MACM's Judge corrected many thoughts that were mistakenly evaluated in ToT, leading to GPT-4 choosing incorrect approaches. This correction process significantly contributed to the increase in accuracy. In addition, We compared the GPT-3.5 model's ability to sort 64 numbers using GoT and MACM. MACM outperformed GoT by 2.94%. Note that some results marked with * were estimated from graphs without specific data. Additionally, GPT-4 Turbo achieved 100% accuracy on the Sequence Sorting task due to its problem-based code construction capability.This section primarily tests two capabilities of MACM:1. The ability to solve more challenging mathematical problems. We applied MACM to two datasets, SciBench [16] and TheoremQA [5] , which claim their difficulty surpasses the middle school level of the MATH dataset, reaching the Undergraduate Level.2. Transferability to General Logic Reasoning Tasks. Although MACM focuses on solving mathematical problems, to test its applicability, we applied it to the Reclor logic reasoning dataset [21] . Table 3 displays the testing results of MACM on the SciBench dataset. The SciBench dataset includes problems in chemistry, physics, and mathematics. We only selected the math-related subset for testing, which includes: the diff, stat and calc subset. The experimental setup was consistent with the testing on the MATH dataset (as shown in Appendix B). The results demonstrate that, in contrast to the Chain of Thought (CoT) method, which resulted in decreased accuracy for both GPT-4 and GPT-4 Turbo on this dataset, MACM led to an approximate 20 percentage points.Table 4 presents the performance of GPT-4 Turbo on the TheoremQA dataset using various prompting methods. The TheoremQA dataset encompasses problems from multiple domains including mathematics, physics, finance, and computer science. Notably, its mathematics subset contains numerous conceptual and definitional questions that do not involve logic reasoning processes; thus, these were not considered in our experiments. We solely tested questions within the TheoremQA mathematics subset that involve logic reasoning and have definite answers (e.g., a specific number, rather than an interpretation of a theorem). The experimental setup was consistent with the testing on the MATH dataset (as detailed in Appendix B). The results indicate that MACM enabled a roughly 30 percentage points increase in accuracy for GPT-4 Turbo on this subset.Table 5 shows the test results of GPT-4o [13] on the Reclor dataset using various prompting methods.For general logical reasoning problems, we adjusted the computation-related prompts in the original MACM to make them suitable for non-mathematical logical reasoning problems, while maintaining the overall structure consistent. Results show that MACM can also improve the accuracy of general logic reasoning tasks, but the increase is smaller compared to math tasks.In this section, we primarily investigate two issues. ➀ Explore the relationship between MACM Accuracy and LLM Queries, comparing it with other methods. ➁ Analyze the proportional impact of each component within the MACM on the overall accuracy improvement. We performed these two experiments on 200 randomly selected questions from the MATH dataset that the original GPT-4 Turbo model answered incorrectly.Trade-off Between Accuracy and LLM Queries: In general, increasing the number of responses generated by LLMs leads to an improvement in accuracy. Each prompting method has parameters that can increase it, such as the length of the chain l in CoT, the number of voters v in SC-CoT, and the Tree breadth b in ToT. To measure the search efficiency of each method, we compared the relationship between the accuracy and the number of responses generated by GPT-4 Turbo.We increased the number of answers generated by various prompting methods. For I-O prompting, we directly adjusted the model's response generation parameter n, which enables the model to n responses. For CoT, we adjusted not only the parameter n but also the length l of the Chain. For SC-CoT, we built on the first two methods by adding an adjustment to the number of voters v. As shown in Figure 6 , although I-O, CoT, and SC-CoT only require simple queries to correct the original errors made by GPT-4 Turbo, their upper limits are not high. Even if we continue to increase the number of queries, they can only correct about 20% of the original errors made by GPT-4 Turbo.In contrast, MACM can correct nearly 90% of the original errors of GPT-4 Turbo when the number of queries is high. This is actually quite reasonable because the MACM structure is more complex, including multiple processes of mining conditions and checking. These processes allow the large model to gradually think and identify errors, thus significantly improving accuracy.To analyze the functions of each component in MACM, we randomly combined the four components within MACM: Condition Mining, Multi-Agent System, Self-Checking, and Voting, and tested their performance. During the experiment, we maintained a maximum of 2 Condition Mining iterations and used 3 Voters.As shown in Figure 7 , the combination of all four components yields the best performance. Among the individual components, Multi-Agents and Condition Mining have comparable error correction capabilities. In the combinations of two components, the pairing of Self-Checking and Condition Mining shows the best performance. Among the three-component combinations, the combination of Multi-Agents, Condition Mining, and either Voting or Self-Checking achieves better results.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The code is provided in supplementary material and the experiment conditions are described in Appendix. 
conclusion: 
 We introduce MACM, a new and generalizable prompting technique that significantly enhances the inferential capabilities of large language models on mathematical problems. MACM can be applied to different types of mathematical questions. Through comparisons in several experiments on the math related datasets, we have verified the superiority of our method over the original prompting methods. With the aid of MACM, the accuracy of the GPT-4 Turbo model on the MATH dataset has increased by 15.14%. Compared to SC-CoT, its accuracy has increased by 7.8%. For the most challenging level 5 mathematical problems in the MATH dataset, its accuracy increased from 54.68% to 76.73%. In the game of 24 points, using the same GPT-4 model, MACM's accuracy is 17% higher than that of ToT. At the same time, by comparing accuracy with the number of times the large model responds, we find that MACM has a higher limit; increasing the number of responses from the large model can significantly improve accuracy. These experiments demonstrate MACM's generalizability and its powerful error-correction capability for complex mathematical problems in original LLMs.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code is provided in the supplemental material with detailed running instruction.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experiment settings are detailed in Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Calling GPT-4 Api is expensive. Due to funding constraints, we use the greedy sampling method (temperature = 0, do_sample = False) to generate the output. Each time, the generated results will be the same. This will avoid the potential inaccurate caused by the randomness.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: It is provided in the supplementary material.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We followed the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: We discussed the positive societal impacts. We did not find the negative societal impacts of this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cited their work in our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: we did not provide new models or new datasets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This section demonstrates specific examples of using ToT and GoT to further illustrate why their generalizability is limited.ToT conducted three sets of experiments in the original study, requiring specially designed prompts for each. The official implementation on their GitHub page https://github.com/princeton-nlp/treeof-thought-llm/tree/master/src/tot/prompts includes the specific prompts set up for each experiment.Taking the 24-point game as an example, specific prompts such as propose prompt, value prompt, and value last step prompt were required (lines 51 to 134 in game24.py). During ToT's operation, the LLM executes traversal searches, voting, filtering, etc., based on these written prompts. The authors also mention in the ToT readme section How to Add a New Task (https://github.com/princeton-nlp/treeof-thought-llm?tab=readme-ov-file#how-to-add-a-new-task) that setting up task-specific prompts is necessary for different problems, further illustrating the limited generalizability of ToT and GoT due to the need for task-specific prompt engineering.GoT faces the same issue, with their original paper conducting experiments in four tasks: Sorting, Set Operations, Keyword Counting, and Document Merging. For each type of problem, specific prompts must be set up on their official GitHub. for the cross word task, the propose prompt is:Let's play a 5 x 5 mini crossword, where each word should have exactly 5 letters.\n {input}\n Given the current status, list all possible answers for unfilled or changed words, and your confidence levels (certain/high/medium/low), using the format "h1. apple (medium)". Use "certain" cautiously and only when you are 100% sure this is the correct word. You can list more than one possible answer for each word.for the creative writing task, the propose prompt is:Write a coherent passage of 4 short paragraphs. The end sentence of each paragraph must be: {input}\n Make a plan then write. Your output should be of the following format:\n Plan:\n Your plan here. Passage:\n Your passage here.Each time the problem is changed, both ToT and GoT require an update to their respective prompts. The requirement to tailor prompts for each specific problem limits the generalizability of ToT and GoT to broader issues. MACM successfully addresses this challenge.For the experiments on the MATH dataset: We utilized the GPT-4 Turbo model (between January 1, 2024, and February 1, 2024) to test MACM's performance on the MATH dataset. For tests that did not distinguish by difficulty, we randomly selected one-third of the questions from the MATH dataset for evaluation. For the high-difficulty tests, we extracted all questions with a difficulty level of 5 and randomly selected half of the questions from each category for testing. The experiment are performed by using I-O, CoT, SC-CoT, and MACM methodologies. For all prompting methods, we standardized the number of responses n generated by GPT-4 Turbo to 1, T op k = 1 , and the temperature t = 0. For CoT, we set the maximum length of the chain l = 5, for SC-CoT, the number of voters v = 5, and the maximum length of the chain l = 5. For these three methods, we consistently maintained max_tokens at 512. For MACM, we kept the thinker's max_tokens at 512, the judge's max_tokens at 4, and the executor's max_tokens at 256.For the 24-point game experiment: We sourced data from 4nums.com, which offers 1,362 games ranked from easy to hard based on the time it takes humans to solve them. We focused on a subset of these games, specifically those ranked 901 to 1,000 (The same as ToT), to test on relatively difficult challenges. Success for each task is defined as producing a valid equation that results in 24, utilizing each of the input numbers exactly once. The performance metric is the success rate across these 100 challenging games.We utilized the GPT-4 and GPT-3.5 model (between January 1, 2024, and February 1, 2024) to perform the experiments. The MACM configuration for this experiment includes setting the number of responses generated by the model n = 1, T op k = 1, temperature t = 0, with the thinker's max_tokens at 512, the judge's max_tokens at 4, and the executor's max_tokens at 256.For the sequence sorting experiment: We randomly generated 100 sequences, each containing 64 elements, for testing. We utilized the GPT-3.5 model (between January 1, 2024, and February 1, 2024) to perform the experiments. The MACM configuration for this experiment includes setting the number of responses generated by the model n = 1, T op k = 1, temperature t = 0, with the thinker's max_tokens at 512, the judge's max_tokens at 4, and the executor's max_tokens at 256.While MACM significantly enhances the accuracy of large language models in tackling complex mathematical challenges, it incurs the cost of multiple invocations of the large language model for inference, leading to increased problem-solving time. Additionally, our evaluations using the MATH dataset indicate limitations in effectively addressing geometry problems. Addressing these challenges necessitates further advancements in the model's own cognitive capabilities. A proposed strategy involves employing prompting methods like MACM to assist the LLM in eliminating incorrect responses. This approach enables the creation of expansive, high-quality datasets, which are otherwise challenging to compile manually, and subsequently refining the LLM with these datasets. Through this iterative process, the model's intrinsic intelligence is progressively augmented. This research direction will constitute our future work.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper presents a prompting technique, MACM, for mathematical reasoning problems. The reviewers found the paper easy to follow and the method working. An extensive discussion emerged, and all reviewers declared that most of their concerns were answered, with three out of four reviewers increasing their ratings. The lowest-scoring reviewer expressed a belief that `the submission has a contribution for proposing a new effective and general prompting method` and left his score as a reminder of presentation issues. It is for that reason I recommend the paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a universal prompting method for solving complex reasoning problems such as mathematical problems and the 24-point game. The method first abstracts the conditions and objectives of the problems and then progressively discovers new conditions until enough information is gathered to solve the problem. Experiments on MATH, the 24-point game, SciBench, and TheoremQA indicate that the proposed method is effective and universal.
strength_weakness: strength_weakness: - In Table 1 and Table 2, the number of responses for each problem should be highlighted to indicate whether the comparison is fair.
questions: questions: - What is the number of responses for each model (including baselines and the proposed MACM) in Table 1 and Table 2?

- On which dataset is the experiment in Figure 6 conducted?
limitations: limitations: Yes
paper_summary: paper_summary: The paper presents a novel prompting technique MACM, which utilises multiple agents to cooperate and perform backtracking for mathematical reasoning problems.
strength_weakness: strength_weakness: My main concern is with the evaluation method. As Appendix B suggest that the authors used GPT4 Turbo as a judge and only tested on a rather randomly selected set of MATH. This is very worrying, because (a) MATH has its own evaluation protocol, and the Minerva paper [1] also gave a good evaluation protocol. The reliance on GPT4 Turbo as a judge seems unjustifiable. (b) Why randomly select a subset, instead of using the whole MATH test set?


[1] Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., ... & Misra, V. (2022). Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35, 3843-3857.
questions: questions: The MACM protocol seems rather convoluted to implement. Is there any venue to simplify it without losing performance in the authors' eyes?
limitations: limitations: The paper did not discuss much about the approach's limitation.
paper_summary: paper_summary: The paper proposes a prompting method called Multi-Agent System for conditional Mining (MACM). MACM involves three agents, Thinker, Judge, and Executor, who maintain a condition list and try to solve the problems when the conditions are sufficient.

The paper conducts experiments with GPT and Llama series on MATH, 24-points game and sequece sorting. The results show MACM’s superiority to prompting methods like ToT and GoT under these settings.
strength_weakness: strength_weakness: - MACM seems to lack novelty. MACM’s idea of maintaining a more free-form context for planning has been explored by previous works like Cumulative Reasoning.
- The experimental setups in the paper seem confused and lack rigor, making the results less convincing. Examples include but are not limited to:
  - In section 4.2, It is unreasonable to state that ToT and GoT are incompatible with MATH and quit comparing with them. Algorithms like ToT have been applied to open-form QA tasks by previous works like LLM reasoners.
  - Across most experiments, it is confusing to distinguish between 0-shot/IO and CoT for GPT-4 on reasoning tasks, for GPT-4 seems to always use CoT on reasoning tasks if not specially prompted.
  - Especially, experimental setups described in Appendix B seem rather unreasonable:
    - The paper states that it always uses “$Top_{k}=1$, and the temperature $t=0$”, i.e. greedy decoding, but this seems incompatible with SC-CoT (though still possible with minor randomness existing).
    - The meaning of “length of the chain $l$” in CoT seems not explained across the paper.
    - The `max_tokens` seems too small for complex tasks like MATH.
- The writing lacks clarity and is hard to follow:
  - The paper tends to list many settings that are different in multiple dimensions and their results in a line (e.g. Figure4,7, Table2,3), making attribution of the results difficult.
  - The paper spends much space on specific examples (\~2 pages across pages 4-6) but seems to show few special things.

Hopefully, the authors could put more work into experimental setup and presentation to improve the quality of the work.

# **After rebuttal and discussion**

The discussion period brings up 

- 2 new weaknesses:
    1. According to Figure 4, MACM's gains on open-source models are not clear and may be marginal. Could you compare MACM with 0-shot and Majority voting, at least on LLaMA3-Instruct 8B?
    2. To test the impact of different models on MACM as **a new method**, it is not sufficient for GPT-3.5 to only test with MACM -- comparisons with baselines are needed. However, this could also be achieved with open-source models, which should be much cheaper and faster.
- 1 new question: The exact versions of GPTs in the paper are unclear. You should at least annotate the exact versions and better compare the methods with the exact same version.

However, most concerns have been resolved by the authors in their replies.

Considering the improvement in the rebuttal and discussion, I believe that the submission has a contribution for **proposing a new effective and general prompting method, which is limited in being only applicable to strong models**.

However, the submitted manuscript is indeed not good enough because it **lacks many details and is not well-formed enough, actually bringing an unnecessary burden for the community to follow the work**, as shown in the long discussion. I tend to take it as not ready for publication in its current form. Hopefully, the authors will add all the necessary content to future versions.

In summary, I would like to keep my final rating as 4 as a reminder of the presentation problem and leave it to AC to decide whether this submission should be accepted.
questions: questions: See above in weaknesses.
limitations: limitations: See above in weaknesses.
paper_summary: paper_summary: This paper aims to address the limitations of existing methods that require distinct prompt designs for different mathematical problems. The authors propose a general Multi-Agent System for conditional Mining (MACM) method that uses three LLM agents (Thinker, Judge, Executor) to iteratively propose the conditions for a problem, verify whether the existing conditions can reach the objective, and execute calculations. Experimental results clearly show the accuracy improvements of MACM.
strength_weakness: strength_weakness: The main weaknesses include:

1.	The essential technical contributions of this paper may be limited, and an explanation is needed as to why the method is effective. (please refer to question 1 below).
2.	The theoretical analysis is a little weird and needs further explanations. (please refer to question 2 below).
3.	More experimental results are needed to demonstrate the effectiveness of the proposed method. (please refer to questions 3-4 below).
questions: questions: 1.	In the first using case in Figure 3, why do the LLM gets two different “New condition” (one correct, one incorrect) under the same prompt (prompt 2)? Does this mean that thinker in this paper is more likely to take advantage of the randomness of large models in repeated condition generation, rather than really improving the ability of LLMs to discover new conditions?
2.	In theoretical analysis, how to ensure that the thought space T can be traversed, i.e., why P_MACM(A_correct|C) = P_MACM(A_correct|T)?
3.	In Section 4.2, it's necessary to rerun the results of GPT4-ToT instead of copying the results from the original paper, because the capability of GPT4 is also improving. It is difficult to tell whether the improvement in Table 2 is caused by improvements in GPT4 itself or by the method in this paper.
4.	In Section 4.4, I think the most important measure of efficiency for applying this method is to compare the number of API calls from this method with different prompt methods (e.g., ToT, GoT) when obtaining the results in Tables 1 and 2. This is the most direct indicator of the cost in practice.
limitations: limitations: Yes, the authors have adequately addressed the limitations.

Comments: title: Response to authors' rebuttal
Reply to reviewer pd1k (Part 3)
Reply to reviewer pd1k (Part 2)
Reply to reviewer pd1k (Part 1)
comment: Thanks for the authors' clarifications. The authors have addressed most of my questions. I am pleased the authors could add the GPT4 experiments and efficiency comparison, and the results generally enhance the work. I would like to increase my score. I hope our discussions can be included in the further version.
Thank you for your detailed responses, which have addressed most of my concerns. I believe I provided a fair rating and intend to maintain it.
Thanks for your reply! You've resolved almost all my concerns. I've updated my final comments in the official review.
Thank you for your reply!

For: 

> I recommend making similar comparisons with ToT. 

We have updated the table, and the ToT experiments were conducted according to their official GitHub. We used their default settings, except for updating the model to `gpt-4-1106-preview` to ensure consistency in testing. The result is as follow:

| Method                 | Total Cost (dollar) |  Total Input Tokens | Total Output Tokens | Accuracy (%)| 
|:----------------------:|:------------:|:------------:|:------------:|:------------:|
|  SC-CoT  (voter = $32$)  |      $16.5636$  |	$897960$	|	$252800$	|	$43$	|
|  SC-CoT  (voter = $40$)  |      $20.9034$|	$1126104$	|	$321411$	|	$47$	|
|  ToT  		|      $30.3382$	|	$1388479$	|	$548441$	|	$78$	|
|    MACM   		|	$20.5040$	|	$1697690$	|	$117569$	|	$91$	|

For: 

> Besides, for SC-CoT, comparisons on MATH/SciBench/TheoremQA are more convincing for me.

We searched for all significant work related to prompting and found no previous detailed token-level statistics for the SC-CoT method on the entire MATH/SciBench/TheoremQA datasets. However, rerunning these three datasets would cost nearly ten thousand dollars. Therefore, we have calculated the cost of the SC-CoT method based on our experimental logs for **Figure 6**. The results are as follows:

| Method                 | Total Cost (dollar) |  Total Input Tokens | Total Output Tokens | Accuracy (%)| 
|:----------------------:|:------------:|:------------:|:------------:|:------------:|
|  SC-CoT  (voter = $20$)  |      $82.64$  |	$4869230$	|	$1131738$	|	$23.5$	|
|    MACM   		|	$84.42$	|	$6932401$	|	$503276$	|	$84$	|
Thank you for your reply!

**For *New Question***

As we mentioned in our previous response to you, we

> ensure consistency in the testing version

In all tests (including the baseline):

> all references to GPT-4 point to gpt-4-1106-preview, and all references to GPT-3.5 point to gpt-3.5-turbo-0125

**For *New Weakness***

As per your request, we conducted new tests at temperatures of 0.3 and 0.7, and used the tokenizer of llama3-8B-Instruct to calculate the average number of input tokens and output tokens. The results are as follows:

| Method                 |  Average Input Tokens | Average Output Tokens | Accuracy (%)| 
|:----------------------:|:------------:|:------------:|:------------:|
|  0-shot  (voter = $32$)  |$65.8$	|	$513.1$	|	$22.7$	|
|  Majority Voting  (t = $0.1$)  |$479.2$	|	$1179.2$	|	$29.5$	|
|  Majority Voting  (t = $0.3$)  |$432.7$		|  $1019.2$	|	$31.2$	|
|  Majority Voting  (t = $0.7$)  		| $526.1$		|	$1345.7$	|	$28.6$	|
|    MACM   		|$784.7$	|	$1137.6$	|	$33.7$	|

**For *New Weakness 2***

Since prompting fundamentally relies on the model's inherent intelligence, our primary experiments, like previous prompting works such as ToT, CSV, and CR, are conducted on GPT-4. For GPT-3.5 and some open-source models, our supplementary experiments mainly provide validation for the extensibility of our method. Specifically for GPT-3.5, we did not merely test with MACM, but rather compared it with GoT in **Table 2**.
Thanks for your reply!

The experiment setting about cost is great! However, it is clear that ToT has great advantage over SC-CoT on 24-points game, which should be a better baseline. I recommend making similar comparisons with ToT. Besides, for SC-CoT, comparisons on MATH/SciBench/TheoremQA are more convincing for me. It is a pity that these results are not enough for me at the moment. Hopefully, you could always compare with the most competitive baselines and on the most meaningful tasks to your best.
Thanks for your reply! Below are my follow-up concerns:

**For *New Question:*** You should also check the versions of GPTs used in baselines ... Please always try to be clearly comparable. It would be best if you could provide all related results at one time.

**For *New Weakness:*** It is unreasonable to use a temperature as low as 0.1 for majority voting. Some more common choices are 0.3 or 0.7. It would be best if you could conduct a search. Besides, you should try to control the computation cost to be comparable (better in tokens). Given you have access to 8 A100 GPUs (80 GB memory), experiments on open-source models should be much cheaper and faster than ones on GPTs, which is a good way to resolve our concerns about the efficiency of the method.

**New Weakness 2**: To test the impact of different models on **MACM as a new method**, it is not sufficient for GPT-3.5 to only test with MACM -- comparisons with baselines are needed. However, this could also be achieved with open-source models, which should be much cheaper and faster.
Thank you for your reply and reviewer pd1k's comments on this issue.

---

To better address the question:
> how much MACM improves upon SC-CoT when the allowed tokens are the same

We conducted a detailed analysis on the 24 Game dataset, comparing SC CoT’s Input Tokens/Output Tokens and accuracy when the total cost is close to that of MACM. The detailed results are as follows:

***Experiment Setup***

We tested SC-CoT with the number of voters set to $32$ and $40$. When the number of voters is $32$, the total cost of SC-CoT is less than that of MACM, while with $40$ voters, the total cost of SC-CoT exceeds that of MACM. The parameters used for MACM and the 24-point game dataset are consistent with those in **Table 2**. We used the model `gpt-4-1106-preview` and the code interpreter is disabled. The version of the `openai` library is `1.35.1`.The Input Tokens/Output Tokens/Total Cost/statistical code is as follows:

```
        # Cost summary (Based on https://openai.com/api/pricing/)
        input_tokens = completion.usage.prompt_tokens
        output_tokens = completion.usage.completion_tokens
        input_cost_per_million = 10.00   # $10.00 / 1M tokens
        output_cost_per_million = 30.00  # $30.00 / 1M tokens
        input_cost = (input_tokens / 1000000) * input_cost_per_million
        output_cost = (output_tokens / 1000000) * output_cost_per_million
        total_cost = input_cost + output_cost
        accumulate_cost = total_cost + accumulate_cost
```

***Experiment Results***

| Method                 | Total Cost (dollar) |  Total Input Tokens | Total Output Tokens | Accuracy (%)| 
|:----------------------:|:------------:|:------------:|:------------:|:------------:|
|  SC-CoT  (voter = $32$)  |      $16.5636$  |	$897960$	|	$252800$	|	$43$	|
|  SC-CoT  (voter = $40$)  |      $20.9034$|	$1126104$	|	$321411$	|	$47$	|
|    MACM   		|	$20.5040$	|	$1697690$	|	$117569$	|	$91$	|

***Experiment Conclusion***

Tests on the 24-point game indicate that when the costs are similar, MACM nearly doubles the accuracy compared to SC-CoT.

---

Thank you again for your reply! And we welcome any further questions you may have.
Thank you for your reply and new suggestions!

---

**For *Weakness 3 (a)***

> Could you make similar revisions for other tables and figures for easier controlled variable comparison?

Certainly, for **the 24-point game in Table 2**, we plan to break it down into several sub-tables, comparing based on prompting method/model/and whether code is used. The results are as follows:

1. Comparison of different Prompting Methods, ensure that the model is set to `gpt-4-1106-preview` and that the code interpreter is entirely disabled.

| Method                | Accuracy (%) |
|:----------------------:|:------------:|
|  IO               |      $23$    |
|  CoT |      $25$       |
| SC-CoT (voter = $4$)  |      $34$      |
| ToT (b = $1$)|     $55$       |
| ToT (b = $5$)|     $78$      |
| MACM           |     $91$      |

2. Test the impact of different models on MACM. Specifically, test MACM using `gpt-3.5-turbo-0125` and `gpt-4-1106-preview`, with the code interpreter disabled.

| Model                | Accuracy (%) |
|:----------------------:|:------------:|
|  gpt-3.5-turbo-0125  |      $67$    |
| gpt-4-1106-preview |      $91$     |

3. Assess the performance improvement of MACM when using the code interpreter, ensuring the model remains `gpt-4-1106-preview`.

| Code interpreter usage | Accuracy (%) |
|:----------------------:|:------------:|
|  Yes  |      $99$    |
| No |      $91$     |

For **the Sequence sorting in Table 2** and **Table 3**, we will split it in the same way as the 24-point game in Table 2.

**For *New Question***

> Could you provide exact versions of GPT-3.5 / GPT-4 / ... used?

Sure, all references to GPT-4 point to `gpt-4-1106-preview`, and all references to GPT-3.5 point to `gpt-3.5-turbo-0125`.

**For *New weakness***

> According to Figure 4, MACM's gains on open-source models are not clear and may be marginal.

This is because open-source smaller models themselves lack sufficient intelligence, making them unable to handle structured prompts effectively. For example, in our analysis of some error cases with LLaMA3-Instruct 8B, we found that sometimes using few-shots not only fails to help solve the problem but also causes the model to repeatedly generate the prompts from the few-shots. This is particularly unfriendly for agent systems.

> Could you compare MACM with 0-shot and Majority voting, at least on LLaMA3-Instruct 8B? 

Sure, the experimental details are as follows:

***Experiment Setup***

We used the `vllm` library for model inference, enabling `flash-attention-2` to accelerate the attention computation process, with the `kv_cache` memory reserved to 95% of the total GPU memory. Inference was conducted using tensor parallelism. The experiments were run in parallel on 8 A100 GPUs (80 GB memory). The temperature was set to $0.1$, and the number of voters for majority voting was set to $4$.

***Experiment Results***

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|0-shot|$22.7$%|
|Majority Voting|$29.5$%|
|MACM|$33.7$%|

***Experiment Conclusion***

The experimental results show that compared to other prompting methods, MACM yields the most significant improvement for LLaMA3-Instruct 8B on the MATH dataset.

---

Thank you again for your reply! And we welcome any further questions you may have.
Thanks for your reply! Below are my follow-up concerns:

**For *Weakness 3 (a):*** The new format is better. Could you make similar revisions for other tables and figures for easier controlled variable comparison?

**For *New Question:*** Could you provide exact versions of GPT-3.5 / GPT-4 / ... used? Please make everything comparable in a controlled variable way.

**New weakness**: 

According to Figure 4, MACM's gains on open-source models are not clear and may be marginal. Could you compare MACM with 0-shot and Majority voting, at least on LLaMA3-Instruct 8B?
Thanks to the authors and the review zB7e for the efforts in the discussion.

I agree that different inference-time methods should be compared under the same computation budget. A previous method might present weaker results only because it used a smaller budget. I will also consider this when rating.
Thank you for your reply. They have addressed my concerns well. I have bumped the score accordingly.

Although, I do not think the question of "how much does MACM improve upon SC CoT if the tokens allowed are the same" is well answered. I've only seen the number of average responses and not the performances when token count is controlled. I would appreciate the effort to conduct such an experiment as I think this will significantly improve my confidence in the proposed method, but I understand if this is not feasible under the time/resource constraint.
**For *Weakness 3 (a)***:

Thank you for your suggestion! We will break down the tables/figures containing multiple variables into a more easily comparable format. Taking our **Figure 4** as an example, we will split the different versions and parameters of the LLaMA models into multiple tables, as shown below:

Table 1: Accuracy comparison of **LLaMA 7B**

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|0-shot|$2.9$%|
|Majority Voting|$6.9$%|
|MACM|$9.5$%|

Table 2: Accuracy comparison of **LLaMA 13B**

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|0-shot|$3.9$%|
|Majority Voting|$8.8$%|
|MACM|$10.2$%|

Table 3: Accuracy comparison of **LLaMA2 7B**

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|4-shot|$14.6$%|
|MACM|$17.2$%|

Table 4: Accuracy comparison of **LLaMA2 13B**

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|4-shot|$28.7$%|
|MACM|$32.1$%|

Table 5: Accuracy comparison of **LLaMA3-Instruct 8B**

| Method                    | Acc. (%)         |
|:---------------------------:|:--------------------:|
|4-shot|$30$%|
|MACM|$33.7$%|


**For *Weakness 3 (b)***:

Thank you for your suggestion! In future versions, we will try to condense the text that repeats the content of **Figure 3** to make room for updates to the figures/tables that follow.

**For *New Question***:

Thank you for your suggestion. As mentioned in **For *Weakness 1 (3)***, we will clearly state this information in the paper: In our experiments, the GPT-4 model we used is `gpt-4-1106-preview`. 

As mentioned in our response to **reviewer upb5 in Question 3**, to ensure consistency in the testing version, we re-tested the performance of ToT on the 24-point game/sequence sorting using the `gpt-4-1106-preview` model from their official GitHub. For the MACM, the code interpreter is disabled. The results are shown in the table below:

| Method                | Accuracy (%) |
|:----------------------:|:------------:|
|  IO               |      $23$    |
|  CoT |      $25$       |
| SC-CoT  |      $34$      |
| ToT (b = 1)|     $55$       |
| ToT (b = 5)|     $78$      |
| MACM           |     $91$      |

Experimental results show that MACM still outperforms ToT. By analyzing the error cases in ToT, we found that many problems that MACM can solve but ToT cannot are due to MACM's judge correcting these errors.

---

Thank you again for your reply! And we welcome any further questions you may have.
**For *Weakness 1 (4)***:

We compared the performance of CR and MACM on the MATH and SciBench datasets and will update the corresponding experimental results in a future version of the paper. The details are as follows:

**MATH:**

Due to experimental funding limitations, we directly collected the data measured by CR on the MATH dataset from the original CR paper. The comparison results are as follows:

| Method         | Algebra | Probability | Geometry | InterAlgebra | NumTheory | Prealgebra | Precalculus | Overall |
|:----------------:|:---------:|:--------------------------:|:----------:|:----------------------:|:---------------:|:------------:|:-------------:|:---------:|
| CR       | $86.3$ | $71.1$              | $53.7$|$51.5$            | $88.7$     | $86.6$   | $51.8$   |$72.2$ |
| MACM       | $96.07$ | $97.95$               | $62.74$ | $78.43$            | $98.04$    | $94.11$  | $88.46$   | $87.92$ |

**SciBench:**

***Experiment Setup***

We used the code from CR's official GitHub to test SciBench. The testing code is located at `cumulative-reasoning/CR-Agent/infer/inference.py`. The testing format prompt was based on the prompt they used for testing the MATH dataset, which can be found in `cumulative-reasoning/CR-Agent/prompts/cr/math.md`. The `model_name_or_path` is set to `gpt-4-1106-preview`. All other parameters were kept at their default values. As in our paper, we tested its performance on the diff/stat/calculus subsets of SciBench. Some CR results are approximately equal to the correct result, we also treat these as correct. For example, if CR's final output is *approximately 0.336*, but the ground truth is $0.3359$, we treat this as correct. the comparison results are as follows.

***Experiment Results***

| Method         | diff (%)| stat (%)| calc (%)|
|:----------------:|:---------:|:--------------------------:|:----------:|
| CR       | $56.36$ | $61.05$              | $59.61$|
| MACM       | $60.98$ | $77.78$               | $76.2$ |

***Experiment Conclusion***

MACM's accuracy on the MATH dataset and the diff/stat/calculus subsets of SciBench is higher than CR.

**For *Weakness 2 (a)***:

***ToT Comparison***

***Experiment Setup***

In SSC-CoT **Section 5.1, baseline part, line 5**, they mention that the model they used is `gpt-3.5`. Since they did not specify the exact version of `gpt-3.5` used, we directly designated the model name as `gpt-3.5` for comparison. Additionally, SSC-CoT authors did not mention using a Code Interpreter for ToT experiments, so we disabled the code interpreter function in MACM for this comparison.  The result on Level 5 problems (1324 cases) is shown below.

***Experiment Results***

| Method         | Algebra | Probability | Geometry | InterAlgebra | NumTheory | Prealgebra | Precalculus | Overall |
|:----------------:|:---------:|:--------------------------:|:----------:|:----------------------:|:---------------:|:------------:|:-------------:|:---------:|
| ToT      | $27.4$ | $14.6$         | $3.8$|$3.2$          | $11.0$     | $34.7$   | $1.5$   |$15.3$ |
| MACM       | $32.9$ | $18.7$    | $5.3$ | $4.6$       | $18.2$    | $54.9$  | $4.4$   | $21.4$ |

***Experiment Conclusion***

On more challenging logical reasoning problems, such as level 5 problems in the MATH dataset, MACM performs better compared to ToT.

***GoT Comparison***

According to **SSC-CoT Section 5.1, baseline part, lines 6-12**, GoT is not tailored specifically for mathematical reasoning, which makes it difficult for us to compare it on the MATH dataset.

**For *Weakness 2 (c) 2***:

We used the following prompt to decompose the chain into steps: 

`Please help me design a plan to solve the following problem: {problem}. Your plan should be within 5 steps`
Thank you for your reply!

---

**For *Weakness 1 (1)***:

> Could you detail the difference between condition mining / claim proposal, rechecking / verification and judge / reporter in MACM / CR (Cumulative Reasoning) respectively?

***For condition mining / claim proposal***

In the first step of the **CR**, *Preliminary Contents* and *Hints* will be proposed based on the topic. These conditions will serve as the foundation for determining the subsequent steps in the later claim proposal. Details are shown in their official github: `cumulative-reasoning/CR-Agent/prompts/cr/math.md` file.

Structurally, the claim proposal of CR is as follows:

***problem*** --> ***(condition 1,...condition n)*** --> ***step 1*** --> ***step 2***... ***step n***--> ***solution***

In **MACM**, **condition mining is an ongoing process**. We do not set Preliminary Contents and Hints. In the first step, the agent needs to perform thorough condition mining for the given problem. All conditions verified to be correct will be added to our condition list. This means we **diminish the hierarchical progression** among these conditions and strive to keep them on the same level. **At this stage, we do NOT consider the specific path** to solving the particular problem but focus on **thoroughly exploring specific conditions** that might be helpful in solving the problem.

Structurally, the condition mining of MACM is as follows:

***problem*** --> ***(condition 1,...condition n)*** --> ***(condition 1,...condition n+1)*** --> ...***(condition 1,...condition n+m)***

**Note**: At this step, MACM has not yet started designing the specific steps to solve the problem.

***For rechecking / verification***

In **CR**, the main purpose of verification is to precisely check the computational process and logical derivation, as indicated in the last sentence of the second paragraph of Section 3.1 in their paper: *verifiers translate these proposals into formal systems for validation, employing either symbolic reasoning systems or incorporating a code environment*.

In **MACM**, the rechecking process involves not only **using a code interpreter for strict mathematical calculations** but also **leveraging LLM for some intuitive judgments**. For example, before adding a condition obtained during the condition mining process to the condition list, we need to determine **whether it is helpful in solving the target problem**. This is a problem that is difficult to precisely judge using strict calculations. Therefore, we hope that during the rechecking process, the LLM can help us recheck these newly generated conditions, not only verifying their mathematical accuracy using the code interpreter but also subjectively judging whether we really need this condition.

***For judge / reporter***

In **CR**, the role of the reporter is to report the final results (e.g., whether the hypothesis is correct, the specific process of the 24-point calculation, etc.), as described in Appendix A *Appendix for Prompts* of their paper.

Structurally, the reporter of CR will be responsible for this step:

***step n***--> ***solution***

In MACM, one of the important tasks of the Judge is to determine whether the current condition list is sufficient to **design a path** that can lead to the final result.

Structurally, the Judge of MACM will be responsible for this step:

***(condition 1,...condition n+m)***-->***(step 1,...step n)***

**Summary**： Compared to CR, MACM conducts condition mining in a broader thought space, which is helpful for solving relatively complex problems with a wide range of thinking (for example, solving certain problems that require the integration of knowledge from different fields, such as combining algebra and geometry)

**For *Weakness 1 (2)***:

Based on our experimental observations, voting for more complex agents can effectively improve their overall accuracy. 

The specific implementation of this can be found in supplementary material code in `main.py` at `line 74` (voting starts) and `lines 114-115` (voting summary).

To save on compute cost, we have adopted a method of voting directly on the overall result, rather than voting on each intermediate step as done in ToT.

**For *Weakness 1 (3)***:

In our experiments, the model we used is the same as in CR, which is `gpt-4-1106-preview`. This can be found in our supplementary material code in `utils/gpt_robots.py`. Thank you for your suggestion, we will clearly indicate this information in future versions of the paper.
Thank you for your reply!

---

**For *Weakness (a)***:

Sorry for any confusion caused. There are a few points we want to clarify. 

1. MACM is a multi-agent interaction framework introduced in our paper, where **each agent is played by an LLM**. The purpose of this framework is to **enhance the original LLM's ability in logical reasoning tasks**.

2. When we say: ***We utilized the GPT-4 Turbo model (between January 1, 2024, and February 1, 2024) to test MACM’s performance on the MATH dataset,*** it means that **we used GPT-4 Turbo to serve as the agent in our MACM framework**. We aim to compare our framework (MACM) with previous similar frameworks to demonstrate that our framework provides the most significant improvement to the original LLM's performance on logical reasoning tasks.

3. > Does it mean that you are using it for inference not judging?

   Yes, we are conducting inference. However, our MACM framework includes a judging step. Please note that **this judging is intended to improve the overall system's performance** and will **NOT** be used as the final criterion for determining correctness. We need to compare the results with the ground truth to determine whether they are correct.

4. >  In the Llama experiments, is GPT4 involved?

    No, In the LLaMA experiments, LLaMA will act as the agent within the MACM framework, and all decisions will be made by LLaMA.

**For *Weakness (b)***:

Thank you for your question. When you were evaluating cost, you used a zero-shot approach. Zero-shot means producing results without any prompts. Some common prompting methods include few-shots, Chain-of-Thought, Tree-of-Thought, Graph-of-Thought, etc. 

To give a simple example with few-shots, few-shots means including some similar questions as prompts before actually asking the model. **This will greatly increase the number of input tokens**, as each similar question is roughly the same length as the original question. A more complex example is Tree-of-Thought, which not only involves multi-step outputs but also requires few-shots prompting for each step. For more details, please refer to their official GitHub at `tree-of-thought-llm/src/tot/prompts/game24.py`.

Although these methods require us to provide additional prompts during inference, they offer significant advantages, such as improving the accuracy of LLMs without the need for additional training, and enabling large models to output in specific formats, making it easier for us to extract specific information.

For the statement: 

>  it takes 1 dollar to evaluate 1 question

As indicated in **the second sentence of the first paragraph in *For weakness (b)* in our rebuttal**, this cost includes the expense of testing the same problem **using different prompting methods**. Since GPT-4 is continuously being updated, we need to retest previous prompting methods for a fair comparison. For example, in our **Table 4**, we need to remeasure each problem using IO, CoT, SC-CoT, and MACM. 

In fact, the cost is more than 1 dollar per question; testing about 4,200 cases cost us more than 6,000 dollars.

> If it increases by a lot, shouldn't that be compared to majority voting in an iso-token manner?

To test the trade-off between the accuracy and the responses generated by GPT-4 Turbo, we conducted a comparison in **Figure 6**. Additionally, in our response to **reviewer 4rbe's Weakness 1 & Question 1**, we compared the average number of responses between MACM and the baseline across different problems. The details are shown in the table below:

**For MATH problem *(Our Table 1)***:

| Method         | Average responses $n$| 
|:----------------:|:-----------------:|
| CoT            | $5$               | 
| SC-CoT         | $25$              |
| CSV†     | $7.552$                 | 
| CSV-Voting†     | $47.26$          | 
| MACM           | $40.48$           | 

†: They did not provide official code implementation and the related data is not involved in their paper. The data come from our own implementation.

**For 24-point problem *(Our Table 2)***:

| Method  | Average responses $n$ | 
|:---------:|:---------------------------:|
|     CoT          |             $4.17$   |
|     ToT* ($b=1$)  |          $23.72$     |
| ToT* ($b=5$) | $67.81$                   |
| MACM (w/o code verification)  | $63.3$  | 
| MACM (w code verification)    | $22.12$ | 

**For sequence sorting *(Our Table 2)***:
| Method  | Average responses $n$| 
|:---------:|:---------------------------:|
|     GoT*  |  $54.3$                  |
| MACM (w/o code verification) | $51.72$| 
| MACM (w code verification)   | $6.882$ | 

*: The related data is not involved in their paper, we obtained the data from their official GitHub repository.

---

Thank you again for your reply! And we welcome any further questions you may have.
Thank you for the clarifying comments.

For weakness (a), I was looking at the sentence **We utilized the GPT-4 Turbo model (between January 1, 2024, and February 1, 2024) to test MACM’s performance on the MATH dataset.**
Does it mean that you are using it for inference not judging? In the Llama experiments, is GPT4 involved? At what capacity? This is very unclear to me.

For weakness (b), I disagree with the premise that it takes 1 dollar to evaluate 1 question. For a vanilla zero-shot prompting method, back-of-the-envelope calculation: if a question takes 1000 input tokens and 500 output tokens, and using gpt4-turbo ($10/M input tokens, $30/M output tokens), the cost is 1000/1e6*10 + 500/1e6*30 = 0.025. So the total cost for the MATH test set (5000 datapoints) is 125 dollars. This makes it seem like at least for the final result, the whole MATH test set should be used. For other models, it's even cheaper.

I understand MACM increases the inference-time tokens. But by how much? If it increases by a lot, shouldn't that be compared to majority voting in an iso-token manner?
Thanks a lot for the effort you put in your rebuttal, which clear up some of my doubts. However, below are my follow-up comments:

**For *Weakness 1:***

1. Could you detail the difference between condition mining / claim proposal, rechecking / verification and  judge / reporter in MACM / CR (Cumulative Reasoning) respectively?
2. Where is the implementation of the voting in MACM detailed? I am sorry to be only able to find a number of voters as 3 in lin 259.
3. The comparison between MACM and CR might be unfair for using different versions of GPT-4 (CR uses `gpt-4-1106-preview`). I notice the time span in lines 369-370 but I want to check which exact version you use. Isi it `gpt-4-0125-preview`?
4. It would be best if you could compare with CR on complex tasks like MATH/SciBench/TheoremQA.

**For *Weakness 2 (a):*** The algorithms of ToT and GoT are fundamental to compare with on complex tasks and should be applicable to tasks like MATH. SSC-CoT [1] implements ToT for MATH. And GSM8K & AQuA are quite similar to MATH in format.

**For *Weakness 2 (c):*** 

2. How you decompose the chain into steps is unclear. As suggested in ToT, "the decomposition of thoughts (e.g. is each zi a phrase, a sentence, or a paragraph) is left ambiguous".

**For *Weakness 3 (a):***  For results related to **multiple independent variables (e.g. model, prompting, code usage, task)**, I recommend dividing them into different dimensions/groups in a table or different tables/figures, which allows **easier comparison in a controlled variable way**.

**For *Weakness 3 (b):*** It is good to provide detailed examples. But Section 3.3 seems highly overlapping with Figure 3, bringing little new information.

**New questions:**

1. The exact versions of GPTs in the paper are unclear. You should at least annotate the exact versions and better compare the methods with the exact same version.

For the improvement so far, the scores in my mind should be:

Soundness: 2
Presentation: 2
Contribution: 2
Rating: 4

I will keep updating the scores as the rebuttal improves.

Again, thanks for your work and efforts!

[1] Zhao, Zilong, et al. "Stepwise Self-Consistent Mathematical Reasoning with Large Language Models." arXiv preprint arXiv:2402.17786 (2024).