MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
Authors: Bin Lei
Yi Zhang
Shan Zuo
Ali Payani
Caiwen Ding
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Agent
Prompting
LLM
Math problem
Primary area: natural_language_processing
TL;DR: This paper introduces the Multi-Agent System for conditional Mining (MACM) prompting method, which increases the GPT-4 Turbo's  accuracy on MATH from 72.78% to 87.92%.

Abstract: Recent advancements in large language models, such as GPT-4, have demonstrated remarkable capabilities in processing standard queries. Despite these advancements, their performance substantially declines in advanced mathematical problems requiring complex, multi-step logical reasoning. To enhance their inferential capabilities, current research has delved into prompting engineering, exemplified by methodologies such as the Tree of Thought and Graph of Thought.
Nonetheless, these existing approaches encounter two significant limitations. Firstly, their effectiveness in tackling complex mathematical problems is somewhat constrained. Secondly, the necessity to design distinct prompts for individual problems hampers their generalizability.
In response to these limitations, this paper introduces the Multi-Agent System for conditional Mining (MACM) prompting method. It not only resolves intricate mathematical problems but also demonstrates strong generalization capabilities across various mathematical contexts.
With the assistance of MACM, the accuracy of GPT-4 Turbo on the most challenging level five mathematical problems in the MATH dataset increase from $\mathbf{54.68\\%}  \text{ to } \mathbf{76.73\\%}$.

MACM: Utilizing a Multi-Agent System for Condition
Mining in Solving Complex Mathematical Problems
Bin Lei Yi Zhang
Ali Payani University of Connecticut (lei00126, dingc)@umn. edu (bin.lei, yi.2. 2.ang,, shan. zuo) Ouconn. edu {apayani)@cisco.. com
Shan Zuo
Caiwen Ding' *CiscoResearch
University of Minnesota
Abstract
Recent advancements in large language models, such as GPT-4, have demonstrated remarkable capabilities in processing standard queries. Despite these advancements, their performance substantially declines in advanced mathematical problems requiring complex, multi-step logical reasoning. To enhance their inferential capabilities, current research has delved into prompting engineering. exemplified by methodologies such as the Tree of Thought and Graph of Thought. Nonethe- less, these existing approaches encounter two sigmificant limitations. Firstly, their effectiveness in tackling complex mathematical problems is somewhat constrained. Secondly, the necessity to design distinct prompts for individual problems hampers their generalizability. In response to these limitations, this paper introduces the Multi-Agent System for conditional Mining (MACM) prompting method. not only resolves intricate mathematical problems but also demonstrates strong gen- eralization capabilities across various mathematical contexts. With the assistance of MACM, the accuracy of GPT-4 Turbo the most challenging level five math- ematical problems in the MATH dataset increase from 54. 68% to 76.73%. The code is available in https: http:::///gttbb com/ wwamm/inniee Maan1iapppee/MMM.
Introduction
Large language models (LLMs) like GPT-4 [12] excel in various problem-solving tasks but still struggle with complex logical deduction, especially in mathematics involving abstract concepts and multi-step reasoning [11, 2]. This limitation hinders their accuracy and reliability in fields requiring precise mathematical reasoning, such as academic research, engineering, and theoretical physics.
contemporary and efficacious method for tackling this issue is the prompting engineering [19]. enhances accuracy in complex problem-solving without necessitating further training of the model. By strategically crafting prompts, prompting engineering optimizes the utilization large language models, guiding their processing pathways more efficiently and effectively [10].
Previous prompting methods mainly include the Chain of Thought (CoT) [18], Self-consistency Chain of Thought (SC-CoT) [17], Tree of Thought (ToT) [20], and Graph of Thought (GoT) [4, 9]. CoT and SC-CoT show limited capabilities in complex logical reasoning, achieving only 4.0% and 9.0% accuracy in simple tasks like the 24-point game using GPT-4 [20]. While ToT and GoT have improved LLMs' groblem-solving abilities, they lack generalizability, requiring specifie prompts for each problem, detailed in Appendix A.
To address two key issues:
1. The insufficient reasoning capability of LLMs for complex mathematical problems.
2. The inadequate generalizability of current prompting methods.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Method
Accuracy Universality
I-0
Few-Shots
CoT
A
I-0
CoT
SC-CoT
ToT
GoT
SC-CoT
Step
Step
Previous Methods;
Manual-CoT
&
Our Method
Auto-CoT
Question
Step
Thought
ToT
A Answer
GoT
Conditon
MACM
Objective
MACM
Figure 1: The comparison between the current mainstream prompting methods and MACM.
We propose the Multi-Agent System for Condition Mining (MACM) prompting method. MACM has moved beyond being restricted by the specific contents of problem. Instead, it first abstracts conditions and the objective of the problem. Subsequently, through a Multi-Agent interactive system, progressively mines conditions conducive achieving the objective, thereby ultimately fulfilling the goal of problem-solye solving..
The comparison between MACM and current mainstream prompting methods in problem-solving is shown in Figure MACM extracts conditions and objectives from each math problem, iteratively adding new insights until enough information is gathered to find a solution. Performance-wee wwie, MACM improves accuracy by over 10 percentage points on datasets like the 24-point game, matching the effectiveness of ToT and GoT. Moreover, MACM is versatile; it can apply the same set of prompts to various mathematical reasoning problems without manual modifications, unlike the tailored prompts needed for ToT and GoT.
Through several experiments on the math related datasets, we verified MACM's generalizability and superior error correction compared to original prompting methods. With MACM, the GPT-4 turbo model's accuracy on the MATH dataset increased by 15. 14.14% and by 7.8% compared to SC-CoT, In the 24 14-nt game, MACM achieved an accuracy rate 1 17% higher than ToT.
2 Related Work
In this section, we summarize several major current prompting methods.
Input-Output (I-O) prompting is the most common method for interacting with large language models, where users specify the problem conditions directly the model, which generates responses through token-level, sequential decision -aaiing process [22].
I-0 Prompting:
CoT Prompting: [18]: Chain of Thought (CoT) prompting refines the model's output into more structured and logically coherent text by methodically constructing and elaborating upon chains of reasoning. This approach enhances the model's ability to produce outputs rooted in logical deduction. There are several variants of CoT,, including Zero-Shot CoT [8], Few-Shot CoT [7], and Auto-CoT [22], each tailored different prompting scenarios and requirements to improve logical reasoning in diverse contexts.
SC-CoT Prompting: [17]: Self-Consistency Chain of Thought (SC-CoT) prompting improves upon CoT by introducing voting mechanism, which emphasizes internal consistency and semantic interconmectednnss. In this method, models evaluate (vote on) their outputs to select the most coherent response, thereby reducing logical fallacies and inconsistencies.
ToT Prompting: [20]: Tree of Thought (ToT) prompting uses hierarchical, tree-like structure to organize and guide the model's 's text generation. This method improves precision and structure in responses, incorporating voting mechanism to refine outcomes and reduce computational demands.

---

GoT Prompting: [4. 9]: Graph of Thought (GoT) prompting enhances ToT by allowing interconnec- tions between thoughts on different branches. decomposes complex tasks into simpler sub-tasks. solves them independently, and merges the results, thus reducing computational costs.
3
Method
3.1 MACM Overall Structure
The overall structure of MACM is shown in Figure 2. We have designed an interactive system
Can
elif
yn
x Back to 2
Figure 2: The overall structure of MACM. Original Math problem: [_ Condition list; True;
False: M:
Discard; Known Conditions; New Conditions; Objective; Thinker;
Judge; Executor; @: Initialize the initial condition list and the objective; @: Explore new
Conditions based on current condition list; 3: Check if the new condition is correct; @: Check if the objective can be achieved based on the current Conditions in the Condition list; @: Designing steps for achieving the objective based on current Conditions; @: Achieve the objective.
comprising three agents: Thinker, Judge, and Executor solve complex mathematical problems.
Thinker: Responsible for generating new thoughts or ideas. This role involves creative thinking and the generation of novel solutions or approaches to problems.
Judge: Evaluate the thoughts generated by the Thinker. It assesses the viability and correctness of new ideas, ensuring that only the most logical and beneficial ones are pursued.
Executor: Performs calculations or actions according to predefined steps. is focused on the implementation of the ideas approved by the Judge, turning steps into tangible outcomes.
When a mathematical problem is input into our system, the Thinker initially sets up the Condition List and defines the final Objective based the given problem. After initialization, the Thinker mines new conditions conducive to the objective from the current Condition List, i.e., the Known Conditions. The Judge then assesses these newly mined conditions. If deemed correct, the Judge incorporates the new condition into the Condition List. Otherwise, the new condition is discarded.
Once all new Conditions have been reviewed. obtain revised Condition List. At this point, the Judge evaluates whether the current conditions are sufficient to achieve the objective. If the answer is False, the process reverts to step @ for further mining of new conditions. In our experiments, set a limit of five iterations; if the objective is not met after five rounds of mining, we consider the problem unsolvable. This prevents the program from entering an infinite loop. If the answer is True, the Thinker designs steps based on the Known Conditions to achieve the Objective. Finally, the Executor performs calculations following these steps to produce the final result.
MACM achieves a high level of generalizability by abstracting conditions and objectives from each specific mathematical problem. Through multi-agent interactive system, where the Thinker is responsible for idcation and design, the Judge for inspection and decision-makinn and the Executor for computation, most potential errors in reasoning and calculation are eliminated. By repeatedly mining for conditions and adding the correct ones to the Condition List, MACM ensures depth in thinking, making it suitable for analyzing complex mathematical problems.
3.2 Theoretical Analysis
MACM moves away from the hierarchical dependencies of previous methods by introducing Condi- tions and Objectives. It continuously expands Known conditions to derive the final answer, eliminating

---

the need for manual, problem problei--pecific prompts. This method compresses information from various Thoughts into existing Condition List. capturing more connections than traditional prompting methods that rely navigating a hierarchical structure.
The Thinker initiates a thought set Ti TT.,TP T....... .T contains
thoughts from question Q and generates subsequent thought sets T. ? ( , based on the current condition list C; {Ci, C2,.... C Each condition C₁-1 is derived from the most accurate thought T in T:-1. At each step i, the Judge selects the correct thought T," in thought set T; such that T arg max, p.ludes (Tī [T, E T {L,... ..)), where pludge is the probability that the Judge confirms the thought as correct. By using this method, we map the whole thoughts space T to the Condition List In ideal situation where pludge + 1, we have T - The probability of arriving at the correct answer Aconect based on the final Condition list C is equal to that based on the entire thoughts space T. Thus we have: (MACM A (C) = MMACM(Aamme (T) > POOT, ToT, CST coT(Acome (Acorrett (Th = 1, 2, .... and = 1, 2,2,... ....... , C T). In practice, where pludge A/ 1, we performed the experiments to test its performance, the results are shown in Section 4.
3.3 Using Cases
Our prompts and use cases are shown in Figure 3. It demonstrates the specific process of MACM analyzing algebra and geometry problems. In these two examples, we have employed OpenAl's GPT-4 Turbo [1] as the intelligent agent, which is capable of performing calculations using code. endowed with three roles: Thinker, Judge, and Executor by using the following instructions:
For Thinker: You take the role of Thinker. I need you help me gradually ponder over some problems following my instructions. You need to answer the question by using the following format: Based Condition and Condition B, we can get: C.
For Judge: You take the role of a Judge. need you to make judgments on some statements. You only allowed to use the True or False the final answer.
For Executor: You take the role of a Executor. need you to calculate the final result based on the given conditions and steps.
In the first algebra problem: Let S be the set of all real numbers such that the function 2"4Te-11 can be expressed quotient of two linear functions. What is the sum of the elements of S?
2245240
GPT-4 Turbo's 's raw response reached an incorrect conclusion: x + 5x a k(r 11)(x - 4)
which then led to issues in the subsequent code design, ultimately resulting in an incorrect output.
In the MACM analysis process, the Thinker initially identifies conditions and objectives from the problem statement and then uncovers new conditions. Although the Thinker initially identifies the same incorrect condition GPT-4 Turbo, the Judge detects and rejects this error, preventing its addition to the Known Conditions. In the second round, the Thinker identifies two new conditions:
(-11)² 5 (-11) + a1 0) and (4)2 + (4) 02 = 0 which the Judge verifies and adds to the Known Conditions list. The Judge then confirms that the Known Conditions are sufficient to achieve the objective. The Thinker designs steps to reach the objectives based these conditions, and finally, the Executor performs the necessary calculations to produce the result.
In the second geometry problem:
Square ABCD has side lengths of 13 units. Point E lies in the interior of the square such that AF units and BF 12 units. What is the distance from side AD?
While GPT-4 Turbo's 's response had the correct theoretical approach, failed identify relationships between points in the problem, leading incorrect expressions and incorrect final result.

---

Question (Algebra): Let be the set of real numbers such that the function s'stesa can be expressed as quociost of
Ruselon (soometry): Square
Primpe
12y
Frongt
ADOD aas vide kmnghe of 12
AS 5 and IrE 12. What te
To find the set of all real numbers such that the function plate can he quetient of two Linuar functions to find conditions under which the suserator and the denominator are propartional to eath octer.
two lincar functions. What the
2
states that in ructangle, If point P is chosen tnxide it, then the sue of the squares of the distances froe to two oggasste connens is equsl to the sou of the squares distances froe P to the other opposite square ABCD leagtts inside
Flag theoree, this theerws
the British
the distanse frome & side AD
<he elomenta of S?
expressed
reed
Conditions:
Pronets
1. ADCD is square. 2. The sides of the square each have lenges of 13 units. 5. Paint E is inside the square. d. AB - units. S.
Conditio::
1. The fanction Z 2-1--4 be
the denuminater can be foctored as follows
Linsar functions.
is the of
e" Ta - 46 = (e 1ijsa 4)
quotiont two Linear functions, the
Now, the function espressible
the set S.
all rmel numbers
calculace the sux of the elements of
jecttiie::
AB untts and BB units. want find the distance fron 8 to side AD, Let be the distance from F to vice AD The divcance from B to sidw BC is also x, using the Brtcish Flng Thearen:
Calculate distance to
dbjective:
= 12 units.
the square.
Prompe s
Frongt
(a 11)fz there exists constant such that of + fr
condition:
11)
right t-iangle
condition:
k/ U1](s 44) Exponding the right hand ssde, ve get
Frompt
Prompe
(MM New Condition to the
Axt + Rt1l - djz - 46k Equating the confficients of like teres on both sides, we get 1- - k(11- - 4),0 - -6th can solw these value couations subsequently
The New Condition, pis reiatal
we can solue this equation find
AB + BC BB2 + ED'
Prompt
known conditioed Lina.
Promet-4
(Back medttos
condit acarch slop?
(Bock to the new
Poont
snarch step)
Fa
Promge
Now concition:
[11) 5x +a 10+
Froe syepy import symbols,
To, salue, sgrt
sysbuls('s) ('x')
Froppt
J.
MP AB
s15e_ Lenghh, AE, BE
New Gondition the! known condittion)
(Tron sy0ey Irport symbols, k- syetols("k") kk)
Prompe 3
13, 5, 12
sgra(side Jength+
Frompt
(Statt to doign
Tru
equrtions tu(3, k)
equation2 Fa(5. (11
4rompt.
(Add New Condition the Truc
Ansnn conditions)
(Start to design the relition stape)
squation EQ(AE"12 (AC**2
iide 1ength""2]
x*),, BE X*12)
Primpe 5
First calculats BY based the known
dhe soltioonstes)
k-veloe solw iequationn, k)
solvetequation, posdistance_
condititon. Than use
aipra_value Volue
Prampts
Calculate d and sie the
Pythagorcan thooren to
bosalihh_ value
Fronyt
The anoer -4 X
Prompta
The arswer 15 1002
(-VE18/2,
The 1.923
MALMI Aadyyss
Andrets
MVOM Correct Ammbyls
Please help ne to solve this questioo: (Question) Prompto
@Dudge: have sone known conditions:
Uaer
@Thinker:Help analyze the conditions and the objective of problen. You don't 't need to provide with solution for the time being.
My objective (Osjective) Aelp no to detersine if the objective can obtained by using current' conditions.
(50 Original LLA
Question: (Question) Prompt
Prompt
@Thinker:Derive the nost
Julge
direct condition with logical relationships, based known conditions. You NOT the conditions
@Thinker:Holp to cone up with one solution targeting at our objective Known conditions: konown_condttions objective: (Objective) Prompt
that not directly shown In the known tondstions.
Problen
Area
Known conditions (Known_comm conditions)
LLN''s
objective: [Objective) Prompta
@cxecutor: need you to calculate the final objective based on sone steps{Steps)
MACM
@Jddee no to carafully varify if the staterent correct. Known
Analysis area
Prorett Area
conditions (Known_ conditions)
Our objoctive iss: (0ojectiive
Staterent: [New_Cond Conditiion) 3
have: :ove:iconttiing } Prompt
Analysts
Code
Aree
Figure 3: MACM's detailed analysis process for complex mathematical problems with specific prompts, illustrated with an algebra problem (on the left) and geometry problem (on the right). We use one set of prompts that can target different types of problems, with prompts 0-6 displayed in the below the dialogue box. In these examples, MACM involves three steps: 1. Extracting conditions and the objective. 2. Iteratively identifying new conditions. 3. Solve the problem based on known conditions.
During the MACM analysis, the Thinker first clarifies the conditions and objectives of the ge- ometry problem and then uncovers new conditions to achieve the goal. Initially, it discovers that AABE is right triangle. After verification by the Judge, this condition is added to the known conditions. The Judge then checks if the known conditions sufficient. A False result means more conditions are needed, so the Thinker continues searching. In the second round, the Thinker deduces AE EB EF x AB which the Judge verifies and adds to the Condition List.
5

---

Upon confirming sufficiency, the Thinker plans the steps to solve the problem, and the Executor performs the calculations to find the result.
In analyzing these two problems, MACM first extracts the specific conditions and objectives from the questions. This allows MACM to directly use these conditions and objectives for prompt design in subsequent processes, enhancing our approach's generalizability. Previous methods like ToT and GoT lack this setup, resulting in poorer generalizability. For example, in the 24-point game experiment with ToT, the lack of this setting necessitated the manual configuration of the following prompt:
Evaluate if given numbers can reach 24 \n (sure/tikellimpossiibe) \n (input)
In MACM, the given numbers can reach 24 is obtained by the first step and the evaluation prompt is generalized to
Evaluate (objective] [inpur] ensuring higher generalizability.
4 Experiment
4.1
Performance on MATH benchmark
The MATH dataset [6] includes a variety of mathematical problems. offers seven types of mathemat- ical problems, including geometry, algebra, probability theory, etc.. with difficulty levels ranging from 1 to 5. We first tested the overall performance of MACM on the MATH dataset without distinguishing difficulty levels. Afterward, we specifically selected the most difficult mathematical problems from the MATH dataset for testing. The detailed experimental setup is presented in the Appendix B.
Table I: Accuracy (%) of GPT-4 Turbo on MATH dataset with different prompting methods.
Counting and Probahility
Intermediate Algehra
Number Therogy
Algebra
Geometry
Prealgebra
Precaleulus
Overall
I-O)
88.24
81.63
45.11
66.67
74.51
81.82
71.15
72.78
CoT
92.99
83,67
42.02
68.07
77.31
82.07
74.18
74.36
SC-C6T
94.96
87.17
50.14
71.99
89.91
86.75
79.67
80.12
CSV [23]
86.9
77.3
54.0
56.6
85.6
86.5
$3.9
73.54
CSV + Voring [23]
95.6
89.0
64.9
74.4
94.1
91.6
67.8
84.32
MACM
96.07
97.95
62.74
78.43
98,04
94.11
88.46
87.92
In Table 1, we compared the accuracy of GPT-4 Turbo on the MATH dataset with various prompting methods. We found that compared to the original GPT-4 Turbo, MACM increased its accuracy by 20%. Compared to CoT, the increase was 13.56%, and compared SC-CoT, it was 7.8%. Among these, MACM led to the greatest improvement in accuracy for the original GPT-4 Turbo model on number theory problems, at 23.53%. In geometry problems, although MACM has increased the accuracy of GPT-4 Turbo by 17.63%, the final accuracy rate is still only 62.74%. Upon analyzing the causes of errors, we found that many mistakes were due to GPT-4 Turbo's 's difficulty in accurately understanding the relationships between various geometric figures, thereby failing to design corresponding code to solve the problems. However, in algebra and number theory problems, MACM, by correcting the erroneous analysis of GPT-4 Turbo and helping it explore potential approaches, achieved accuracy rates of 96.07% and 98.049 04,, respectively. Moreover, compared to the previous SOTA method, CSV prompting, on the MATH dataset, MACM achieves 3.6 percentage points higher accuracy rate on the same dataset. This demonstrates the effectiveness of MACM solving mathematical problems.
In Figure 4, we tested the performance of MACM on the open-source LLLMM series models on the MATH dataset and compared it with other prompting methods. Since LLaMA models do not have a code interpreter like GPT-4 Turbo, we disabled the code-checking function of MACM in this group of tests. The rest of the experimental setup was consistent with GPT-Turbo. In zero-shot scenarios, the accuracy rates of LLaMA 7B and LLaMA 13B were both below 5% [14]. Majority voting could enhance their accuracies to 6.9% and 8.8% respectively [14], while MACM further increased them to 9.5% and 10.2%. On LLaMA 2 [15] and LLaMA 3 [3], compared to shotss, MACM could further improve the accuracy on the MATH dataset by 3-5 percentage points. Overall, We found that MACM can also be applied to LLaMA models, although the performance improvement was not as significant as with GPT-4 Turbo. This because GPT-4 Turbo has a better understanding of MACM's intrinsic directive prompts, enabling it to find the correct results more effectively.

---

Accuracy Comparise LLaMA Noddls on MATH Dataset
Gerpary
LLaMA
52.05
30
LLaNAZ
30%
LLaMA3-Inttrut
Intereslica
Seding HEJBS
SouTh Mate
14.6%
9.5%%
10.2%
Thureky
st.0%
4
2.9%
3.9%
Prinkoto
128
MaM
M8
MAGM
MACH
118
6-thit
70
1.30
MACN
MACH
130
4-tbot
MACH
Figure 5: GPT-Turbo's perfor- mance on MATH dataset Level problems with/without MACM.
Figure Accuracy comparison of LLaMA models on MATH
dataset with different methods. Maj-V V.: Majority Voting,
In Figure 5, We focused on the ability of MACM to solve the Level 5 mathematics problems in MATH. As shown in the figure, MACM improved the accuracy of GPT-4 Turbo in all seven categories of level 5 problems. The two types of problems that saw the most significant improvement with MACM were the very categories where the original GPT-4 Turbo performed the worst: Geometry and Intermediate Algebra. The original GPT-4 Turbo had an accuracy rate of only 18.18% on Geometry problems and 34.04% Intermediate Algebra problems. With the support of MACM, it's accuracy rate in Geometry problems increased to 50.0%, and Intermediate Algebra problems, increased to 65.96%. This demonstrates MACM's effectiveness in solving diffieult mathematical problems.
4.2 Comparison with ToT and GoT
Due to the lack generalization of ToT and GoT prompting methods (See Appendix A for the reason), were unable to test them the MATH benchmark. To compare MACM with them, selected two mathematical problems where their methods are applicable: the 24-point game and sequence sorting. Among these, ToT tested the 24-point game, while GoT studied the sequence sorting problem. The detailed experimental setup is presented in the Appendix B.
Table 2: Accuracy (%) comparison of different methods on various tasks.
Code
Task
Model
Method
Accuracy (%)
Verification
X
GPT-4
10
7.3
X
GPT-4
CoT [20]
4
X
GPT-4
SC-CoT [20]
X
GPT44
ToT (b [20]
45
24-points game
X
GPT-4
ToT (b = 5) [20]
74
X
GPT-3.5
MACM
67
X
GPT-4
MACM
91
GPT-4 Turbo
MACM
99
X
GPT-3.5
GoT [4]
89.06
Sequence sorting
X
GPT-3.5
MACM
92
(64 elements)
GPTT4 Turbo
MACM
100
In Table 2, We compared MACM with 1O, COT, SC-CoT, and ToT models on the 24-point game. When the model GPT-4, MACM 17% higher than ToT (b 5). Note that here, ensure fair comparison, we used the standard GPT-4 without any code capabilities. Additionally, with the support of MACM, GPT-3.5 also achieved an accuracy of 67% the 24-point game, which higher than the GPT-4 model with ToT (b- 1) support. Upon analyzing the reasons for the improvement in accuracy, we found that MACM's s Judge corrected many thoughts that were mistakenly evaluated in ToT, leading to GPT-4 choosing incorrect approaches. This correction process significantly contributed to increase in accuracy. addition, We compared GPT-3.5 model's ability sort 64 numbers using GoT and MACM. MACM outperformed GoT by 2.94%. Note that some results marked with * were estimated from graphs without specific data. Additionally, GPT-4 Turbo achieved 100% accuracy the Sequence Sorting task due its problem- code construction capability.

---

4.3 Performance on other datasets
This section primarily tests two capabilities of MACM:
1. The ability to solve more challenging mathematical problems. We applied MACM to two datasets, SciBench [16] and TheoremQA [5], which claim their difficulty surpasses the middle school level of the MATH dataset, reaching the Undergraduate Level.
2. Transferability to General Logic Reasoning Tasks. Although MACM focuses solving mathematical problems, to test its applicability, we applied it to the Reclor logic reasoning dataset [21].
Tahle 3: Accuracy (%) comparison of different methods the math-related
Table 4: TheoremQA re-
subset of the SciRench dataset. Except for MACM, all results from [16].
sults with GPT-4 Turbo.
Method
diff (%)
stat (%)
calc (%)
Method
Acc. (%)
GPT-4
32
49.33
54.76
O-shot 46,45
GPT-4 CoT
22
50.67
42.86
CoT 41.18
GPT-4 + Few-shots CoT
30
49.33
45.24
PoT 55,88
GPT-4 + Few-shots + Python
44
68
38.1
MACM
79.41
GPT-4 Turbo
46
61.33
52.38
GPT-4 Turbo CoT
38
64
54.76
Table S: Reelor results.
GPT-4 Turbo Few-shots + CoT
32
65.33
50
Method
Acc. (%)
GPT-4 Turbo + Few-shots+ Python
34
42.67
30.95
O-shots
88.1
GPT-4 MACM
48.78
69.44
64.29
CoT 88.5
GPT-4 Turbo MACM
60.98
77.78
76.2
MACM 88.9
Table 3 displays the testing results of MACM the Scißench dataset. The SciBench dataset includes problems in chemistry, physics, and mathematics. We only selected the math-related subset for testing, which includes: the diff. stat and cale subset. The experimental setup was consistent with the lesting on the MATH dataset (as shown in Appendix B). The results demonstrate that, in contrast to the Chain of Thought (CoT) method, which resulted decreased accuracy for both GPT-4 and G Turbo on this dataset, MACM led to an approximate 20 percentage points.
Table 4 presents the performance of -PT-4 Turbo the TheoremQA dataset using various prompting methods. The TheoremQA dataset encompasses problems from multiple domains including mathe- matics, physics, finance, and computer science. Notably, its mathematics subset contains numerous conceptual and definitional questions that do not involve logic reasoning processes; thus, these were not considered in our experiments. We solely tested questions within the TheoremQA mathematics subset that involve logic reasoning and have definite answers (c.g., specific number, rather than an interpretation of a theorem). The experimental setup was consistent with the testing on the MATH dataset (as detailed in Appendix B). The results indicate that MACM enabled a roughly 30 percentage points increase in accuracy for GPT-4 Turbo on this subset.
Table 5 shows the test results GPT-4o [13] on the Reclor dataset using various prompting methods. For general logical reasoning problems, we adjusted the computation-related prompts in the original MACM to make them suitable for non-mathematical logical reasoning problems, while maintaining the overall structure consistent. Results show that MACM can also improve the accuracy of general logic reasoning tasks, but the increase is smaller compared to math tasks.
4.4 Ablation Study
In this section, we primarily investigate two issues. ① Explore the relationship between MACM Accuracy and LLM Queries, comparing it with other methods. @ Analyze the proportional impact of each component within the MACM on the overall accuracy improvement. We performed these two experiments on 200 randomly selected questions from the MATH dataset that the original GPT-4 Turbo model answered incorrectly.
Trade-off Between Accuracy and LLM Queries: In general, increasing the number of responses generated by LLMS leads to an improvement in accuracy. Each prompting method has parameters that can increase it, such as the length of the chain / in CoT,, the number of voters in SC-Co'T, and the Tree breadth à in ToT. To measure the search efficiency of each method, we compared the relationship between the accuracy and the number of responses generated by GPT-4 Turbo.
8

---

We increased the number of answers generated by various prompting methods. For I-O prompting. we directly adjusted the model's response generation parameter n, which enables the model to responses. For CoT, we adjusted not only the parameter but also the length I of the Chain. For SC-CoT, we built on the first two methods by adding an adjustment to the number of voters v.
50
0.8
I-O
Oni component companents
CoT
4242
40
Three components
SC-CoT
Problems Solved
four componintss
35335
Accuracy
0.6
MACM
30
3131
0.4
14
1212
0.2
0.0
20
40 60
100
Responses generated GPT-4 Turbo
Combinations Components
Figure 6: The trade-off between the accuracy and the responses generated by GPT-4 turbo. Com- pared I-O, CoT, and SC-CoT, MACM has stronger error correction capabilities when the GPT-4 Turbo generates more responses.
Figure 7: Effectiveness of Component Combi- nations in MACM. Red: Voting; Blue: Self- Multi-Agents;; Yellow: Condi-
Checking: Green:
tion Mining; A circle containing multiple colors: combinations of different components.
As shown in Figure 6, although I-O, CoT, and SC-CoT only require simple queries to correct the original errors made by GPT-4 Turbo, their upper limits are not high. Even if we continue to increase the number of queries, they only correct about 20% of the original errors made by GPT-4 Turbo. In contrast, MACM can correct nearly 90% of the original errors of GPT-4 Turbo when the number of queries is high. This is actually quite reasonable because the MACM structure is more complex, including multiple processes of mining conditions and checking. These processes allow the large model gradually think and identify errors, thus significantly improving accuracy.
Proportional Impact of Various Components: To analyze the functions of each component in MACM, we randomly combined the four components within MACM: Condition Mining, Multi- Agent System, Self- Self-Checking, and Voting, and tested their performance. During the experiment, we maintained maximum of 2 Condition Mining iterations and used 3 Voters.
As shown in Figure the combination of all four components yields the best performance. Among the individual components, Multi-Agent and Condition Mining have comparable error correction capabilities. In the combinations of two components, the pairing of Self Self-Checking and Condition Mining shows the best performance. Among the threeccomponent combinations, the combination of Multi- Multi-Agents, Condition Mining, and either Voting Self Self-Checking achieves better results.
Conclusion
We introduce MACM, new and generalizable prompting technique that significantly enhances the inferential capabilities of large language models on mathematical problems. MACM can be applied different types of mathematical questions. Through comparisons in several experiments on the math related datasets, we have verified the superiority of our method over the original prompting methods. With the aid of MACM, the accuracy of the GPT-4 Turbo model on the MATH dataset has increased by 15. 14... Compared to SC-CoT, its accuracy has increased by 7.8%. For the most challenging level mathematical problems in the MATH dataset, its accuracy increased from 54.68% to 76.73%. In the game of 24 points, using the same GPT-4 model, MACM's 's accuracy is 17% higher than that of ToT. At the same time, by comparing accuracy with the number of times the large model responds, we find that MACM has a higher limit; increasing the number of responses from the large model can significantly improve accuracy. These experiments demonstrate MACM's generalizability and its powerful coror-corection capability for complex mathematical problems in original LLMs.

---

Acknowledgement
This work was supported in part by a research grant from Cisco Research and by an Amazon Research Award.
References
[1] Gpt-4 turbo. https: opeaae httpa///////aaattt///////5555111 4- gpt-4-turbo, 2024.
[2] Katherine Abramski, Salvatore Citraro, Luigi Lombardi, Giulio Rossetti, and Massimo Stella. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety high-school students. Big Data and Cognitive Computing, 7(3):124, 2023.
[3] Meta AL. Introducing meta Ilama 3: The most capable openly available Ilm date, April 2024. Accessed: 2024-05-11.
[4] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiiv:2308. 09687, 2023.
[5] Wenhu Chen, Ming Yin. Max Ku, Pan Lu, Yixin Wan, Xueguang Ma. Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem- question answering dataset. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023.
[6] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
[7] Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thoouhh fine-tuning. arXiv preprint arXiv:2305. 14045, 2023.
[8] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-222133 2022.
[9] Bin Lei, Chunhua Liao, Caiwen Ding, et al. Boosting logical reasoning in large language models through new framework: The graph of thought. arXiv preprint arXiv:2308.085 08614, 2023.
[10] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[11] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems. arXiv preprint 2rXiv:2303.13375 2023.
[12] OpenAl. Gpt-4.
2024. Accessed: 2024-02-01.
[13] OpenAL. Hello gpt-4. https://opene httppeaai..cm//n/d//e/eeeeaeeeea goteioo-pppoo///oo/ 2024.
[14] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977, 2023.
[15] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint
Stt::2307.09288 2023.
[16] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307. 10635, 2023.
10

---

[17] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. //711, 2022.
[18] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoon of-thhthht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
[19] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt, A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302. 1382, 2023.
[20] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
[21] Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. Reclor: A reading comprehension dataset requiring logical reasoning. arXiv preprint Stti::200020033266 2020.
[22] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint mxXiv2211003399.) 2022.
[23] Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with bose- self-verification. arXiv preprint wrXVv:2308.0721,, 2023.

---

A Why is the generalizability of ToT and GoT limited
This section demonstrates specific examples of using ToT and GoT further illustrate why their generalizability is limited.
ToT conducted three sets of experiments in the original study, requiring specially designed prompts for each. The official implementation on their GitHub page http://ituub.oo//piinntttt -llp//ree--- of-thought- wtthought-Ilm/rre//mete http://e//masseerr////o////oeooe
includes the specific prompts set up for each experiment. Taking the 24- point game as an example, specific prompts such as propose prompt, value prompt. and value last step prompt executes traversal searches, voting, filtering, etc., based on these written prompts. The authors also mention in the ToT readme section How to Add New Task (htps://githu..com///)) wtt.:///o//ccncooon///eeee wwtee-oovoiiietoow----ooo-------
required (lines 51 to 134 in game24.py). During ToT's operation, the LLM
that setting up task-specific prompts is necessary for different problems, further illustrating the limited generalizability of ToT and GoT due to the need for task-specific prompt engineering.
GoT faces the same issue, with their original paper conducting experiments in four tasks: Sorting, Set Operations, Keyword Counting, and Document Merging. For each type of problem, specific prompts must be set up their official GitHub. Taking Sorting http://itttucooomp/////raaaa
example, the specific prompts for sorting are displayed in pttp://eaaaeeppppppssse prompts_sorting 032.md. They provide the LLM with the instruction: Split the following list of 32 numbers into lists of 16 numbers each, the first list should contain the first 16 numbers and the second list the second 16 numbers. Only output the final 2 lists in the following format without any additional text or thoughts! This instruction is clearly tailored this specific problem, illustrating the limited generalizability of GoT due to the necessity for problem-specific prompt engineering, similar to ToT.
Take ToT as example, they tested three tasks, for the game of 24 task, the propose prompt is:
Input: 2 8 In Possible next steps: In 2 + 8 10 (left: 8 10 14) 8/2 = (left: 48 14) in 14 = 16 (left: 8 16) In 2 8 = 16 (left: 8 14 16) \n 8-2 = (left: 68 8 14) \n 14-8 8 = (left: 2 6 8) 14 / 2 = 7 (left: 7 8) 14 = 12 (left: 8 12) \n Input: [input] \n Possible
next steps:
for the cross word task, the propose prompt is:
Let's 's play a x mini crossword, where each word should have exactly 5 letters. In (input] 'n Given the current status, list all possible answers for unfilled changed words, and your confidence levels (ceruin/high/medduu/loo "certain" cautiously and only when you are 100% sure this is the correct word. You can list more than one possible answer for each word.
using the formal "kl. apple (medium)". Use
for the creative writing task, the propose prompt is:
Write a coherent passage of 4 short paragraphs. The end sentence of each paragraph must be: [input] \n Make plan then write. Your output should be of the following format: \n Plan: In Your plan here. Passage: \n Your passage here.
Each time the problem changed, both ToT and GoT require an update to their respective prompts. The requirement to tailor prompts for each specific problem limits the generalizability of ToT and GoT to broader issues. MACM successfully addresses this challenge.
B Experimental Setup
For the experiments on the MATH dataset: We utilized the GPT-4 Turbo model (between January
1, 2024, and February 1, 2024) test. MACM's performance on the MATH dataset. For tests that did not distinguish by difficulty, we randomly selected one-third of the questions from the MATH dataset
12

---

for evaluation. For the high- high-difficulty tests, we extracted all questions with a difficulty level of 5 and randomly selected half of the questions from each category for testing. The experiment are performed by using I-O, CoT, SC-CoT, and MACM methodologies. For all prompting methods, we standardized the number of responses n generated by GPT-4 Turbo to 1, Top: = and the temperature For CoT,, we set the maximum length of the chain 1 = 5, for SC-CoT, the number of voters v = 5, and the maximum length the chain / = 5. For these three methods, we consistently maintained max_ tok_eooeessa at 512. For MACM, we kept the thinker's max_ tokens at 512, the judge's max fox__toees at 4, and the executor's max max_tokeens at 256.
For the 24-point game experiment: We sourced data from 4nums.com, which offers 1,362 games ranked from easy hard based the time it takes humans solve them. We focused on subset of these games, specifically those ranked 901 to I 1,000 (The same as ToT), to test on relatively difficult challenges. Success for each task is defined as producing a valid equation that results in 24, utilizing each of the input numbers exactly once. The performance metric is the success rate across these 100 challenging games. We utilized the GPT-4 and GPT-3.5 model (between January 1, 2024, and February 1, 2024) to perform the experiments. The MACM configuration for this experiment includes setting the number of responses generated by the model n = 1, T Top: = 1, temperature = O. with the thinker's max max_tokens at 512, the judge's 's max_tokens at 4, and the executor's max_ max__okens at 256.
For the sequence sorting experiment: We randomly generated 100 sequences, each containing 64 elements, for testing. We utilized the GPT-3.5 model (between January 1, 2024, and February 1, 2024) to perform the experiments. The MACM configuration for this experiment includes setting the number of responses generated by the model n = 1, Tops 1, temperature / = O, with the thinker's max_ max_tokens at 512, the judge's max_ max_tokens at 4, and the executor's max max__ookes 256.
C Limitation and Discussion
While MACM significantly enhances the accuracy of large language models in tackling complex mathematical challenges, it incurs the cost of multiple invocations of the large language model for inference, leading to increased problem-solving time. Additionally, MATH dataset indicate limitations effectively addressing geometry problems. Addressing these challenges necessitates further advancements in the model's own cognitive capabilities. A proposed strategy involves employing prompting methods like MACM to assist the LLM in eliminating incorrect responses. This approach enables the creation of expansive, high-quality datasets, which otherwise challenging to compile manually, and subsequently refining the LLM with these datasets. Through this iterative process, the model's intrinsic intelligence is progressively augmented. This research direction will constitute our future work.
evaluations using the
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions is included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation and discussion are included in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
The Theoretical Analysis is shown in Section 3.2. And we provide the corresponding assumptions for each conclusion.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code provided supplementary material and the experiment conditions are described in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The code is provided in the supplemental material with detailed running instruction.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to rum to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cctpublic//eeeccccccccccccccecct habllccggiides//// /Coo//eoeeemeeeee ietsSuumiisioni. swiissolisiyyoooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiment settings are detailed in Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Calling GPT-4 Api is expensive. Due to funding constraints, the greedy sampling method (temperature = O, do do__smplle False) to generate the output. Each time, the generated results will be the same. This will avoid the potential inaccurate caused by the randomness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: is provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: atttps://eeur// /teuris.u...p//
Answer: [Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We discussed the positive societal impacts. We did not find the negative socictal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited their work in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: we did not provide new models or new datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "WDmJ0Dguyh", "reviewer": "Reviewer_4rbe", "summary": "This paper proposes a universal prompting method for solving complex reasoning problems such as mathematical problems and the 24-point game. The method first abstracts the conditions and objectives of the problems and then progressively discovers new conditions until enough information is gathered to solve the problem. Experiments on MATH, the 24-point game, SciBench, and TheoremQA indicate that the proposed method is effective and universal.", "questions": "- What is the number of responses for each model (including baselines and the proposed MACM) in Table 1 and Table 2?\n\n- On which dataset is the experiment in Figure 6 conducted?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and easy to follow.\n\n- The experimental results are solid, showing significant improvements.\n\n- The proposed method outperforms other prompting methods with a comparable number of responses.", "weaknesses": "- In Table 1 and Table 2, the number of responses for each problem should be highlighted to indicate whether the comparison is fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions regarding our work. Here are our responses to the comments you raised:\n\n---\n\n**For *Weaknesse 1 & Question 1***\n\n> What is the number of responses for each model (including baselines and the proposed MACM) in Table 1 and Table 2?\n\nThanks for the question! In **Table 1**, for the methods we self-implemented, we standardized the number of responses $n$ generated by GPT-4 Turbo to $1$ for all prompting methods (as stated in **line 375** of our manuscript). For *SC-CoT*, we set the number of voters $v = 5$. The specific implementation method involves first asking the model to generate $v = 5$ different solution approaches for the problem, and then instructing the model to solve the problem according to each of these approaches. In Baseline *CSV + Voting*, their setting is: the number of sampled paths $k = 16$.\n\nRegarding the average number of responses $n$, we have summarized the results in the following table.\n\n| Method         | Average responses $n$| \n|:----------------:|:-----------------:|\n| CoT            | $5$               | \n| SC-CoT         | $25$              |\n| CSV†     | $7.552$                 | \n| CSV-Voting†     | $47.26$          | \n| MACM           | $40.48$           | \n\n†: They did not provide official code implementation and the related data is not involved in their paper. The data come from our own implementation.\n\nIn **Table 2**, for the 24-point experiment, the data for *ToT* are sourced from their original paper, where the breadth b of the corresponding tree for *ToT* was set to $b=5$. In *SC-CoT*, the best of k samples parameter was set to $k=100$. For the sequence sorting experiment, the data for *GoT* also come from their original paper. They did not provide specific settings for this. For the *MACM*, we standardized the number of responses $n$ generated by GPT to $1$.\n\nRegarding the average number of responses $n$, we have summarized the results in the following table.\n\n**For 24-point problem**:\n\n| Method  | Average responses $n$ | \n|:---------:|:---------------------------:|\n|     CoT          |             $4.17$   |\n|     ToT* ($b=1$)  |          $23.72$     |\n| ToT* ($b=5$) | $67.81$                   |\n| MACM (w/o code verification)  | $63.3$  | \n| MACM (w code verification)    | $22.12$ | \n\n**For sequence sorting**:\n| Method  | Average responses $n$| \n|:---------:|:---------------------------:|\n|     GoT*  |  $54.3$                  |\n| MACM (w/o code verification) | $51.72$| \n| MACM (w code verification)   | $6.882$ | \n\n*: The related data is not involved in their paper, we obtained the data from their official GitHub repository.\n\n**For *Question 2***\n\nThank you for your question! As stated in **lines 237-239**, we performed these two experiments on 200 randomly selected questions from the MATH dataset **that the original GPT-4 Turbo model answered incorrectly**. **Figure 6** demonstrates the corrective capabilities of different prompting methods under various parameter settings for these questions that GPT-4 originally answered incorrectly.\n\n---\n\nThank you again for all your feedback! We hope our answers can address all your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oEec6RKPBi", "reviewer": "Reviewer_zB7e", "summary": "The paper presents a novel prompting technique MACM, which utilises multiple agents to cooperate and perform backtracking for mathematical reasoning problems.", "questions": "The MACM protocol seems rather convoluted to implement. Is there any venue to simplify it without losing performance in the authors' eyes?", "limitations": "The paper did not discuss much about the approach's limitation.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The prompting method seems to work well for mathematical reasoning tasks and shows a degree of generalisation.", "weaknesses": "My main concern is with the evaluation method. As Appendix B suggest that the authors used GPT4 Turbo as a judge and only tested on a rather randomly selected set of MATH. This is very worrying, because (a) MATH has its own evaluation protocol, and the Minerva paper [1] also gave a good evaluation protocol. The reliance on GPT4 Turbo as a judge seems unjustifiable. (b) Why randomly select a subset, instead of using the whole MATH test set?\n\n\n[1] Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., ... & Misra, V. (2022). Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35, 3843-3857.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. Here are our responses to your concerns: \n\n---\n\n**For weakness (a)**\n\n> MATH has its own evaluation protocol, and the Minerva paper [1] also gave a good evaluation protocol. The reliance on GPT4 Turbo as a judge seems unjustifiable.\n\nThank you for your question! GPT-4 serves as a judge for voting/judging **intermediate steps** in the reasoning process, thereby helping the overall system achieve higher accuracy. It does **NOT** determine the correctness of the final result. \n\nIn fact, our evaluation protocol follows the standards set in [1], Section D.1 of their paper. Initially, results are output in an easily extractable box and then verified using SymPy, as shown in **line 450 of our code in prompt/prompt.py**.\n\n**For weakness (b)**\n\n>  only tested on a rather randomly selected set of MATH. This is very worrying\n\nThe MATH dataset contains tens of thousands of questions, and testing multiple methods on the entire dataset is often a costly endeavor. Even if testing one question with different prompting methods only costs 1 dollar, testing the entire dataset just once would require more than 12500 dollars, which is generally unaffordable. If a code interpreter is used, the cost would be even higher.\n\nWe referred to the evaluation methods of other top-performing methods on the MATH dataset leaderboard and selected a subset of the data for testing. For example:\n\n1. [2] includes the code implementation for the methods ranked **2nd** and **3rd** in accuracy on the leaderboard for the MATH dataset. Their results are based on a self-selected **1000 test cases**. **In their paper, Section 4.1 states: *We report results on the 1K-sized validation subsets for MATH and GSM8K created by us*.**\n\n2. [3] includes the code implementation for the method ranked **4th** in accuracy on the leaderboard for the MATH dataset. Their results are based on a **500 test cases**. **In their paper, the first line of Table 5 states: * *denotes using 500 test examples subset*.**\n\n3. In OpenAI's prompting-related paper [4], their evaluation results for the MATH dataset are also based on a self-selected **500 test cases**. **In their paper, Appendix C states: *We therefore evaluate our models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random.***\n\nWe randomly selected one-third of the data from the MATH dataset and ensured that the distribution of level 1-5 questions is consistent with the original dataset. To ensure a fair comparison, all self-implemented experiments were conducted on this one-third subset. The size of this subset already exceeds the 500 or 1000 cases chosen in previous work.\n\n**For Question**\n\nAccording to **Figure 7**, the four components of MACM each contribute to logical reasoning and solving complex problems. To adapt to problems of varying difficulty, we have set some external parameters, such as the number of voters and the number of Condition Mining iterations. For simpler problems that do not require complex logical reasoning, users can lower these parameters to effectively increase problem-solving speed.\n\n**Reference**\n\n[1] Solving quantitative reasoning problems with language models\n\n[2] OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset\n\n[3] Cumulative Reasoning With Large Language Models\n\n[4] Let's Verify Step by Step\n\n---\n\nThank you again for your feedback! We hope our response can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3QIdU6Zbr8", "reviewer": "Reviewer_pd1k", "summary": "The paper proposes a prompting method called Multi-Agent System for conditional Mining (MACM). MACM involves three agents, Thinker, Judge, and Executor, who maintain a condition list and try to solve the problems when the conditions are sufficient.\n\nThe paper conducts experiments with GPT and Llama series on MATH, 24-points game and sequece sorting. The results show MACM’s superiority to prompting methods like ToT and GoT under these settings.", "questions": "See above in weaknesses.", "limitations": "See above in weaknesses.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper does show significant improvement of models’ performance with MACM unber specific settings.", "weaknesses": "- MACM seems to lack novelty. MACM’s idea of maintaining a more free-form context for planning has been explored by previous works like Cumulative Reasoning.\n- The experimental setups in the paper seem confused and lack rigor, making the results less convincing. Examples include but are not limited to:\n  - In section 4.2, It is unreasonable to state that ToT and GoT are incompatible with MATH and quit comparing with them. Algorithms like ToT have been applied to open-form QA tasks by previous works like LLM reasoners.\n  - Across most experiments, it is confusing to distinguish between 0-shot/IO and CoT for GPT-4 on reasoning tasks, for GPT-4 seems to always use CoT on reasoning tasks if not specially prompted.\n  - Especially, experimental setups described in Appendix B seem rather unreasonable:\n    - The paper states that it always uses “$Top_{k}=1$, and the temperature $t=0$”, i.e. greedy decoding, but this seems incompatible with SC-CoT (though still possible with minor randomness existing).\n    - The meaning of “length of the chain $l$” in CoT seems not explained across the paper.\n    - The `max_tokens` seems too small for complex tasks like MATH.\n- The writing lacks clarity and is hard to follow:\n  - The paper tends to list many settings that are different in multiple dimensions and their results in a line (e.g. Figure4,7, Table2,3), making attribution of the results difficult.\n  - The paper spends much space on specific examples (\\~2 pages across pages 4-6) but seems to show few special things.\n\nHopefully, the authors could put more work into experimental setup and presentation to improve the quality of the work.\n\n# **After rebuttal and discussion**\n\nThe discussion period brings up \n\n- 2 new weaknesses:\n    1. According to Figure 4, MACM's gains on open-source models are not clear and may be marginal. Could you compare MACM with 0-shot and Majority voting, at least on LLaMA3-Instruct 8B?\n    2. To test the impact of different models on MACM as **a new method**, it is not sufficient for GPT-3.5 to only test with MACM -- comparisons with baselines are needed. However, this could also be achieved with open-source models, which should be much cheaper and faster.\n- 1 new question: The exact versions of GPTs in the paper are unclear. You should at least annotate the exact versions and better compare the methods with the exact same version.\n\nHowever, most concerns have been resolved by the authors in their replies.\n\nConsidering the improvement in the rebuttal and discussion, I believe that the submission has a contribution for **proposing a new effective and general prompting method, which is limited in being only applicable to strong models**.\n\nHowever, the submitted manuscript is indeed not good enough because it **lacks many details and is not well-formed enough, actually bringing an unnecessary burden for the community to follow the work**, as shown in the long discussion. I tend to take it as not ready for publication in its current form. Hopefully, the authors will add all the necessary content to future versions.\n\nIn summary, I would like to keep my final rating as 4 as a reminder of the presentation problem and leave it to AC to decide whether this submission should be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. Here are our responses to your concerns: \n\n---\n\n**For *Weakness 1:***\n\nMACM and Cumulative Reasoning differ significantly, especially in the rechecking and voting processes. Intuitively, Cumulative Reasoning has an accuracy of only $72.2$% on MATH. According to OpenAI's official data [1] , GPT-4 Turbo itself can achieve an accuracy of $72.6$% on math problems. Meanwhile, MACM achieves an accuracy of $87.92$% on MATH, which is significantly higher than Cumulative Reasoning.\n\n**For *Weakness 2 (a):***\n\n> In section 4.2, It is unreasonable to state that ToT and GoT are incompatible with MATH and quit comparing with them.\n\nToT and GoT have significant limitations in practical applications. They can only be applied to data with relatively fixed formats and low information content, as stated in our **Appendix A**. In contrast, MACM has greater versatility and applicability. In LLM reasoners, ToT has only been applied to relatively simple tasks like GSM8K, AQuA, and Game24.\n\nTo our knowledge, there has not yet been any work that applies ToT to an entire MATH dataset.\n\n**For *Weakness 2 (b):***\n\n> for GPT-4 seems to always use CoT on reasoning tasks \n\nCurrently, there is no evidence to suggest that GPT-4 **always** uses CoT on reasoning tasks. To ensure a comprehensive and accurate comparison, we adopted the same experimental settings as previous baselines, including CSV, ToT, and GoT prompting methods. This involved thoroughly testing both IO and CoT methods.\n\n**For *Weakness 2 (c):***\n\n> experimental setups described in Appendix B seem rather unreasonable\n\n1. The specific process for performing **SC-CoT** using **greedy decoding** is as follows: First, let LLM generate several plans using a prompt such as, `Please help me to list several different plans for solving this problem.` Then, add each of these plans to different execution paths and continue the subsequent reasoning separately. Finally, aggregate the results and conduct a vote. We use greedy decoding here to reduce the randomness in the experiments.\n\n2. The length of the chain $l$ refers to the number of reasoning steps the model performs in CoT. We set this parameter to fix the number of reasoning steps to facilitate the evaluation of efficiency.\n\n3. For MACM, we set the thinker's `max_tokens` to $512$, the judge's max_tokens to $4$, and the executor's `max_tokens` to $256$. These `max_tokens` values are for each step of the process. MACM decomposes each problem in the MATH dataset into multiple reasoning and judgment steps, making these `max_tokens` settings reasonable and feasible for each step.\n\n**For *Weakness 3 (a):***\n\n> The paper tends to list many settings that are different in multiple dimensions and their results in a line (e.g. Figure4,7, Table2,3), making attribution of the results difficult.\n\nDifferent models and prompting methods can affect the final results. \n\nIn **Figure 4**, we tested various LLaMA models using different prompting methods on the MATH dataset. Different model versions are distinguished by different colors, with the model parameters labeled on the first row of the x-axis and the prompting methods labeled on the second row of the x-axis.\n\nIn **Figure 7**, we tested the contribution of four different components in MACM to the improvement in model accuracy. Different components are indicated by different colors on the x-axis, and combinations of different numbers of components are represented by bars of various colors.\n\nIn **Table 2**, we have 5 columns: the first column lists the tasks tested, the second column indicates whether a code interpreter was used for verification, the third column specifies the model used, the fourth column details the prompting method employed, and the fifth column shows the resulting accuracy.\n\nIn **Table 3**, we have 4 columns: the first column lists the methods used, which include information about the model, prompting method, and whether Python was used. The second to fourth columns correspond to three different subsets of SciBench.\n\n**For *Weakness 3 (b):***\n\n> The paper spends much space on specific examples (~2 pages across pages 4-6) but seems to show few special things.\n\nAs with previous prompting papers, we have listed specific examples to help readers better understand the details of our method. For instance, in the Graph of Thought [2], refer to Figures 2, 3, 4, and pages 3, 5, 6. Similarly, in the CSV paper [3], refer to Figures 1, 3, 4, and pages 3, 5, 6, 8.\n\n**Reference**\n\n[1] GPT-4o text evaluation report.\n\n[2] Graph of Thoughts: Solving Elaborate Problems with Large Language Models, AAAI 2024\n\n[3] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification, ICLR 2024\n\n---\n\nThank you again for your feedback! We hope our response can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWOtCWGqpF", "reviewer": "Reviewer_upb5", "summary": "This paper aims to address the limitations of existing methods that require distinct prompt designs for different mathematical problems. The authors propose a general Multi-Agent System for conditional Mining (MACM) method that uses three LLM agents (Thinker, Judge, Executor) to iteratively propose the conditions for a problem, verify whether the existing conditions can reach the objective, and execute calculations. Experimental results clearly show the accuracy improvements of MACM.", "questions": "1.\tIn the first using case in Figure 3, why do the LLM gets two different “New condition” (one correct, one incorrect) under the same prompt (prompt 2)? Does this mean that thinker in this paper is more likely to take advantage of the randomness of large models in repeated condition generation, rather than really improving the ability of LLMs to discover new conditions?\n2.\tIn theoretical analysis, how to ensure that the thought space T can be traversed, i.e., why P_MACM(A_correct|C) = P_MACM(A_correct|T)?\n3.\tIn Section 4.2, it's necessary to rerun the results of GPT4-ToT instead of copying the results from the original paper, because the capability of GPT4 is also improving. It is difficult to tell whether the improvement in Table 2 is caused by improvements in GPT4 itself or by the method in this paper.\n4.\tIn Section 4.4, I think the most important measure of efficiency for applying this method is to compare the number of API calls from this method with different prompt methods (e.g., ToT, GoT) when obtaining the results in Tables 1 and 2. This is the most direct indicator of the cost in practice.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strengths include:\n\n1.\tThe motivation to design a method that does not require prompt design for different mathematical problems is important and makes sense.\n2.\tThe authors clearly explain the details of MACM, which makes it easy to follow.\n3.\tThe authors conduct experiments on various datasets, and the accuracy improvements are significant.", "weaknesses": "The main weaknesses include:\n\n1.\tThe essential technical contributions of this paper may be limited, and an explanation is needed as to why the method is effective. (please refer to question 1 below).\n2.\tThe theoretical analysis is a little weird and needs further explanations. (please refer to question 2 below).\n3.\tMore experimental results are needed to demonstrate the effectiveness of the proposed method. (please refer to questions 3-4 below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. Here are our responses to your concerns: \n\n---\n\n**For *Weakness 1* and *Question 1***\n\n> why do the LLM gets two different “New condition” (one correct, one incorrect) under the same prompt (prompt 2)?\n\nThanks for the question! In MACM, the conversation takes place in **multiple rounds**, with previous dialogue information added to the conversation history. This means that in Figure 3, after the first prompt 2 generates an incorrect new condition, the judge's evaluation of this new condition will also be recorded in the conversation history (the leftmost **prompt 3** and **prompt 4** in **Figure 3**). This reduces the likelihood of the model considering incorrect conditions in subsequent sampling processes.\n\n> Does this mean that thinker in this paper is more likely to take advantage of the randomness of large models in repeated condition generation, rather than really improving the ability of LLMs to discover new conditions?\n\nMACM includes various thought-guiding processes that help the LLM continuously correct potential past errors and guide the large model to gradually use existing conditions to obtain the correct new conditions, ultimately finding the correct result. In addition, In the process of the judge evaluating each new condition, we have configured the use of a code interpreter. This can significantly reduce potential errors and gradually guide the model towards producing correct answers.\n\n**For *Weakness 2* and *Question 2***\n\nIn the theoretical analysis section, our aim is to highlight the advantages of MACM over ToT and GoT in **ideal situations**. Specifically, each condition search in MACM can be viewed as a new start, meaning it doesn't need to begin its search from a few initial paths like ToT and GoT. Therefore, theoretically, as the search time increases, MACM's exploration space will gradually approach the thought space  $T$\n\nIn contrast, ToT and GoT are constrained to specific trajectories, making it difficult for them to consider alternative perspectives. Thus, MACM offers greater flexibility and potential in exploring and traversing the thought space, which can enhance problem-solving effectiveness.\n\n**For *Question 3***\n\nThank you for your suggestion. We have standardized our testing by using GPT-4-1106-preview to re-evaluate the results for the 24-point game, with all code-interpreter functionalities disabled during the testing process. The testing code for IO/CoT/ToT is sourced from ToT's official GitHub repository.\n\n| Method                | Accuracy (%) |\n|:----------------------:|:------------:|\n|  IO               |      $23$    |\n|  CoT |      $25$       |\n| SC-CoT  |      $34$      |\n| ToT (b = 1)|     $55$       |\n| ToT (b = 5)|     $78$      |\n| MACM           |     $91$      |\n\nExperimental results show that MACM still outperforms ToT. By analyzing the error cases in ToT, we found that many problems that MACM can solve but ToT cannot are due to MACM's judge correcting these errors.\n\n**For *Question 4***\n\nThanks for your suggestion!\n\nRegarding the average number of responses $n$ generated by LLM, we have summarized the results in the following table.\n\nFor Table.1 \n\n| Method         | Average responses $n$| Accuracy (%) |\n|:----------------:|:-----------------:|:------------:|\n| CoT            | $5$               | $74.36$|\n| SC-CoT         | $25$              |$80.12$|\n| CSV†     | $7.552$                 | $73.54$|\n| CSV-Voting†     | $47.26$          | $84.32$|\n| MACM           | $40.48$           | $87.92$|\n\n†: They did not provide official code implementation and the related data is not involved in their paper. The data come from our own implementation.\n\n**For Table.2 24-point problem**:\n\n| Method  | Average responses $n$ | Accuracy (%) |\n|:---------:|:---------------------------:|:------------:|\n|     CoT*          |             $4.17$  |  $25$       |\n|     ToT ($b=1$)*  |          $23.72$     |     $55$       |\n| ToT ($b=5$)* | $67.81$                   |    $78$       |\n| MACM (w/o code verification)  | $63.3$  |     $91$      |\n| MACM (w code verification)    | $22.12$ | $99$      |\n\n**For Table.2 sequence sorting**:\n| Method  | Average responses $n$|  Accuracy (%) |\n|:---------:|:---------------------------:|:------------:|\n|     GoT*  |  $54.3$                  |     $89.06$       |\n| MACM (w/o code verification) | $51.72$|      $92$       |\n| MACM (w code verification)   | $6.882$ |      $100$       |\n\n*: The related data is not involved in their paper, we obtained the data from their official GitHub repository.\n\nExperimental results show that, compared to baselines such as CSV, ToT, and GoT, MACM requires fewer LLM responses even without a code interpreter function, while achieving higher accuracy.\n\n---\n\nThank you again for your feedback! We hope our response can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a prompting technique, MACM, for mathematical reasoning problems. The reviewers found the paper easy to follow and the method working. An extensive discussion emerged, and all reviewers declared that most of their concerns were answered, with three out of four reviewers increasing their ratings. The lowest-scoring reviewer expressed a belief that `the submission has a contribution for proposing a new effective and general prompting method` and left his score as a reminder of presentation issues. It is for that reason I recommend the paper for acceptance.
Decision: Accept (poster)