Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Authors: Tengxiao Liu
Qipeng Guo
Yuqing Yang
Xiangkun Hu
Yue Zhang
Xipeng Qiu
Zheng Zhang
Venue: EMNLP 2023 Main (2023)
Keywords: Math Reasoning
Chain-of-Thought

Abstract: As large language models (LLMs) have shown effectiveness with different prompting methods, such as Chain of Thought, Program of Thought, we find that these methods have formed a great complementarity to each other on math reasoning tasks. In this work, we propose XoT, an integrated problem solving framework by prompting LLMs with diverse reasoning thoughts. For each question, XoT always begins with selecting the most suitable method then executes each method iteratively. Within each iteration, XoT actively checks the validity of the generated answer and incorporates the feedback from external executors, allowing it to dynamically switch among different prompting methods. Through extensive experiments on 10 popular math reasoning datasets, we demonstrate the effectiveness of our proposed approach and thoroughly analyze the strengths of each module. Moreover, empirical results suggest that our framework is orthogonal to recent work that makes improvements on single reasoning methods and can further generalise to logical reasoning domain. By allowing method switching, XoT provides a fresh perspective on the collaborative integration of diverse reasoning thoughts in a unified framework.

Plan, Verify and Switch:
Integrated Reasoning with Diverse X-of-Thoughts
Tengxiao Liu'; Qipeng Guoi, Yuqing Yang', Xiangkun Hu²,
Yue Zhang"! Xipeng Qiu'! Zheng Zhang "School of Computer Science, Fudan University
2 AAmazon AWS AI, 'School of Engineering, Westlake University
(txliu21, yuqingyang21 Jem. fudan. edu. cn, {gqipeng, xiangkhu, zhaz)@amazon.com com xpqiuεfudan edu.cn, zhangyue@westlake. edu.
Abstract
Input
Input
Input
As large language models (LLMs) have shown effectiveness with different prompting methods, such as Chain of Thought, Program of Thought, we find that these methods have formed a great complementarity to each other on math reason- ing tasks. In this work, we propose XoT, an in- tegrated problem solving framework by prompt- ing LLMs with diverse reasoning thoughts. For each question, XoT always begins with select- ing the most suitable method then executes each method iteratively. Within each iteration, XoT actively checks the validity of the gen- erated answer and incorporates the feedback from external executors, allowing it to dynami- cally switch among different prompting meth- ods. Through extensive experiments 10 pop- ular math reasoning datasets, we demonstrate the effectiveness of our proposed approach and thoroughly analyze the strengths of each mod- ulc. Moreover, empirical results suggest that framework is orthogonal to recent work that makes improvements on single reasoning methods and can further generalise to logical reasoning domain. By allowing method switch- ing, XoT provides a fresh perspective on the collaborative integration diverse reasoning thoughts in unified framework.
Planning
Reasoning
Reassnning
Reasoning
failled
Refinement
Verification
passed
Oulpul
Oulput
Oulput
CoT
Self-refine
XoT
Figure CoT only reasons in single pass, while self- refine involves refinement using the same method. XoT integrates a verification module that makes difference in method planning, enabling the attempts of diverse reasoning thoughts within an iterative framework.
(Brown et al., 2020; Chowdhery et al., 2022; Tou- vron et al., 2023a; OpenAl, 2023) are able to solve mathematical problems through textual ra- tionales with Chain-of-Thought prompting (Wei et al., 2022) (CoT) or through Python functions with Program-Aided Language Model (Gao et al., 2022) and et al., 2022) (PAL or PoT)) These prompting meth- ods exhibit unique strengths and limitations. CoT generates a step-by-step reasoning flow in natural language and performs calculations on the fly. This approach enables a more flexible solution format, but may result in a loss of precision since language models often struggle with arithmetic calculations (Lewkowycz et al., 2022: Wei et al., 2022). On the other hand, PoT PAL resolves problems through Python statements, relying on Python interpreters to ensure calculation accuracy. Another noteworthy and intriguing prompting method is to form math problems as linear cquation systems (He- Yueya et al., 2023). Similarly, inspired by Linear Algebra, we propose Equation-of-Thon Thouugh (EoT), which per- forms math reasoning in more direct way.
prompting (Chen
1 Introduction
The AI community has long sought achieve automated reasoning (Hewitt, 1969), which is an important component of Artificial General Intel- ligence (Steunebrink et al., 2016). Mathemati- cal reasoning, as a cognitive skill essential for humans yet challenging for language models, at- tracts increasing interests and commitment from re- searchers (Feigenbaum and Feldman, 1963; Wang al., 2017: Lu et al., 2022).
With the abilities endowed by in-context learn- ing (ICL), Large Language Models (LLMss)
"Wor done during internship at AWS Shanghai Lab. Coorespponding authors.
The diversity inherent each method does not

---

Method
Method 2
Method 3
render them as competing or mutually exclusive alternatives. On the contrary, in practical problem solving scenarios, possessing multiple methods can always yield a range of complementary advantages. The distinct problem-solving approaches can con- tribute to synergistic benefits that surpass the out- comes of any single approach. We find that this intuition also applies to the realm of math reason- ing. With the availability of CoT, PoT and EoT, hold the hypothesis that model has the potential solve a problem if reaches the correct answer using any one of the prompting methods. As il- lustrated in Figure 2, our analysis shows that the model exhibits the potential to solve 92.72% of the problems, surpassing the best performing single method by over 10%.
100
92.7
92.7
90
90.1
89.1
Accuracy (%
80
80.4
79.8
76.0
70
70.9
60
55.9
50
GSM8K
SVAMP
AQuA
Figure 2: Complementarity of X-of- methods on different datasets. The stacked bars indicate the best performance achieved by using one, two and three methods separately. Employing multiple methods under oracle setting can offer significant performance gains.
Motivated by this observation, we propose XoT, an integrated math problem solving framework, which improves the LLM's reasoning ability by switching among diverse reasoning thoughts. Since there is no guarantee that LLMs can always solve the problem in a single attempt, we follow the hu- man intuition and allow the model to rethink and switch to different method when encountering difficulties or obstacles. We apply two complemen- tary verification methods to facilitate the model to decide whether it is time to switch to another method: passive and active verification. Passive verification relies on the external executors to pro- vide determinable results based on the generated programs (Chen et al., 2023; Le et al., 2022). offers shallow inspections, such as program syntax issues or the runtime errors. For active verification, we ask the model to verify the solution by check- ing whether the answer adheres to the conditions outlined in the original question.
To demonstrate the effectiveness of XoT, we conduct extensive experiments on 10 popular math- ematical reasoning datasets and achieve consistent improvement. Empirical results suggest that XoT can accommodate recent work that focuses on im- proving single reasoning methods. Additional ex- periments also indicate that XoT can generalise to other domains such logical reasoning tasks.
We summarize the main contributions as follows. First, we propose an integrated problem solving framework XoT. utilising the complementarity of different reasoning thoughts. Second, we introduce EoT which solves math problems with system of linear equations, serving complementary method to existing approaches. Third, incor- porate passive and active verification to facilitate the framework to switch among diverse reasoning thoughts, empowering the framework to make in- formed decisions regarding the subsequent steps to be taken. More generally, XoT sheds lights on a new direction of interacting with diverse reasoning methods and tools. As shown in Figure 1, instead of sticking to one determined method, LLMs benefit from the verification and the flexible switch- ing among available reasoning thoughts."
As shown in Figure 1, XoT consists of three mod- ules that work in iterative framework: planning, reasoning and verification. Given a problem as in- put, the plamning module first proposes the most appropriate method. The reasoning module then generates one solution using the planned prompt- ing method. With the outputs and the results from external executors, the model is asked to assess the answers in the context of the questions. If the answer fails the verification, the planning module for another round of iteration and attempt alternative methods. The iterative pro- cess concludes when the verification confirms the correctness of the answer or after exhausting all available methods.
Related Work
will go back to
2.1 Math Reasoning with L.MSs
As the field of large language models continues to prosper, many prompting techniques have emerged
'Code is publicly available at: tengxiaoliu/XoT.
com/

---

unlock the reasoning abilities of LLMs (Qiao al., 2022). Early success includes reasoning with step-by-step chain of thought (Wei et al., 2022), decomposing questions into sub-questions in a least-to-most fashion (Zhou et al., 2022), zero-shot prompting LLMs with simply one sentence (Ko- jima et al., 2022), writing programs to solve pro- cedural tasks (Gao et al., 2022: Chen et al., 2022). Despite generating solutions in single forward pass, one line of work employs multiple reasoning re- sults and ensembles them by majority vote (Wang et al., 2022), and stepwise verifier (Li ct al., 2022). Additionally, Tree-of- Thoughhss (Yao et al., 2023) deliberately explores multiple reasoning paths and searches over a tree-structured reasoning states. Imani et al. (2023) propose to vote over multiple solutions generated with algebraic and program prompts. One concurrent work (Zhao et al., 2023) considers the difference of CoT and PoT and asks the LLM to choose one better reasoning rationale. contrast to their work, XoT involves more re- liable verification modules and switches methods when necessary.
Question Together Lily, more than David. David found half of what Bodhi found. How many Lily find?
collected insects. Lily found 7
Gold: 16.0
CoT
Let's start by using the information given to set equstions: (to represent
insects collected)
L D (since Lity found 7 more than David) found)
(since David half
= 24
...... Tharafore, Lily found 19 insects. The answer is 19.
X
insects, total 43
/otal bodhi, insects = david insects 2 Ily insects david insects ans Ily. insects
david
17.75
EoT fly david + bodhi david
fly david a bodhi = 43
Figure 3:
In particular cases where CoT and PoT fall short, BoT successfully solves the problem, which serves as good complement.
to iterative refinement, as we have the flexibility to switch solutions when refining no longer leads to further improvement.
2.2 Iterative Refinement
One stream of work is dedicated to iteratively en- hancing LLMs by continuously reevaluating and refining outputs until the desired quality is achieved. Madaan et al. (2023) prompts the model to write feedback based on previously generated drafts and leverages the feedback generate high-quality out- puts. Similarly, Chen al. (2023) iteratively de- bugs the code by utilizing external program exccu- tion results and code explanations generated by the model itself. In order to avoid repetitive mistakes, Shinn et al. (2023) builds a memory of previous errors, while Wang and Li (2023) collects all mis- takes during the training phase to provide global insight. When considering sources of hints guide rethinking, Paul et al. (2023) focuses on interme- diate reasoning steps, while Zheng et al. (2023) directly utilizes the previously generated answers. Qi et al. (2023) propose to emulate the divide- and-conquer fashion of human thinking strategy and involve self gulf-iussti and recursive thinking processes in the problem solving framework. Al- though these approaches contribute to improving the reasoning quality of LLMs, they are limited in retrying without looking around for other possible thoughts. contrast, our proposed method aims to explore alternative solutions, and it is orthogonal
Preliminary
3.1 Prompting methods
For math reasoning tasks, we use three rea- soning thoughts in this work, namely Chain-of- Thought (CoT), Program-of-Thought (PoT) and Equation-of-TTe Thoughh (EoT). Despite the well- known strengths of CoT and PoT methods, our proposed FoT excels particularly in reasoning with unknown variables. For each problem, EoT at- tempts to model the questions linear equations and involves unknown values in the description. A detailed formulation of EoT prompting can be found in Table 12 of Appendix C. As illustrated in Figure 3, while CoT correctly sets up the equa- tions, it fails in accurately performing the calcu- lations. PoT falls short in dealing with unknown variables, as Python requires that every variable is defined with value. Assigning value to unknown variable PoT to generate a mislcading step (the highlighted line). In comparison, EoT manages to express the question context in straightforward equations and solves them with deterministic equation solver.
hallucinates

---

3.2 Complementarity
Algorithm 1 XoT Reasoning Algorithm
Given question we denote the correctness of the reasoning answers using each method Rx (q), where X E {CoT, Pof, Bof') denotes the diverse reasoning methods. (ạ) (0, 1} represents whether the generated answer correct according the gold label. We define the accuracy under the oracle setting as:
Require: input question q, method set M, plan-
ning module P. reasoning module R, verifica-
tion module V
<-0
2: while M > do
3:
my - P(M)
b. Choose method
4:
M < M [m;]
5:
y + (g)
ACCoracle = E R (P) ReO (2)-
6:
if (v) then
break
> Verification passed
(1)
else
The oracle setting represents that the model has the potential for solving one given problem if any the methods accurately generates the answer. also implies that in cases where the generated an- swer does not match the gold answers, XoT will make further attempts using alternative methods to answer the question. Under oracle setting, the model can potentially achieve more than 10% gains various datasets. In Figure 2, the bar at the bot- tom represents the highest performance achieved by employing a single method, followed by the optimal performance achieved through the use of two methods. The overall stacked bar shows the utilization of all three methods, which indicates the upper bound that can be reached through the combined collaboration of various methods.
t ++
Continue next iteration
10: end if
11: end while
12: return y
> Return the solution
one solution y using the proposed method m, Fol- lowing this, the verification module evaluates the solution by rethinking the answer within the given conditions. If the answer successfully passes the verification, we proceed to return the current so- lution. Otherwise, XoT will move forward to the next iteration. Every module is implemented with LLM through inference under few-shot setting. We will elaborate each module with details.
4.2 Planning and Reasoning
4 XoT
The planning module is responsible for selecting the appropriate method at the beginning of cach round of iteration. Recent work shows the neces- sity to equip reasoning framework with the ability to plan ahead (Lu et al., 2023). As elaborated Section 3, it is evident that each method possesses distinct strengths. Our intuition is to consistently initiate the process with the optimal method to en- hance reasoning efficiency.
Our goal is to develop a generalized problem solv- ing framework that can automatically select the appropriate method for different problems and has the capability to switch among reasoning thoughts using both active and passive verification. We first describe the overall framework and introduce each module in detail.
The reasoning module performs few-shot rea- soning with the planned prompting method. Each round of reasoning operates independently, mean- ing that subsequent iterations do not rely on the failed reasoning attempts of previous iterations.
4.1 Overall Framework
The overall pipeline is described in Algorithm 1. The inputs of our framework include a question q and predefined set of methods M. With the user input, XoT employs its three built-in modules to output the final solution, namely planning module P, reasoning module R and verification module V. These three modules collaborate in an iterative manner. Suppose iteration the planning mod- ule P first chooses the most appropriate method available: m, = P(M). The chosen method subsequently excluded from the set of methods. The reasoning module is then tasked to generate
4.3 Verification module
The verification module assesses the effectiveness of the reasoning solution through two approaches: passive verification and active verification.
When solutions involve offloading computation to external tools, the execution results naturally serve as a passive verification. Any occurrence errors or exceptions during the execution directly

---

Input
bringshis dog
hearworrm
heartworm checs
brought
Planning Module
Reasoning Module
Verification Module
Method
Program-of-Tee Srggram-of-hhougtt
Passive feedback passed
Thouram-o-Thought
vaccine cost 28
Active verification
iinss
The total cast should be equal to the sun of the cost of vaceines and the cost of the heartworn check.
heartwirn (vaccine
0-6
vaccines)
(leartworn peeerceeeeeepptetttttc (saccire_cns)
vaccines))
assert total rost heartvoms chack
money brought
vaccines
voney_left money_brougpe total_cost ans = monnyyy left
Iteration
continue
Equation-af-Thought
Equation-gf-Thuught vaccine
heartwors, check, cast 8.6 (vaccine cost weartiors_check_cost
total
heartwors
check should equal to senrtwem_check_oost
total
total, soney _otall 125
asount c.
totel vazcine_rost
assert vaccine_cost tatal soney 9.6 and left_ sine total ronay. (vaccins. cost heartwom check cost)
heartwors charttoorrcccck.ccoo cast
Iteration
tutal _rnny total_cost
Figure 4: Overview XoT. Following the suggestion of the plamning module, XoT first reasons with PoT. However. the generated answer fails in the verification module. In the second iteration, the selected method is FoT. The reasoning module successfully generates the solution that passes the verification.
Datasct
Data
Steps
format for this evaluation is an assertion statement, as shown in Figure This assertion is subsequently combined with the original solution for external tools to execute. If no issues arise during this ex- ecution phase, it means the solution successfully passes the verification. A detailed illustration of the prompts we use can be found in Appendix C. The verification module is specially designed for PoT and EoT as the intermediate values can be eas- ily obtained. We leave the exploration of more effective verification for CoT future work.
GSM8K (Cobbe et al., 2021)
1.319
3.25
SVAMP (Patel et al., 2021)
1,000
1.24
AQuA (Ling al., 2017)
253
3"
Algebra Yucya et al., 2023)
222
2:
GSM-hurd (Gao et al., 2022)
1,313
3.25
MATH (Hendrycks al., 2021)
5,000
3'
AddSub (Hosseini et al., 2014)
SingleOP (Roy et al., 2015)
SingleEQ (Koncel- Kedziorski et al., 2015)
508
1.31
MultiArith (Roy and Roth, 2015)
600
2
Table 1: Statistics of the datasets we used. # Steps denotes the average number of reasoning steps gold answers. indicates rough estimate due to the inconsistent rationale formats.
Experiments
5.1 Experimental Setting
results in a failure in the verification process. So- lutions that pass the passive verification stage then proceed to active verification.
Datasets Our experiments are conducted comprehensive set of 10 math reasoning datasets, encompassing various challenging math reason- ing scenarios. Some widely used datasets in- clude GSM8K. SVAMP, AQuA, MATH and MAWPS (AddSub, SingleOP, SingleFQ, Multi- Arith) (Koncel-Kedziorski et al., 2016). Besides, we also incorporate several recently introduced datasets, namely Algebra, GSM-hard. Algebra comprises collection of solely algebraic word problems that be resolved through the use of equations. To increase the complexity of calcula- tions, GSM-hard replaced small numerical values with larger ones. The details of the statistics of the datasets can be found in Table 1.
In the case of active verification, the module re- thinks the answer within context of the given question. It first acquires all intermediate values associated with each variable mentioned in the so- lution. These values are computed by external ex- ecutors. We intentionally exclude the reasoning process (expressions) leading the results to pre- vent the verification module from emulating the solution's thinking process. With the intermediate results and final answer hand, the module is ex- pected to recheck whether the answer satisfies the conditions specified in the question. The desired

---

Model We query OpenAl API for experiments*. Specifically we use gpt-3. 5-turbo as the infer- ence engine. not further explained, we manually construct the prompts with 8 examples sampled from the training set. For CoT and PoT,, we directly use the examples released by published paper (Fu et al., 2022; Gao et al., 2022; Chen et al., 2022). For model generation strategy, employ greedy decoding in all runs. Due to the non-deterministic APIs, we report the average performance and the standard deviation across 3 runs. We also evaluate XoT with various base models Appendix A.2.
datasets that exhibit stronger enhancement under oracle setting. The bars in the figure represent the improvement under XoT, while the line indicates the upper bound of the improvement under oracle setting. The comparison indicates that MultiArith and SingleEQ allow minimal room for improve- ment, therefore the overall XoT performance is negatively impacted by the accumulated errors in- troduced by the verification module.
Additionally, we conduct experiments on logical reasoning task to evaluate the generalisability of XoT. Details can be found in Appendix A.I.
5.2
Main Results
Analysis
The main results are shown in Table 2. We consider three prompting methods as baselines, namely CoT, PoT and EoT. On average, XoT achieves signif- icant improvement of 5.49% across the datasets. For MATH dataset, we show the breakdown results of different question subtopics in Table 3. We also represent the performance enhancement over the strongest baseline as A. As questions MATH are too complex for equation systems to solve, we only consider CoT and PoT with passive verification. Specifically, on the AQuA dataset, which consists of multiple-choice questions, we observe that PoT EoT often fails generate valid answer due the diverse answer formats. Across the three runs, 24.4% of the PoT answers and 30.3% of the EoT answers cannot executed. Therefore, ap- plying passive verification adequate to ensure the explortion of other method options. When post processing the generated results, we further enforce a restriction that the model cannot make random guess if it fails to extract answer from the gener- ated output. Such instances should be proceeded to the next iteration to guarantee a fair evaluation of the performance.
In this section, we first analyze the effectiveness and necessity each module within XoT. Then we provide comparison with majority voting and describe how model's s self refinement can be inte- grated in our framework.
6.1 Ablation Study
Planning The plamning module decides which method to attempt the beginning of each itera- tion. We are curious about how well it performs in selecting the most suitable method among the available options. The planning module expected to select from PoT and EoT at the beginning be- cause these two methods can be verified with both active and passive verification. To demonstrate the necessity of the planning module, we conduct an experiment in which XoT is asked to execute each method in a predefined order. Whether to switch the method is still determined by the verification module. We break down the performance of each dataset with respect to different combinations methods in Table 4.
Our findings align with two design ethos of the planning module. First, it demonstrates robustness across different datasets. While specific combi- nations excel different datasets, XoT equipped with the planning module outperforms all other predetermined combinations on average. For in- stance, on GSM-hard, the combination of PoT and HoT achieves the best performance, which high- lights the importance of leveraging external tools to handle calculation involving large numbers. Ad- ditionally, on SingleEE and MultiArith where XoT fails offer improvement, the combination of methods proves to efficient, surpassing the sin- gle method baselines. With the inclusion of the planning module, XoT can dynamically adjust the
Notably, we observe that the enhancements are more pronounced for the challenging datasets com- pared to the easier ones. Difficult datasets usually contain longer questions and more than 3 reason- ing steps while easier datasets such as SingleEE require only one equation to solve the problem. We find that the improvement directly correlates with the complementary nature of the three methods em- ployed across different datasets. On easier datasets, each method performs well individually, resulting in only minor complementarity. Figure 5 reveals that XoT demonstrates superior performance
Phttps://opene

---

GSM8K
SVAMP
AQuA*
Algebra
GSM-hard
AddSub
SingleOP
SingleEQ
MultiArith
Average
CoT
80.2g.2
79.5p.6
$5.11.
81.5o.s
42.4o.1
88.4a.s
93.40.
94.3g.1
97.5p.3
79.14
PoT
77.20.3
79.5o.3
49.21.0
62.50.r
61.8p.4
88.4a.2
93.40.4
98.1n.1
97.20.0
78.59
EoT
63.8p.
69.6p.7
46.70.5
82.30.5
53.80.2
71.61.0
75.4p.4
85.8p.s
78.6p
69.73
XoT
83.3p.
83.6p.6
61.70.6
89.90.a
63.4
90.5u.4
94.3p.3
97.7g.1
97.3p.a
84.63
oracle
92.50.2
92.70.3
77.01.4
95.50.5
74.30.4
93.90.3
97.5p.
99.1p.1
99.30.0
91.31
A
+3.1
+4.1
+6.6
+7.6
+1.6
+2.1
+0.9
-0.4
-0.2
+5.49
Table 2: Main experiment results across various math reasoning datasets. Under oracle setting, XoT switches the method if the generated answer does not match the gold answers. denotes we only use passive verification. A represents the improvement over the best performing baseline.
InterAlgebra
Precalculus
Geometry
NumTheory
Probability
PreAlgebra
Algebra
Overall
CoT
17.8p.
20.3p.1
24.4u.
32.21.0
30.4u.6
56.60.
49.70.1
35.770.4
PoT
14.4p.1
15.5p.1
8.80.3
31.20.7
19.60.2
36.5p.2
38.2a.1
25.79a.1
XoT
25.1p.1
26.0p.3
25.30.7
48.1o.6
36.1n.4
62.0p.2
57.30.4
42.81g.
oracle
28.10.2
31.2p.1
27.60.4
50.5o.1
39.0p.7
68.0p.4
64.10.4
47.350.2
A
+7.3
+5.7
+0.9
+15.9
+5.7
+5.4
+7.6
+7.04
Table 3:
Experiment results MATH dataset. We only employ methods and passive verification on MATH.
NoT
1st iter 2nd iter 3 iter
100
16
Xod (40
Accurucy (aak:
92.72
90
85.60
84.08
82.71
Aleboa
SWAMP
oodd
CSMA
Addash
Suglecp
Mallarsh
80
78.39
78.62
Figure 5: The correlation between oracle performance and final improvement. A higher oracle gain allows more room for XoT to improve.
PoT P
XoT
PoT
PoT XoT
oracle
oracle
oracle
Figure 6: Repeatedly exploiting the same method (PoT*) results in limited complementarity compared to XoT with three methods. PoT denotes we use different few-shot examples in three iterations.
execution order based on different questions, which ensures more consistent and robust performance. Second, the planning module enhances effi ciency, facilitating XoT to reach the final answer in fewer iterations by always starting from the most possible method. To illustrate, on GSM8K, XoT needs 1 1.46 iterations on average in comparison with 1.58 iterations with the fixed EPC order (BoT->PoT- >CoT, the best performing fixed order). Specifi- cally, 68.8% of the questions are resolved in the first iteration with XoT, as opposed to 57.2% when employing the fixed EPC order.
as PoT In other words, if the generated solution fails to pass the verification, it reconsiders its rea- soning using the same prompting method instead changing to another. The results are demonstrated in Figure 6. P uses the same few-shot exam- ples in three iterations while P uses differente examples randomly sampled from the training set. It is observed that under orcale setting, repetitive exploitation of single method has limited com- plementarity of 84. 08%, which is 8.64% less than XOT. As a result, the final performance reflects such a gap with 3 78.39% and XoT of 82.71 %.
How important is try different methods instead of exclusively relying on a single method? To investigate this, we restrict the avail- able method options to utilizing PoT only, denoted
Reasoning

---

Methods
GSM8K
SVAMP
AQuA
Algebra
GSM-hard
AddSub
SingleOP
SingleBQ
MultiArith
Average
PE
77.70.3
80.7p.2
56.71.0
81.70.5
63 63.40.3
89.6a.3
93 93.80.3
98.0n.2
95.00.2
81.85
PC
81.8p.2
82.7p.6
61.71.s
83.6n5
59.6 59.60.4
90.40.0
94,40.2
98.3g.1
97.8p.2
83.36
LP
8
80.8p.4
58.0p.6
83.80
64.60.3
88.40.1
94.10.5
96.7p.0
97.8p.2
82.80
FC
82.40.5
81.40.6
60.0p.6
92.00.3
56.20.4
87.30.1
93.70.2
15.
97.30.2
82.82
EPC
82.60.5
82.6n.6
63.11.
89. 90.3
lo.4
88.7a.8
94.50.3
96.70.0
97.50.0
84.29
PEC
82.6p.4
83.
61.81.0
85 85.3n.5
63.3p.3
90.1a.3
94.4p.
98.2p.1
97.4p.3
84.02
XoT
83.30.5
83.6
61.70.6
89.90.3
63.40.5
90.5o.4
94.30.3
97.7p.1
97.30.3
84.63
Table 4:
Results across different datasets without the planning module. We manually define the execution sequence, denoted as the combination of the first letter in each method. For example, 'PEC' indicates PoT-BoT-CoT.
0.8
GSM8K
SVAMP
180
125
32
273
Lubel
Label 0
PoT
77.20.3
79.50.6
13.65%
9.48%
2.43%
20.
70%
65
EoT
63.8p.4
69.60.7
167
1014
XoT (only PE)
79.4u.7
81.3u.3
Label
Label
82
12.60%
64.22%
0.00%
7G.89%
XoT (w/o verification)
74.514
79.20.4
EaT posskre aclive verificatioon
Eai passive verifcation
Table 6:
Ablation results of excluding the entire verifi- cation module on GSM8K and SVAMP. XoT (only PE) equipped with the verification module. The lack this module compromises its ability for iterative method- switchnng resulting in diminished performance.
0.7
395
155
29 95%
6.98%
14%
11.75%
169
676
845
Lubel
Label
12.81%
$1 25%
0.00%
64.
84.08%
Pridee
Precid
Pradic
Prodet
such simplistic verification approach yiclds alarmingly high false positive rate of 89.5% and 41.0%, as shown in Table This drawback is par- ticularly critical as our XoT's essence lies in the ability to adaptively switch methods, and a high false positive rate restricts the model's ability to explore alternative method options. By additionally incorporating active verification, despite slight compromise in accuracy, the false positive rate is substantially reduced by 56.8% and 24.3%. We also note that this approach inevitably leads increase in the false negative rate. However, this is a minor drawback as the subsequent method options still have chances to get it correct. Conse- quently, employing active verification offers 2.3% gains to the overall XoT performance.
Figure Comparison of passive and active verifications. The blue and green matrices represent verifications for PoT and EoT respectively.
active
ACC
FPR.
FNR.
XoT
PoT
X
79.3
89.5
0.0
80.4
EoT
X
88.3
32.7
0.0
PoT
77.9
41.0
16.5
82.7
EoT
81.2
16.7
20.0
Table 5: Ablation results of different verification meth- ods on GSM8K. Employing active verification signifi- cantly reduces false positive rate and results notable improvement in the overall XoT performance.
Additionally, we explore the necessity of the iterative nature of XoT by removing the entire veri- fication module. In this scenario, we only reason once with the most suitable method suggested by the planning module. The results are presented in Table 6. As our planning module mainly chooses the method from PoT or EoT, we here restrict the available methods to PoT and BoT only in XoT framework, which is denoted as "XoT (only PE)', By removing the verification module, the frame- work, denoted by 'XoT (w/o verification) is no more capable of rechecking the answer thus cannot
This suggests the necessity of employing various reasoning methods in our framework.
The verification module facilitates seamless switching between iterations. We here explore how helpful the active and passive verifi- cations are. Figure 7 illustrates the performance comparison when considering different verification aspects. If solely depend on passive verifica- tion, only 2.43% of the PoT results and 24. 18% of the E results are decmed "incorrect" and subse- quently advanced to the next iteration. However,
Verification

---

GSM8K
SVAMP
AQuA
Algebra
GSM-hard
AddSub
SinglcOP
SingleEQ
MultiArith
Averuge
#Tokens
XoT
83,3p.s
83.6n.6
61.70s
89.9a.
63 4a.5
90.5p.4
94.3
97.70.1
97.3p.3
84.63
4.5k
Vote
82,402
84.70.
55.619
79.7a.5
61.31.1
89.4g.u
94.40.1
97.20.1
98.5u.2
82,59
5.4k
Table 7: Comparison between XoT and Majority Voting. XoT outperforms the majority vote approach in a more efficient manner, yielding an average gain of 2.04 with a reduction of 16.7% 6.7% in token count. # #Tokens denotes the average number of tokens consumed for case (including prompts, question and response).
ACC
ACC + refine
by asking the model to rethink on its previous re- sponse, serving as good alternative for the reason- ing module in XoT. Specifically, before moving on to another method at each iteration, we allow the model to first make self refinement its current approach, making the best use of current method.
CoT
80.4
81.7
PoT
76,9
76,9
EoT
64.1
66.5
XoT 82.7
84.5
Table 8: Results of adding self-refinement within rea-
soning module on GSM8K test set.
Inspired by previous work (Madaan et al., 2023), after reasoning with one method for the first time, we require the model to analyze its response line- by-line and summarize several advice to mitigate the potential errors. Then, the model answers the question for second time in the same method, with the summarized advice as a hint. After that, we verify the results produced by the second round and determine whether to switch to another method.
perform iterative attempts to switch methods. This leads to performance degradation of 4.9% and 2.9% GSM8K and SVAMP respectively.
6.2 Comparison with Majority Voting
We additionally conduct experiments involving the majority vote of three distinct methods. The vote based on three answers generated by three methods (one answer per method). As shown in Table 7, tak- ing the majority vote of the three methods achieves 82.59 on average, while XoT achieves better per- formance at 84.63. Additionally, we observe that the majority vote fails datasets containing ques- tions that align exceptionally well with specific method. Specifically, the majority vote achieves 79.73 on Algebra, while XoT achieves 89.94.
To achieve the iterative refinement in CoT, we follow Zheng et al. (2023) to progressively hint the model with the answers generated before. For PoT and EoT, we follow the released self-refinement prompts from Madaan et al. (2023). The re- sults are shown in Table 8. We only allow the model to think twice using cach prompting method. Though adding only one round of refinement yields marginal improvement within each single method, their collaboration contributes to more significant improvement under XoT framework.
The majority vote needs to execute all three methods to reach an answer, while XoT will stop when the answer passes the verification. We cal- culate the total token count #total_token prinintttokeen + #output_tokeet token 2, according OpenAl's pricing policy?. As shown from the ta- ble, XoT is able achieve higher performance with a lower budget, exhibiting a reduction of 16.7% in expenses. The token count includes all the in- context examples used and is averaged across the number of the total questions in 9 datasets.
Conclusion
We propose XoT, integrated problem solving framework that utilizes diverse reasoning thoughts to prompt LLMs. XoT integrates planning, rea- soning and verification into a unified framework. enabling the model to explore multiple methods based on the active and passive verification of the solutions. We conduct extensive experiments on 10 math reasoning datasets to thoroughly evaluate the ad advantages of each module and showcase the effi- cacy of our proposed approach. Further results also show that the design ethos of XoT can generalize to logic reasoning domain. We consider its gen- cralisation to more diverse tasks as a compelling avenue for future exploration.
6.3 Self-refinement
The design principle underlying XoT is its adapt- able capability to switch methods, allowing for smooth integration with research aimed improv- ing individual methods. The line of itcrative refine- ment methods enhances the model performance
pectps://openai.com

---

Limitations
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Douglas Eek, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CORR, abs/2204.00311..
We acknowledge that our approach falls short on casier and more straightforward datasets where dif- ferent methods exhibit limited complementary re- lations. Our current approach relies on the avail- ability of diverse prompting methods for reasoning tasks. Further research is required to explore new problem solving methods for general reasoning tasks. Moreover, we observe that our method works better larger base models. Although different reasoning methods do exhibit notable complemen- tarity on smaller models, the inherent potential is not yet fully unleashed current XoT design.
Ethics Statement
The data used in our work all comes from public dataset, and our proposed method can be further integrated with other methods. Our work is confor- mant to ACL Ethics Policy.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.1433
Acknowledgements
We would like to thank the anonymous reviewers for their valuable suggestions and feedback. This work was supported by the National Natural Sci- ence Foundation of China (No. 62236004 and No. 62022027).
Edward A. Feigenbaum and Julian Feldman. 1963. Computers and thought.
Yao Fu, Hao Peng. Ashish Sabharwal, Peter Clark, and Tushar Khot. 2022. Complexity-basee prompting for multi-step reasoning. CORR, abs/2210.00720.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. PAL: program-aided language models. CoRR, Abs/2211.1045
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi. Martin Riddell, Luke Benson, Lucy Sun, Eka- terina Zubova, Yujie Qiao, Matthew Burtell, David Peng. Jonathan Fan, Yixin Liu, Brian Wong. Mal- colm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq R. Joty, Alexander R. Fab- bri, Wojeiech Kryscinski, Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. 2022. FOLIO: natu- ral language reasoning with first-order logic. CoRR, abs/2209. .00840.
Joy He- Yueya. Gabriel Poesia, Rose E. Wang, and Noah D. Goodman. 2023. Solving math word prob- lems by combining language models with symbolic solvers. CoRR, abs/2304.09102.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from rea- soning for numerical reasoning tasks. CoRR, abs/2211.12588.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Erie Tang, Dawn Song. and Ja- cob Steinhardt. 2021. Measuring mathematical prob- lem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. CoRR, abs/2304.05122

---

Carl Hewitt. 1969. PLANNER: A language for prov- ing theorems in robots. In Proceedings of the 1st International Joint Conference on Artificial Intelli- gence, Washington, DC, USA, May 7-9, 1969, pages 295-302. William Kaufmann.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song- Long-Chun Zhu, and Jian- feng Gao. 2023, Chameleon: Plug-and-play compo- sitional reasoning with large language models. CoRR, abs/2304.09842.
Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doka, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 523-533. ACI..
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang. 2022. A survey of deep learning for mathematical reasoning. CoRR, abs/2212. 10535.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. CoRR, abs/2303.17651
Shima Imani. Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathematical reasoning using large language models. CoRR, abs/2303.05399 05.399..
OpenAl. 2023. GPT-4 technical report. CORR, abs/2303.08774.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurIPS.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NL. models really able to solve simple math word problems?
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL- 2021, Online, June 11, 2021, pages 0880-2094.20 Association for Computational Linguistics.
Rik Koncel-Kedzionsk,, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Ezzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Trans. Assoc. Comput. Linguistics, 3:585-597.
Rik
Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. 2016, MAWPS:
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. REFINER: reasoning feedback on in- termediate representations. CoRR, Ahss2234401194
A math word problem repository. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Son Diego California, USA, June 12-17, 2016, pages 1152-1157. The Association for Computational Linguisties.
Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, and Lifu Huang. 2023. The art of SOCRATIC QUESTIONING: zero-shot mul- timodal reasoning with recursive thinking and self- questioning. CORR, htss/3305.1999)
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. In NeurlPS.
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang. and Huajun Chen. 2022. Reasoning with language model prompting: survey. CoRR, abs/2212.09582
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wa, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quan- titative reasoning problems with language models. In NewrIPS.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In Proceedings of the 2015 Conference Empirical Methods in Natural Lan- guage Processing. EMNLP 2015, Lisbon, Portuaal, September 17-21, 2015, pages 1743-1752. The As- sociation for Computational Linguistics.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen, 2022. On the advance of making language models better reasoners. CoRR, abs/2206.02336.
Subhro Roy, Tim Vieira, and Dan Roth. 2015, Rea- soning about quantities in natural language. Trans. Assoc. Comput. Linguistics, 3:1-13.
Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: autonomous agent with dynamic mem- ory and self-reflection. CoRR, abs/2303.1 13366.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume I: Long Papers, pages 158 158-167. Association for Computational Linguistics.
Bas Steunebrink, Pei Wang. and Ben Goertzel, ed- itors. 2016. Artificial General Intelligence 9th In- ternational Conference, AGI 2016, New York, NY, USA, July 16-19, 2016, Proceedings, volume 9782 of Lecture Notes in Computer Science. Springer.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. CoRR, 133771.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hin prompting improves reasoning in large language models. CoRR, abs/2304.09797.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet. Quoe Le, and Ed H. Chi. 2022. Least-to-most prompting enables complex reasoning in large language models. CoRR, abs/2205.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inam, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Di- ana Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog, Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan. Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 .09288.
A Further Analysis
A.1 Generalisation to logical domain
We analyze the generalisability of XoT framework to logical reasoning domain. One recent work (Yao et al., 2023) proposed LogicLM to solve logical reasoning questions using First Order Logic ex- pressions and executed them in external symbolic reasoners. Following LogicLM, we design similar formal language expressions to represent First Or- der Logic and conduct experiments on FOLIO (Han et al., 2022), an expert- written, logically complex and diverse dataset for natural language reasoning. Our findings in Table 9 suggest that different meth- ods in logical domain also show strong complemen- tarity, achieving 77.45% under oracle setting. After involving the verification module, XoT performs at 62.75% on the validation set of FOLIO. These results underscore the applicability of XoT as a general problem solving framework.
expert-written,
Danqing Wang and Lei Li. 2023. Learn from mistakes through cooperative interaction with study assistant, CoRR, hbb/22355.1889)
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. CORR, 101/2203.1111....
Method
FOLIO ACC
CoT
58.82
FOL 42.65
Oracle
77.45
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 845-854. Association for Computational Linguistics.
XoT
62.75
Table 9: XoT performance on logical reasoning task FOLIO validation set. Normal text reasoming and for- mal language FOL complement to each other under oracle setting and XoT framework.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS.
A.2 Experiments on other models
We further assess the performance of XoT across various base models, such as Llama-2 series (Tou- vron et al., 2023b). The results are shown in Ta- ble 10, and we illustrate the performance scaling curve in Figure 8. With less capable models, dif- ferent prompting methods still demonstrate strong complementarity under oracle setting. Our obser- vations suggest that smaller models tend to yield suboptimal results, likely due to the unbalanced performance across different reasoning approaches
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths. Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRR, abs/2305.10601.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Junxian He, and Qizhe Xie. 2023. Automatic model selection with large language models for reasoning. CoRR, abs/2305.14333. 14333.

---

verification will determine whether use the cur- rent solution change another method.
80
50
Oradi
XoT
Viec
Input
54
Planning
40
Reasoning
failed
Sua---ontt- apt-3.--uzbo
Refinement
Figure 8: Performance scaling curve on different base models. The performance is averaged across the four datasets shown in Table 10.
Verification
passed
and the models' limited capability for active verifi- cation. This limitation inhibits the model's ability timely switch between methods. However, as the model's size increases, XoT consistently shows its strength across the datasets.
Output
Figure 10: Self refinement can be integrated in the XoT framework. The dashed block indicates the reasoning module with the inclusion self refinement. Within each self refinement process, the model repeatedly exploits the same method.
A.3 Proportion of XoT
Figure 9 illustrates the proportion different meth- ods that XoT selects as the final answers. On GSMSK, 56.7% questions end up being solved with POT, while 28.3% are tackled by EoT. The remaining 15% is left for CoT to solve.
C Examples
In this section, show the input and output examples of each module in XoT. Full prompts are available in public Github repository: https: //gi thub. com/ wenggiaoliu/Xool For ËoT, we use sympy library to solve the linear equations.
56.7%
PoT
Bel
CoT
15.0%
28.3%
Figure 9: The proportion of different methods that XoT finally chooses as the answer on GSMSK.
B XoT with self refinement
We here offer the details of how we combine it- erative self-refinement with XoT framework. As shown in Figure 10, the self refinement process be integrated in the reasoning module, where the dashed line indicates rethinking using the same method. When the desired number of self refine- ment iterations is reached, the generated solutions will proceed to the verification module. Then the
hottps://ww.sst sympy.orpgyg... .org/

---

Method
GSM8K
GSM-hard
Algebra
SVAMP
Average
CoT
14.3
3,8
17.6
33,3
17.23
PoT
10.2
8.3
14.9
32.8
16.52
E.T
8.0
5.0
16.2
23.2
13.12
LLaMA2-7B
Oracle
24.6
14.6
32.0
58.9
32.50
XoT
13.5
5.9
19.4
35.2
18.50
Vote
13.9
7.0
19.8
37.4
19.54
CoT
27.5
8.5
20.3
40.3
24.13
PoT
26.4
22.7
26.1
50.4
31.40
EoT 14.5
12.1
32,9
29.9
22.33
LLaMA2-13B
Oracle
46.1
32.2
49.6
68.7
49.13
XoT
30.2
17.4
30.2
30.98
Vote
28.6
18.9
31.5
51.0
32.50
CoT 59.1
27.1
43.2
75,2
51.16
PoT
$2.3
35.6
73.6
51.14
EoT 31.0
25.0
63.5
44.4
40.99
LLaMA2-70B
Oracle
78.3
59.3
78.8
89.0
76.36
XoT $7.9
43.8
64.0
77.0
60.68
Vote
58.6
38.9
56.3
80.0
58.45
CoT 77.9
44.4
82.0
73.8
69.52
PoT
72.5
59.5
68.70
EoT
41.3
36.1
65.3
47.6
47.58
gpt-3.5tturb-iinstruct
Oracle
90.1
73.7
95.1
90.2
87.27
XoT 77.8
58.8
90.5
80.1
76.80
Vote
77.9
58.2
85.6
81.5
75.80
Table 10: XoT performances with various models.
Input:
You need to choose the best method for the given question. Each method has its own strength. The methods are described as follows:
Python Program: This method generates a Python program that can solve the given question. It takes in the question and possible context and produces program. Normally, consider using this method when the questions and contexts involve forward reasoning, such arithmetic operations multiple numbers, when the questions involve complex logical operations, such "if-else" statements.
System of linear equations: This method builds a math model and generates a system of linear equations that contains the answer as an unknown variable. Normally, we consider using this method when the questions and contexts involve an unknown variable that must be used build an equation, especially when the question can be better modeled with abstract mathematical declarations, or when the unknown variable appears at the beginning of the questions and needs backward reasoning to solve. Below are some examples that choose the most appropriate method for the math word problems. (Examples)
Question: Alyssa, Keely, and Kendall ordered 100 chicken nuggets from fast-food restaurant. Keely and Kendall each twice as many Alyssa. How many did Alyssa eat?
Method:
Output:
System of linear equations
Table 11: Example of planning module.

---

Input:
(Examples)
Question: Alyssa. Keely, and Kendall ordered 100 chicken nuggets from a fast-food restaurant. Keely and Kendall each ate twice as many Alyssa. How many did Alyssa eat?
System of linear equations: (Do not simplify)
Output:
% Assume Alyssa ate x chicken nuggets:
alyssa
% Alyssa, Keely, and Kendall ordered 100 chicken nuggets from fast-food restaurant: total 100
alyssa + keely kendall total
% Keely and Kendall each ate twice as many as Alyssa:
keely = 2 alyssa
kendall = 2 alyssa
% How many did Alyssa eat? The answer is alyssa:
ans alyssa
Table 12: Example of reasoning in EoT.
Program:
nuggets_tooks total 100
nuggets_kedde = 2 # nuggets_atd suggets_alyssa
nuggets jengets_kendall 2 nuggets_alyy
nuggets_alyy (nuggets_ total nuggets nuggets_keelyy nuggets / ans wainnnuggggglaeeee
Python Interpreter:
NameError: name nuggets_alysse is not defined
Table 13: Example of failed passive verification PoT. The Python interpreter throws error, indicating the solution does not pass the passive verification.
Input:
[Examples]
Question: Alyssa, Keely, and Kendall ordered 100 chicken nuggets from fast-food restaurant. Keely and Kendall each ate twice many Alyssa. How many did Alyssa eat?
alyssa 20
total 100
keely = 40
kendall 40
# Assertion
Output:
The total number of chicken nuggets should be equal to the sum of the nuggets Keely, Kendall and Alyssa ate. The number of Keely and Kendall each ate should be twice as many as Alyssa. assert total == alyssa keely + kendall
assert kecly == 2 alyssa and kendall = 2 alyssa
Table 14: Example of an active verification on an EoT solution.

---

Input:
(Examples)
Question: Alyssa, Keely, and Kendall ordered 100 chicken nuggets from a fast-food restaurant. Keely and Kendall each ate twice as many as Alyssa. How many did Alyssa eat?
alyssa = 20
total 100
keely = 40
kendall 40
# Assertion
Output:
# The total number of chicken nuggets should be equal to the sum of the nuggets Keely, Kendall and Alyssa ate. The number of Keely and Kendall each ate should be twice as many Alyssa. assert total alyssa keely + kendall
assert keely * alyssa and kendall = 2 * alyssa
Table 15: Example of an active verification EoT solution.


Reviews:
[{"review_id": "6Nf2D3toTb", "reviewer": "Reviewer_JpnC", "summary": "", "questions": "1. Can you provide some insights on the planning module. What \n\n2. in the active verification, how often does the verification successfully predict the correctness of the answer.\n\n3. can you also provide the performance of XoT with active verification in Table 3? \n\n4. what happens if the verification module considers the answers from all 3 prompting methods as incorrect?\n\n5. can you provide a distribution of the iterations to answer questions?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We provide specific responses and clarifications as follows.\n\n> Reason 1: Clarification for EoT\n\nSorry for the confusion on the naming of EoT. We name prompting methods that utilize equations to solve problems as EoT, in line with CoT and PoT. To address your questions:\n(1) Yes. EoT serves as a minor contribution. In our work, we employed distinct prompts in comparison to the concurrent work (He-Yueya et al., 2023). \n(2) Due to constraints on the page limit, we briefly introduced EoT in section 3.1 and showed its format in Figure 3. For a more direct illustration, we included a detailed example of EoT in Table 9 of the Appendix. We will add more details about EoT in the main body of the paper.\n\n> Reason 2: EoT on 'MATH' dataset\n\nWe acknowledge the limitations of the EoT approach in addressing complex mathematical problems, e.g. on the MATH dataset. We did not include EoT as a method option, because most questions in MATH cannot be solved with an equation system, such as questions related to geometry: \n```\nUsing the side lengths 2,3,5,7, and 11, how many different triangles with exactly two equal sides can be formed?\n```\nThe above example inherently lacks the requirement for an unknown variable within an equation system to arrive at the answer.\n\nIt is worth emphasizing, however, that such an exclusion does not in any way compromise the efficacy of the overall XoT framework. The evolving research community continues to produce novel prompting methods, which could potentially excel in the intricate subdomains of the MATH dataset and be further integrated in XoT. As designing new reasoning methods for specific datasets is out of the main scope of this work, we regard this exploration as a promising avenue for future endeavors.\n\n> Reason 3 & Question 5: Add majority vote as baseline & Distribution of the iterations needed\n\nFor the convenience of response, we combine questions that share similar insights.\n\nThanks for your suggestion! In response to your recommendation, we additionally conduct experiments involving the majority vote of three distinct methods. Our findings indicate that XoT outperforms the majority vote approach in a more efficient manner, yielding a gain of 2.04 with a reduction of 16.7% in token count.\n\n|               | Average ACC | #token / case |\n|:-------------:|:-----------:|:-------------:|\n| Majority vote |    82.59    |     5.4k      |\n|      XoT      |    84.63    |     4.5k      |\n|     Delta     |    +2.04    |    -16.7%     |\n\nCaption: #token / case denotes the average number of tokens consumed for one case (including prompts, question and response).\n\nWe’ll add detailed results of each dataset in the experiment section. The vote is based on three answers generated by three methods (one answer per method). For a quick preview, on 9 datasets in Table 2, taking the majority vote of the three methods achieves 82.59 on average, while XoT achieves better performance at 84.63. Additionally, we observe that the majority vote fails on datasets containing questions that align exceptionally well with a specific method. Specifically, the majority vote achieves 79.73 on Algebra (most questions require equations to solve), while XoT achieves 89.94. \n\nThe majority vote needs to execute all three methods to reach an answer, while XoT will stop when the answer passes the verification. We calculate the total token count as ``#total_token = #input_token + #output_token * 2``, according to OpenAI’s pricing policy. As shown from the table, XoT is able to achieve higher performance with a lower budget, exhibiting a reduction of 16.7% in expenses. The token count includes all the in-context examples used and is averaged across the number of the total questions in 9 datasets.\n\n- _“can you provide a distribution of the iterations to answer questions?”_\n\nPer your recommendation in Question 5, we find it would be very helpful to add a distribution of the iterations for further efficiency analysis. If we consider a round of planning-reasoning-verification as an iteration, XoT takes 1.46 iterations on average to reach an answer on GSM8K. In specific, 68.8% of the questions are answered in the first iteration, 15.8% in the second and 15.4% in the third iteration. We will integrate these analyses into section 5.2.\n\n> Reason 4 & Question 3: Active verification on 'MATH' dataset\n\nWe will add more discussion on the active verification on MATH dataset in the revision. In general, we find adding active verification does not provide expected improvement on MATH dataset. The possible reason is that gpt-3.5-turbo model is not capable enough to apply assertion check on difficult questions in MATH. For example, it simply repeats the statements present in the provided code, which weakens the necessity of the active verification process. Since it can be mitigated by replacing a more capable backbone (e.g., gpt-4), we did not include the detailed discussion in the paper due to the page constraints. Moreover, 60.0% of the programs generated by PoT contain runtime errors - very few of them can proceed to active verification, limiting their feasibility for active verification.\n\nPer your request, we additionally add the active verification on the MATH dataset. Due to the time and budget limit, we conduct experiments on a random subset of 1000 out of 5000 examples on MATH. We try both gpt-3.5-turbo and gpt-4 for active verification on the PoT reasonings generated by gpt-3.5-turbo. With gpt-3.5 verifying, the false negative rate increases intensively (28.3) which negatively impacts the overall performance. Verifying with gpt-4 realizes a balance between the tradeoff of False Positive Rate (FPR) and False Negative Rate (FNR), leading to an increase in the overall XoT performance (outperforming CoT at 37.2, PoT at 26.5). With the availability of gpt-4, we will add more detailed results on each module in the experiments.\n\n|                    | Verification ACC  | FPR &#8595; |  FNR &#8595; | Overall XoT ACC |\n|:------------------:|:---------:|:------:|:-----:|:-----------------------:|\n| passive            |   83.8    | 20.68  |  3.77 |          43.4           |\n|   + gpt-3.5 active |   83.8    | 11.84  | 28.30 |          42.5           |\n|   + gpt-4 active   |   86.2    | 14.42  | 12.08 |          **44.4**           |\n\n Caption: ‘Verification ACC’ denotes how well the verification module specifies the correctness of the answers. 'Overall XoT ACC' denotes the overall performance with the verifications involved in the whole framework.\n\n> Question 1: Insight on the planning module\n\nThe design ethos of the planning module is for **robustness** and **efficiency**, instead of a huge performance boost.\n\n(1) For robustness, although different fixed method orders demonstrate their respective advantages across various datasets, none of them excel across all datasets (Table 4). The planning module can dynamically adjust the execution order based on different questions, which ensures a more consistent and robust performance on average\n\n(2) For efficiency, the strategic prioritization of the most promising method allows XoT to arrive at answers with fewer iterations. Specifically on GSM8K, XoT needs 1.46 iterations on average in comparison with 1.58 iterations with the fixed EPC(EoT->PoT->CoT) order. The efficiency is also indicated in the token count analysis above. We will add the clarification into the analysis section accordingly.\n\n> Question 2: how often does the verification successfully predict the correctness of the answer? \n\nIn the last two rows of Table 5, we present the accuracy of each verification module. For PoT, the verification successfully predicts the correctness of 77.9% cases, while it achieves 81.2% for EoT answers. In addition, as indicated in Line 453 - 456, although marginally affecting verification accuracy, the inclusion of the active verification module plays a pivotal role in mitigating an initially alarming false positive rate, which contributes more to the overall performance.\n\n> Question 3: Please see the response of [Reason 4 & Question 3]\n\n> Question 4: What if all verifications fail?\n\nIf the verifications fail on the first two methods, XoT will automatically choose the answer of the third method. That is, we do not apply verification on the last method. \n\n> Question 5: Please see the response of [Reason 3 & Question 5]\n\nWe appreciate your kind reminder on the missing references, we will add them in the related work! We would be very grateful if you could reassess our paper based on our response and let us know if you might find our work more positive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the paper is well written and easy to follow.\n2. the motivation to combine different prompting techniques and perform verification is clear.\n3. the experiments are thorough and clearly support the effectiveness of the proposed method.", "reasons_to_reject": "1. On line 056- 061 the author mentioned they name an existing prompting method as EOT. On line 126-127, the authors claim to introduce EOT. However, the authors didn't clearly explain the details of EOT before the experiment section. My question is: (1) is EOT your new contribution? (2) if so, can you add details of the method? (3) If not, can you revise line 125 - 127?\n\n2. The proposed verification method is limited and EOT is limited to some math problems. On table 3 all the datasets are verified with only passive verification and the authors didn't provide EOT performance.\n\n3. Since this is a assemble method, stronger baselines should be considered. such as self-consistency chain of thought. What happens if you use take majority vote on three different prompting methods?\n\n4. based on table 5 active verification can reduce the false positive rate and improve the performance but why you only use passive verification in table 3?", "questions_to_authors": "1. Can you provide some insights on the planning module. What \n\n2. in the active verification, how often does the verification successfully predict the correctness of the answer.\n\n3. can you also provide the performance of XoT with active verification in Table 3? \n\n4. what happens if the verification module considers the answers from all 3 prompting methods as incorrect?\n\n5. can you provide a distribution of the iterations to answer questions?", "paper_topic_and_main_contributions": "The authors propose XOT, a iterative framework that consists of three modules: planning, reasoning and verification to improve the LLM reasoning capability on mathematic problems. At each iteration, the planning model proposes a method, the reasoning model generates a solution based on the planning module and the verification verifies the solution. If the solution is incorrect the process is repeated again. XOT also assembles the three different prompting methods: COT, POT and EOT.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, Karthik Narasimhan\n\nThe Art of SOCRATIC QUESTIONING: Zero-shot Multimodal Reasoning with Recursive Thinking and Self-Questioning. Jingyuan Qi, Zhiyang Xu, Ying Shen, Minqian Liu, Di Jin, Qifan Wang, Lifu Huang"}}, {"review_id": "h6V9yXlkRT", "reviewer": "Reviewer_hMke", "summary": "", "questions": "1) How do you run the generated EoT program? I do not find any implementation details.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and suggestions. We provide specific responses and clarifications as follows.\n\n> Reason 1: The ensemble improvement is not surprising.\n\nWe’ll add more discussion on the difference between XoT and ensemble methods in the introduction section. While XoT builds upon various reasoning methods, our approach is similar to but different from the ensemble methods. In traditional ensemble methods such as majority voting (or self-consistency), each sample is executed multiple times following the same procedure. In contrast, XoT chooses different execution processes for different questions by prioritising the most possible method, which represents a more nuanced cognitive process akin to human reasoning. \n\nAdditionally, we add the effectiveness and efficiency analysis between XoT and majority vote. Our findings indicate that XoT outperforms the majority vote approach, yielding a gain of 2.04 with a reduction of 16.7% in token count. A detailed description of the results is available in the response to [Reason 3 & Question 5] for Reviewer JpnC.\n\n> Reason 2: Insights on the planning module\n\nThanks for your suggestion on the efficiency analysis, we will include it in the Analysis section! The design ethos of the planning module is for **efficiency** and **robustness**, instead of a huge performance boost. \n\n(1) For **efficiency analysis**, the planning module facilitates XoT to reach the final answer in fewer iterations by always starting from the most possible method. Specifically on GSM8K, XoT needs 1.46 iterations on average in comparison with 1.58 iterations with the fixed EPC order (EoT->PoT->CoT, the best performing fixed order). Specifically, 68.8% of the questions are resolved in the first iteration with XoT, as opposed to 57.2% when employing the fixed EPC order.\n\n(2) For robustness, although different fixed method orders demonstrate their respective advantages across various datasets, none of them excel across all datasets (Table 4). The planning module can dynamically adjust the execution order based on different questions, which ensures a more consistent and robust performance on average. \n\n> Reason 3: Analysis of active verification\n\n- _\"Active verification seems to decrease the overall accuracy.\"_\n\nIt appears that there might be a slight misunderstanding regarding the relationship between accuracy and active verification. We’d like to take this opportunity to clarify that the active verification leads to **overall performance gain**. We provide the analysis of the active verification in Line 453-462. Specifically, in Table 5, the ‘ACC’ indicates how well the verification module specifies the correctness of the answers, while the last column ‘XoT’ indicates the overall accuracy on the dataset. The inclusion of the active verification contributes to +2.3 gain on the overall accuracy. The main effect of the active verification is to decrease false positive rate (FPR), which enables XoT to try another method instead of keeping a false answer that passes passive verification (false positive samples).  We will revise the description for a clearer presentation of the results in Table 5.\n\n- _\"consider showing the results without any passive or active verification in the ablation study\"_\n\nIt is a great suggestion to add more ablations regarding the verification module! Without the verification module, the framework cannot decide when to change to another method. Therefore, we choose the answer of the method suggested by the planning module for each question. We will add the detailed results of each dataset in the Ablation section. For a quick preview, on 9 datasets shown in Table 2, such ablation only achieves 79.33 on average, compared to XoT at 84.63. This indicates the importance of the verifications and method switching within the XoT framework.\n\n> Question 1: EoT implementation\n\nSorry for the confusion on the implementation of EoT. Given the page limit, we present the implementation examples in Table 9 of the appendix on page 12. We will add more details about EoT in the main body of the paper. For hyperparameter details, we use 8 examples for each prompting and use greedy sampling in all runs.\n\nWe appreciate your efforts in reviewing our paper and providing insightful suggestions! We hope our clarification above could provide a comprehensive understanding of our work and address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper proposes XoT, a unified framework that integrates diverse reasoning modules collaboratively with planning and verification modules to improve the reasoning ability of LLMs.\n2) The paper further explores a new prompting method called Equation-of-Thought (EoT), inspired by Linear Algebra, which forms the math problem as a linear equation system, enabling a more direct approach to math reasoning.\n3) Experimental results on 10 math datasets show XoT is better than any single prompting method.", "reasons_to_reject": "1) The ensemble of three different prompting strategies leads to better performance is not surprising. \n2) The planning module in XoT seems not to lead to significant gains compared with verification only. Adding efficiency analysis might be needed to further illustrate the benefits of the planning module. \n3) Active verification seems to decrease the overall accuracy. A deep analysis of this part might be needed. The authors may also consider showing the results without any passive or active verification in the ablation study.", "questions_to_authors": "1) How do you run the generated EoT program? I do not find any implementation details.", "paper_topic_and_main_contributions": "The paper proposes a problem-solving framework called XoT that utilizes large language models (LLMs) with different prompting methods, such as Chain of Thought (CoT), Program of Thought (PoT), and Equation-of-Thought (EoT), to improve math reasoning tasks. XoT selects the most suitable method for each question and iteratively executes them, actively checking the validity of generated answers and incorporating feedback from external executors. The proposed approach is evaluated on 10 popular math reasoning datasets, demonstrating its effectiveness and the strengths of each module. XoT allows for dynamic switching among different prompting methods, providing a unified framework that integrates diverse reasoning thoughts collaboratively. The results suggest that this approach complements recent improvements in single reasoning methods and opens up new possibilities for math problem-solving.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ye8GipmLpZ", "reviewer": "Reviewer_wiK8", "summary": "", "questions": "A. Some details of the planning method are missing. In your prompt, you introduce just PoT and EoT, but in Table 4 there is CoT. And it seems that EoT+PoT+CoT could achieve a much more desirable performance (84.29 v.s. 84.63). Can it be understood that simply combining several prompting methods will achieve good results? Why don't you try other orders, such as CoT+EoT+PoT or CoT+PoT+EoT?\nB. It is obvious that combining various methods can bring improvement, but where this improvement comes from is still unknown. Is it because CoT, PoT, and EoT are good at different types of mathematical problems, or just because the model is allowed to think more times? Increased analysis in this aspect will help.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive feedback. We provide specific responses and clarifications as follows.\n\n> Reason 1 & Question A: Clarification on the ablation results of the planning module\n\n- _\"Can it be understood that simply combining several prompting methods will achieve good results?\"_\n\nSorry for the confusion on the interpretation of 84.29 v.s. 84.63. Regarding the result, we would like to clarify that 84.29 (EoT->PoT->CoT) represents the ablation result of the planning module. In other words, 84.29 is achieved with both reasoning and verification modules in XoT, which demonstrates the effectiveness of these designs in our method. \n\nThe main difference between 84.29 and 84.63 is the planning module, which can dynamically determine the execution order of different prompting methods. Specifically, the design principle of the planning module is for **robustness** and **efficiency**, instead of a huge performance boost.\n\n(1) For robustness, although different fixed method orders demonstrate their respective advantages across various datasets, none of them excel across all datasets (Table 4). The planning module can dynamically adjust the execution order based on different questions, which ensures a more consistent and robust performance on average. \n\n(2) For efficiency analysis, the planning module facilitates XoT to reach the final answer in fewer iterations by always starting from the most possible method. Specifically on GSM8K, XoT needs 1.46 iterations on average in comparison with 1.58 iterations with the fixed EPC order (the best performing fixed order). We will add the clarification into the analysis section accordingly.\n\n- _“Why don’t you try other orders?”_\n\nThe initial decision to not select CoT as the first method is mainly due to the absence of a reliable verification module to specify the correctness of CoT answers. We provided the intuition of the verification module design in Line 306-310. In detail, the planning module mainly selects from PoT and EoT as these two methods incorporate external executors that can provide intermediate values and deterministic execution results, which will be used by both passive and active verification. \n\n- _\"the novelty is somewhat lacking\"_\n\nWe appreciate the reviewer's attention to the novelty of our work. While XoT builds on the foundation of various independent reasoning methods, our work provides initial empirical evidence that different prompting methods have complementarity rather than exclusive competencies. The main highlight of our work is to present a new problem solving framework that involves different methods to work together, which has not been explored comprehensively before. \n\nAdditionally, we add the effectiveness and efficiency analysis between XoT and the traditional ensemble method majority vote. Our findings indicate that XoT outperforms the majority vote approach in a more efficient manner, yielding a gain of 2.04 with a reduction of 16.7% in token count. A detailed description of the results is available in the response [Reason 3 & Question 5] for Reviewer JpnC.\n\n> Reason 2: Generalizability to other domains\n\nDue to space constraints, this paper mainly employs math reasoning as a problem solving scenario. XoT represents a new problem solving framework which is not limited to the reasoning methods described in this paper. We noticed a recent work (Pan et al., 2023) proposed to solve logical reasoning questions using First Order Logic expressions and executed them in external symbolic reasoners. We conducted initial analysis on the complementarity of this method (denoted as FOL) and normal CoT. As shown in the table below, these two methods are well complement to each other, performing at 70.44 under the oracle setting. This finding underscores the potential applicability of XoT within the domain of logical reasoning. We leave the further exploration as a promising avenue for future investigation.\n\n| Method | Acc on FOLIO (Han et al., 2022)  |\n|:------:|:----------------------------:|\n| CoT    |            57.84             |\n| FOL    |            45.32             |\n| Oracle |            70.44             |\n\nCaption: FOLIO is an expert-written, logically complex and diverse dataset for natural language reasoning.\n\n> Reason 3: Experiments on other models\n \nWe thank the reviewer for this suggestion, which we believe helps strengthen our results. Per the reviewer’s request, we conduct additional experiments on GSM8K with both text-davinci-003 and text-davinci-002, which are less performing models than gpt-3.5-turbo (text-davinci-002 is trained with supervised fine-tuning w/o RL). The results are shown below:\n\n| Method | text-davinci-003 | text-davinci-002 |\n|--------|------------------|------------------|\n| CoT    | 67.63            | 56.86            |\n| PoT    | 66.41            | 59.82            |\n| EoT    | 50.42            | 42.76            |\n| XoT    | 71.11 (+3.48)    | 64.29 (+4.47)    |\n| Oracle | 84.91            | 78.85            |\n\nWith less capable models, different prompting methods still demonstrate strong complementarity under oracle setting. Specifically, XoT is able to achieve consistent gains (+3.48,+4.47) with both models. In addition, we note that XoT is agnostic to the specific models, as long as the models exhibit the capability to perform CoT reasoning. The determinant influencing XoT performance resides in the complementary nature manifested between the dataset and the prompting methods.\n\n> Question B: where this improvement comes from is still unknown\n\nWe acknowledge the importance to analyze the source of  XoT's improvement. \nThe underlying reason for the improvement indeed aligns with your first viewpoint: different prompting methods are good at different types of questions and they are complementary to each other.\nWe analyzed the complementarity of these methods in section 3.2 and Figure 2. Our oracle analysis provided empirical evidence that various methods have unique strengths on different problems. For instance, the question illustrated in Figure 3 is especially appropriate to be solved with EoT. We will emphasize the source of the improvement in the writing for a clearer presentation.  \n\nWe agree that thinking more times is beneficial for models to achieve better performance. Our empirical findings suggest that XoT could provide further gains over simply thinking more times. We conducted experiments on reasoning with a single method for the same number of iterations and provided comparison in section 6.1 (reasoning ablation) and Figure 6. Although the model is allowed to think multiple times with PoT (denoted as PoT^3), a single method results in limited complementarity (-8.64  than XoT) and overall gain (-4.32 than XoT), which indicates the importance of employing different prompting methods.\n\nWe are grateful for your insight and constructive feedback, and we are committed to refining the paper to present a more comprehensive and accurate representation of our work. We would highly appreciate it if you could reassess our paper based on our response and let us know if you might find our work more positive.\n\nReferences:\n- Pan et al., 2023. LOGIC-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning.\n- Han et al., 2022. FOLIO: natural language reasoning with first-order logic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes an automatic problem-solving framework XoT, utilizing the complementarity of different reasoning thoughts.\n2. This paper incorporates passive and active verification to facilitate the framework.", "reasons_to_reject": "1. The main motivation of this paper comes from the combination of various prompting methods, while from experimental results it seems that simply combining several prompting methods could already achieve promising results (84.29 v.s. 84.63). Therefore, the novelty is somewhat lacking.\n2. CoT, PoT, and other prompting methods are applicable to various reasoning scenarios, while this paper only conducts experiments on the math reasoning datasets, thus the generalizability of the methods remains unknown.\n3. Experiments conducted only use the gpt-3.5-turbo model, hence the inclusion of more LLMs (e.g., davinci from Openai, and other open-source models such as Llama & Alpaca) would be desirable.", "questions_to_authors": "A. Some details of the planning method are missing. In your prompt, you introduce just PoT and EoT, but in Table 4 there is CoT. And it seems that EoT+PoT+CoT could achieve a much more desirable performance (84.29 v.s. 84.63). Can it be understood that simply combining several prompting methods will achieve good results? Why don't you try other orders, such as CoT+EoT+PoT or CoT+PoT+EoT?\nB. It is obvious that combining various methods can bring improvement, but where this improvement comes from is still unknown. Is it because CoT, PoT, and EoT are good at different types of mathematical problems, or just because the model is allowed to think more times? Increased analysis in this aspect will help.", "paper_topic_and_main_contributions": "This paper proposes a problem-solving framework for math reasoning tasks. Specifically, it iteratively selects one of the existing prompting methods and then executes it. At the same time, it actively checks the validity of generated answers using passive and active verification.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes XoT to improve math problem solving by LLMs by assembling a diverse group of reasoning paths with verification. Reviewers agree that while the proposed method is not exceptionally novel, it still presents a nice contribution to improving the reasoning capabilities of LLMs, and the experiments have demonstrated solid results. Based on the new results from the rebuttal, we strongly encourage authors to add experiments for more LLMs and reasoning tasks in their final version.
Decision: Accept-Main