SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As DELI focuses on deliberating over LLM-generated solutions, we aim to initialize a high-quality step-by-step solution for the given question that covers useful math knowledge and arithmetic operators. Therefore, we propose to retrieve relevant problems and solutions as the exemplars, and then utilize the chain-of-thought prompting method [7] to generate the initial solution based on them. Concretely, given a math problem  $p$ , we first retrieve top-  $k$  relevant problems  $C = \{\langle p_i, s_i \rangle\}_{i=1}^k$  from the candidate pool based on question-question matching, where the retriever can be either lexicon-based [79] or dense retrieval models [80]. Then, the retrieved problems with their associated step-by-step solutions, will be employed to compose the input prompt, to elicit LLMs for performing chain-of-thought reasoning. The pattern of the input prompt is denoted as: "You are a helpful assistant for solving math problems in LaTeX format:  $[p_1], [s_1], \dots, [p_k], [s_k], [p]$ ". In this way, LLMs would follow the exemplars to perform step-by-step reasoning, and can also refer to useful math knowledge from them, leading to high-quality initial solutions for deliberation. Note that such a way also supports other prompting methods to initialize solutions.
Figure 2: The overview of our DELI. DELI initializes the step-by-step solution via retrieval-augmented strategy, and then performs iterative deliberation with tool manipulation and chain of thought, respectively.
4.2 Iterative Deliberation
Based on the initial solution, we iterate two types of deliberation procedures, i.e., deliberation with tool manipulation and deliberation with chain of thought, until reaching the stop condition. In deliberation procedures, we adopt specific in-context exemplars to guide LLMs, for checking and correcting the errors in the current solution. Next, we first introduce details of the two deliberation procedures, and then present the stop conditions.
Deliberation with Tool Manipulation Prompting. Since LLMs are prone to make mistakes in numerical calculation, we design the procedure of deliberation with tool manipulation, for seeking help from external tools to address it. Based on our devised interfaces in Section 3.3, we aim to rewrite the current solution into a process that orderly invokes the interfaces to produce the result. In this way, the deliberation procedure is divided into a sequence of steps, where the LLM should select the interface and then invoke it to produce the intermediate result in each step.
First, we construct an instruction introducing the goal and formats of this procedure, and details of available interfaces. For each interface, we not only list its name, arguments and description, but also provide an example to exhibit the usage, e.g., "expand(expression: str)  $\rightarrow$  new expression: str: Expand the expression into a polynomial. For example, expand((x + 1) $^2$ )  $\rightarrow$  x^2 + 2x + 1". Then, we demonstrate several exemplars to guide LLMs to invoke the interfaces. Each exemplar consists of four parts, i.e., a question, a trial, multiple actions, and their outputs. The trial is a step-by-step solution either from the initial stage or the last iteration, which may contain errors requiring correction. Actions are a series of interface invocations derived from the trial, and outputs are the intermediate results by executing the actions, e.g., "Action: solve_eq(2k + 1 = 0). Output:  $[k = -\frac{1}{2}]$ ". Based on the instruction and exemplars, the LLM would be elicited to generate the action in formal language iteratively (i.e., selecting the interface and setting its arguments), then execute it to obtain the output, until reaching the answer. To guarantee the continuity of the deliberation procedure, we set a special token after the generated action, for pausing the generation process and waiting for the result of interface invocation. In the iterative selection-then-execution process, we can deliberate the intermediate steps of the generated solution, and benefit from tool manipulation for accurate computation.
Deliberation with Chain of Thought Prompting. After deliberation with tools, we can obtain the solution consisting of a series of actions to invoke interfaces and their outputs. Next, we further deliberate the solution with the chain of thought to reorganize it into the natural language, which can better leverage the learned textual knowledge from LLMs to recheck it and also improve the readability.
Similarly, we also leverage an instruction with in-context exemplars to compose the input prompt. The instruction is "You have access to both natural language problem solving processes and formal problem solving processes, but there may be errors within them. You need to learn the correct methods in order to better solve problems." to introduce the goal of the deliberation procedure. All
the exemplars are composed of four components, i.e., a question, a given solution, the verification, and the revised solution. The given solution is the last natural language solution that is either the initial solution or the solution from the last deliberation iteration with chain of thought, and the verification is the formal language solution from the last deliberation procedure with tool interfaces. The revised solution is the result of integrating the two types of solutions into the chain-of-thought manner, where the errors and unreasonable steps have been corrected. Guided by the exemplars, LLMs would deliberate the intermediate steps from in-context solutions, and generate a new natural language solution. Besides, as there are often inconsistent intermediate computation results in the in-context solutions, we also add an instruction to elicit LLMs to trust more on the result from tool manipulation, i.e., "If the computed result in the verification differs from the computed result in the given solution, the computed result in the verification must be used as the standard".
Stop Conditions of Iteration. The two deliberation procedures would be alternated multiple times, where the solution might be iteratively revised and improved. To control the cost, we set the stop conditions of the iteration process. First, once the solution of the new iteration is the same as the last one, the iteration stops, since the iteration has converged. Second, if the answers to the two deliberation procedures are consistent, we will also stop the iteration. Third, if we have reached the maximum number of iterations, the answer from the last deliberation procedure with tool manipulation will be regarded as the final answer, as the procedure can better solve computation subproblems, leading to a more accurate answer.
5.1 Main Experiments
Evaluation Datasets. In addition to CARP, we collect 6 existing computation-intensive math datasets for evaluation, including Algebra, Prealgebra, Counting & Probability (CP) and Number Theory (NT) from MATH [15], and GK-Cloze (GKC) and SAT-Math (SAT) from AGIEval [13]. The datasets involve multi-step reasoning and computation with knowledge ranging from middle school to competition level. Details about the datasets are listed in Appendix B
Baselines. We compare our proposed DELI with several competitive prompting methods for LLMs. (1) CoT prompting methods: Random CoT [7] randomly selects exemplars from the training set. Complex CoT [81] samples the most complex problems and their solutions as exemplars. Retrieval CoT retrieves the most relevant problems and solutions from the training set as exemplars. (2) Tool-augmented prompting methods: PAL [26] converts the reasoning process into a Python program and executes it to get the answer. ReAct [32] interleave reasoning and interface invocations multiple times to get the answer, which is a basic component of DELI. (3) Iterative prompting methods: Learning to Program (LP) [74] aims to iteratively learn solutions from training sets to guide LLMs in solving similar problems based on in-context learning. Learning to Program (LP) [74] employs iterative learning from training sets to steer LLMs in context-based problem solving. As a variant of our framework, Iterative CoT integrates the existing CoT solution and self-generated feedback into a refined solution. Similarly, Iterative ReAct refines interface invocations using pre-existing ones and self-feedback.
Implementation Details. We employ OpenAI gpt-3.5-turbo (May 2023) API as the solver and reasoning tool and implement the computation tool based on SymPy [78]. We set the temperature to 0 and top_p to 1 for determined outputs. To retrieve similar problems, we train a sentence embedding model following SimCSE [80] to index MATH datasets and employ the BM25 algorithm [79] for the CARP dataset. The maximum number of iteration turns is set to 3 for all datasets and iterative methods. For each dataset, we specify the descriptions of interfaces that may be useful to solve the problems in prompts.
We initialize the solution with Retrieval CoT in most datasets. For GK-Cloze and SAT-Math, we initialize the solution with Random CoT, since these datasets only provide few-shot exemplars but not training sets. Following the settings in Zheng et al. [69], the initial solutions of PHP are from Complex CoT in the subsets of MATH (Algebra, Prealgebra, CP, NT), while using the same initial solutions as DELI in other datasets.
Table 4: Results on 7 computation-intensive math reasoning datasets. We copy results of LP from Guo et al. [74]. The best and second-best methods are marked in bold and underlined respectively.
Methods	CARP	Algebra	Prealgebra	CP	NT	GKC	SAT	Avg.
Random CoT	49.39	49.37	55.57	32.91	29.81	14.41	65.91	42.48
Complex CoT	48.06	51.64	53.73	32.91	32.22	-	-	-
Retrieval CoT	63.93	53.75	56.72	33.12	30.00	-	-	-
PAL	40.00	34.29	50.52	35.86	31.30	5.93	47.73	35.09
ReAct	64.11	54.51	54.53	41.77	31.67	16.94	72.27	48.07
LP	-	49.60	52.30	30.20	29.80	-	-	-
PHP	61.68	54.42	57.86	36.71	35.37	16.94	71.82	47.82
Iterative CoT	61.27	52.74	55.34	33.97	29.81	14.41	69.55	45.30
Iterative ReAct	61.17	53.92	52.12	37.34	32.22	15.25	70.00	46.00
DELI	73.46	59.65	58.32	39.03	33.15	17.80	74.54	50.85
Figure 3: The results of combining DELI with existing CoT methods.
Main Results. Table 4 shows the results of DELI and baselines on the 7 datasets. For the comparison among CoT prompting methods, Retrieval CoT outperforms Random CoT [7] and Complex CoT [81] on average, indicating that involving relevant problems and their solutions is beneficial for answering complex math problems. Augmented with tools, ReAct [32] achieves better average performance than CoT prompting methods, showing the effectiveness of manipulating tools in solving computation-intensive math problems. Besides, ReAct can also perform better than competitive baselines, e.g., PAL [26], LP [74] and PHP [69] on the benchmark. A possible reason is that ReAct adopts our devised interfaces to manipulate tools, which enables it to better deal with fine-grained computation operators. Finally, DELI performs better than all baselines in most cases. In DELI, we incorporate an iterative two-stage deliberation strategy to check and refine the generated step-by-step solutions from the LLM. Such a way is also more effective than just iterating CoT or ReAct, as it can better integrate the manipulation of tools and reasoning in natural language.
Combining with Existing CoT methods. In DELI, we initialize a step-by-step solution by retrieving problems and solutions as exemplars. Actually, our DELI also supports other prompting methods to initialize solutions. We report the performance of combining DELI with different CoT methods on CARP, Algebra, and CP. As shown in Figure 3, DELI can greatly boost the performance of all CoT methods, which demonstrates that our DELI is general to various CoT prompting methods to fix part of their intermediate errors. Among all methods, DELI can improve the performance of Retrieval CoT a lot. It indicates that our adopted retrieval-augmented solution initialization way is more suitable for our deliberation approach.
Impact of Iterative Deliberation Turns. We also study how the performance of DELI changes w.r.t. the maximum iterative deliberation turns. To comprehensively investigate it, we select CARP, Algebra, and Prealgebra for evaluation, and also report the performance of Iterative CoT, Iterative ReAct, Retrieval CoT and ReAct as reference. As shown in Figure 4, the performance of DELI consistently increases w.r.t. the increasing of maximum iteration turns. For comparison, the performance of other
Figure 4: Accuracy of different methods w.r.t. the maximum number of iteration turns.
iterative methods does not always increase, and can even drop with an increase in maximum turns. It demonstrates that LLMs are not easy to be steered to deliberate and correct errors, and DELI is a suitable way that deliberates from the perspectives of reasoning in natural language and tool manipulation, like humans that iteratively check the solution by thinking critically and using tools.
Evaluating Intermediate Reasoning Steps. We evaluate the accuracy of intermediate reasoning steps for different methods via our proposed metrics ExpAcc and Fail@where on 128 challenging problems from CARP that are incorrectly answered by all the methods. As shown in Table 5, DELI achieves better ExpAcc than CoT and ReAct. Besides, the results of Fail@where show that DELI is less declined to generate completely wrong solutions (with lower Fail@first),
Table 5: ExpAcc and Fail@where on a challenging subset of CARP.
Methods	ExpAcc	Fail@where
first	middle	last
CoT	13.91	67.97	22.65	9.38
ReAct	12.58	66.41	29.69	3.91
DELI	18.90	60.16	25.78	14.06
and has a larger percentage of near-correct solutions (with higher Fail@last). It indicates that DELI is able to correct wrong intermediate results by the proposed iterative two-stage deliberation strategy.
First, due to the need for quality control of the problems and the requirement of formulated annotation of intermediate reasoning steps, we retain and annotate relatively small-scale data the dataset size (at the order of thousands), future work should focus on expanding the data sources and increasing the scale of annotation. Second, to assess the tool manipulation ability of LLMs, the dataset primarily includes middle school-level math problems. Higher difficulty levels, such as high school or college-level mathematics, have not been incorporated. Future efforts can involve annotating intermediate steps for more challenging problems and designing tools for advanced math concepts. Third, the experiments conducted with OpenAI API are limited to GPT-3.5 series, as we do not acquire access to GPT-4. Consequently, the evaluations are constrained to the capabilities of GPT-3.5. To provide a more comprehensive assessment, future research can consider expanding the evaluation scope to include a wider range of base models.
In this paper, we proposed CARP, a computation-intensive algebra problem dataset with formulated annotation of intermediate reasoning steps for systematically evaluating LLMs in tools manipulation and math reasoning. Based on experiments in CARP, we found that popular LLMs with chain-of-thought prompting can not solve over half of the problems in CARP, and they are more likely to make mistakes in early steps, leading to wrong answers. To alleviate it, we proposed DELI, a new approach that can deliberate the intermediate reasoning steps with tool interfaces. DELI incorporated two iterative deliberation procedures to check and refine the intermediate reasoning steps of the generated step-by-step solution, from the perspectives of tool manipulation and natural language reasoning. To verify the effectiveness of DELI, we conducted extensive experiments on CARP and 6 other computation-intensive math reasoning datasets. Experimental results have shown that DELI outperforms baselines and can boost the performance of various CoT prompting methods.
This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. L233008 and 4222027. And this work is also partially supported by the Outstanding Innovative Talents Cultivation Funded Programs 2021 of Renmin University of China. Xin Zhao is the corresponding author.
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[2] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[4] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[5] Rohan Anil, Andrew M Dai, Orhan First, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
[6] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXiv, abs/2303.18223, 2023.
[7] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903, 2022.
[8] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. ArXiv, abs/2206.14858, 2022.
[9] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv, abs/2211.09085, 2022.
[10] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt. ArXiv, abs/2301.13867, 2023.
[11] Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang, Cong Liu, and Ji rong Wen. Jiuzhang: A chinese pre-trained language model for mathematical problem understanding. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
[12] Wayne Xin Zhao, Kun Zhou, Beichen Zhang, Zheng Gong, Zhipeng Chen, Yuanhang Zhou, Ji rong Wen, Jing Sha, Shijin Wang, Cong Liu, and Guoping Hu. Jiuzhang 2.0: A unified chinese pre-trained language model for multi-task mathematical problem solving. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
A Interface Definition
We provide interface definitions of our tools in Table 6.
Table 6: Interface definitions of tools. Num. Comp., Eq. Solving, and Expr. Trans. refer to numerical computation, equation solving, and expression transformation, respectively.
Category	Interface	Description
Num. Comp.	calculate(e) → v	Calculate the value v of e.
substitute(e, {c}) → v	Substitute the contextual conditions {c} into e.
Eq. Solving	solve_eq(e) →, {e'}	Solve the equation e to get the solution set {e'} .
solve_ineq(e) → {e'}	Solve the inequation e to get the solution set {e'} .
solvemulti_eq({e}) → {e'}	Solve the system of equations to get the solution set {e'} .
solvemulti_ineq({e})	Solve the system of inequations to get the solution set {e'} .
→ {e'}	Solve the equation e assuming that u is an unknown to get the solution set {e'} .
partial.solve(e, u) → {e'}
Expr. Trans.	expand(e) → e'	Expand e to get e'.
factor(e) → e'	Factorize e to get e'.
collect(e, x) → e'	Collect e based on the symbol x to get e'.
complete_the_square(e) → e'	Complete the square of e to get e'
Thinking	think(l) → l'	Draw a conclusion l' based on the free thought l .
B Details of Evaluated Datasets
We provide details of evaluating datasets in Table 7.
Table 7: Basic information about datasets in evaluated datasets. MS and HS refer to "middle school" and "high school", respectively.
Dataset	Source	Language	Domain	Difficulty	Train	Test
CARP	Ours	Chinese	Algebra	MS	3,410	976
Algebra	MATH	English	Algebra	HS	1,744	1,187
Prealgebra	MATH	English	Algebra	HS	1,205	871
Count. & Prob.	MATH	English	Probability	HS	771	474
Num. Theory	MATH	English	Number Theory	HS	869	540
GK-Cloze	AGIEval	Chinese	Mixture	HS	-	220
SAT-Math	AGIEval	English	Mixture	HS	-	351
To better present the process of DELI, we provide a case study that shows the solving process of DELI on CARP, which is shown in Figure 5. We also report the solution of Retrieval CoT and ReAct in the figure. It is noted that the solution of Retrieval CoT is also the initial solution in DELI.
First, both Retrieval CoT and ReAct make minor mistakes in the solving process. Although following the correct solving idea from relevant solutions, Retrieval CoT struggles with expanding the expression  $(x - 2)(1 - kx) - (2x - 3)(2x + 3)$ , leading to an incorrect intermediate result. Besides, ReAct fails at understanding the condition the expression does not contain the primary form of  $x$ , thus collecting the expression according to a wrong term  $x^2$ . Therefore, both CoT and ReAct can not solve the case individually due to the challenges of computations and reasoning.
DELI iterates over the existing solutions. In deliberation with tool manipulation, the model reviews the existing natural language solution, and invokes interfaces based on the ideas therein. In this case,
Figure 5: Case study of our method with baselines on the CARP dataset. The case is translated into English.
the model invokes interfaces expand and collect in a row to get the correct expanded expression  $(x - 2)(1 - kx) - (2x - 3)(2x + 3) = -kx^2 + 2kx - 4x^2 + 7$  with the help of tools. Then, the model solves the equation derived from the expanded expression and gets the correct answer.
In deliberation with chain of thought, the model reviews both natural language and formal language solutions from the previous iteration and generates a revised CoT solution, which fixes the computation error in the original CoT solution according to interface invocations. In this case, due to the consistent answers between the revised CoT solution and the previous ReAct solution, the iteration terminates. In general cases, the iteration continues until solutions converge or the answer is consistent, or the maximum number of iterations is reached.
D Prompts for Two-Stage Deliberation
We list the prompts for two-stage deliberation on CARP. The prompts are translated into English.
Listing 1: Prompt for Deliberation with Tool Interfaces.
You are ChatGPT, a math problem solver equipped with multiple functions to tackle various math problems. While you may have access to existing problem-solving processes, there is a possibility of errors. Therefore, you need to learn the correct approaches to solve problems more efficiently. You can use the following functions:
calculate(expression: str) -> new_expression: str: Calculate the value of the expression and return it as a string. For example, calculate("34 * 2") -> "68". solve_eq(expression: str) -> newExpressions: list: Solve the equation expression and return the result as a list. For example, solve_eq("3 x + 4 = 1") -> ["x = -1"]. solve_ineq(expression: str) -> new_expression: str: Solve the inequality expression and return the result as a string. For example, solve_ineq("3 x + 4 < 1") -> "x < -1".