Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks
Authors: Wenhu Chen
Xueguang Ma
Xinyi Wang
William W. Cohen
Venue: Accepted by TMLR ()

Abstract: Recently, there has been significant progress in teaching language models to perform step-by-step reasoning to solve complex numerical reasoning tasks. Chain-of-thoughts prompting (CoT) is the state-of-art method for many of these tasks. CoT uses language models to produce text describing reasoning, and computation, and finally the answer to a question. Here we propose `Program of Thoughts' (PoT), which uses language models (mainly Codex) to generate text and programming language statements, and finally an answer. In PoT, the computation can be 
delegated to a program interpreter, which is used to execute the generated program, thus decoupling complex computation from reasoning and language understanding. We evaluate PoT on five math word problem datasets and three financial-QA datasets in both few-shot and zero-shot settings.  We find that PoT has an average performance gain over CoT of around 12% across all datasets.
By combining PoT with self-consistency decoding, we can achieve extremely strong performance on all the math datasets and financial datasets. All of our data and code will be released.

Published in Transactions on Machine Learning Research (10/2023)
Program of Thoughts Prompting: Disentangling Computa- tion from Reasoning for Numerical Reasoning Tasks
Wenhu Chen
wenhuchen @wwaterloo. x93ma wwwaateroo.. on zingi wang@ucshhbb.ooo....ooo edu wcoken @google. com
Xueguang Ma
Xinyy Wang
William W. Cohen
University of Waterloo, Canada
University of Colifornia, Santa Barabra, USA
Google Reseorch, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id= df22Z22Pt
Abstract
Recently, there has been significant progress in teaching language models to perform step-by- step reasoning to solve complex numerical reasoning tasks. Chain-offthoughls prompting (CoT) the state-of-art method for many of these tasks. CoT uses language models to produce text describing reasoning, and computation, and finally the answer to a question. Here we propose 'Program of Thoughts' (PoT), which uses language models (mainly Codex) to generate text and programming language statements, and finally answer. In PoT, the computation generated program, thus decoupling complex computation from reasoning and language understanding. We evaluate PoT on five math word problem datasets and three financial- QA datasets in both few-shot and zero-shot settings. We find that PoT has an average performance gain over CoT of around 12% across datasets. By combining PoT with self-consistency decoding, we can achieve extremely strong performance on all the math datasets and financial datasets. All of our data and code will be released.
be delegated to program interpreter, which is used to execute the
1 Introduction
Numerical reasoning is a long-standing task in artificial intelligence. A surge of datasets has been proposed recently benchmark deep-learning models' capabilities perform comeerical/rrthhmeen widely used benchmarks are based on Math word problems (MWP) (Cobbe al.) 2021 Patel et al. 2021
reasoning. Some
Lu et al.] 2022 Ling et al.] 2017). where systems supposed answer math questions expressed with
natural lext. Besides MWP, some datasets also consider financial problems (Chen et al. 2021b 2022
Zhu
et al.
2021 where systems need to answer math-driven financial questions.
Prior work (Ling et al. 2017 Cobbe et al. 2021 has studied how to train models from scratch or fine-tune models to generate intermediate steps to derive the final answer. Such methods are data-intensive, requiring a significant number of training examples with expert-annotated steps. Recently, Nye et al. (2021) have discovered that the large language models (LLMs) (Brown al., 2020 Chen et al. 2021a Chowdhery al., 2022) can be prompted with few input-output exemplars solve these tasks without any training fine- tuning. In particular, when prompted with a few examples containing inputs, natural language 'rationales', and outputs, LLMs can imitate the demonstrations to both generate rationales and answer these questions. Such a prompting method is latter extended as 'Chain of Thoughts (CoT)' (Wei et al., 2022), and it is able to achieve state-of-the-art performance on a wide spectrum of textual and numerical reasoning datasets.
COT uses LLMs for both reasoning and computation, i.c. the language model not only needs generate the mathematical expressions but also needs perform the computation in each step. We argue that

---

Published in Transactions on Machine Learning Research (10/2023)
Question: In Fibonacei sequence, it follows the rule that each number equal sum of the preceding two numbers. Assuming the first two numbers are and 1, what is the 50th number in Fibonacci sequence?
length_of foffffioaacci___ fibonacciseequence arbonaccissequenan = np.zeros(tength.or
sequence 50
The first number is O, the second number is 1, therefore, the third number 0+1=1. The fourth number is 1+1=2. The fifth number is ++2=3. The sixth number is 2+3=5. The seventh number is 3+5=8. The eighth number is 5+8=13.
fibonacci_ [ecaacccss =
fibonacci_ [etrracccssquuecee[[]]
For in range(3,
..... (Skip 1000 tokens)
fibonacci sequence(] = fibonacci_ [aaaacciissqueecaa--111]
The 50th number is 32,442,268,459.
fibonacci. sequencefi-2]
CoT
ans fibonacci arooaaci_seeuencceffe
PoT
python
32,432,268,,4 459
12,586,269,023
Question: Ketty saves 20000 dollars to the bank. After three years, the sum with compound interest rate is 1000 dollars more than the sum with simple Interest rate. What the interest rate of the bank?
interest_ taretttrttteeee = Symbol(xx)
Assuming the interest rate x. The sum after two years with simple interest rate is 20000 + x 20000 3 20000 + 60000x. The sum after two years with compoud interest rate Is 20000 * (1 + x) A 200000 60000
sum_ in_two _eo__yerr__aa with htt__smmiipeeeeesteeeee interest= 20000 +
interest_ rate 20000 33
hiwo__eers_with_commmeee
_interest 20000 (1
6 +
interest_ interest_ratee"3
20000x^3. The difference can be written 60000x^2 + 20000x^3 1000. In order solve x, we can use the quadratic formula. x (-b +- sqrt(b>2 4ac)) 2a, ..... x
Since compound interest is 1000 more than simple interest.
ans solve(sum sarti_afftt____eeeee aerra_wwtt__cmm feathhhoomoud___eee
sum sum_after after_ strrr_to__w___ [wo__eers__ihe itmpppounddmmdddd_e-mmmm sonrrpound__tteeestttt interest 1000,
(-20000 +- 61 6160) / 120000, x = -0.051333.
CoT
interesttrate rate)
PoT
python
Symly
-0.511333
0.24814
Figure 1: Comparison between Chain of Thoughts and Program of Thoughts.
language models are not ideal for actually solving these mathematical expressions, because: 1) LLMs are very prone to arithmetic calculation errors, especially when dealing with large numbers; 2) LLMs cannot solve complex mathematical expressions like polynomial equations even differential equations; 3) LLMs are highly inefficient at expressing iteration, especially when the number of iteration steps is large.
In order to solve these issues, we propose program-of-thoughts (PoT) prompting, which will delegate com- putation steps to an external language interpreter. In PoT, LMs can express reasoning steps as Python programs, and the computation can be accomplished by a Python interpreter. We depict the difference between CoT and PoT in Figure T In the upper example, for CoT the iteration runs for 50 times, which leads to extremely low accuracy[ in the lower example, CoT cannot solve the cubic equation with language models and outputs wrong answer. In contrast, in the upper example, PoT can express the iteration pro- cess with few lines of code, which can be executed a Python interpreter to derive accurate answer; and in the lower example, PoT can convert the problem into a program that relies on 'SymPy' library in Python to solve the complex equation.
We evaluate PoT prompting across five MWP datasets, GSMSK, AQuA, SVAMP, TabMWP, MultiArith: and three financial datasets, FinQA, ConvFinQA, and TATQA. These datascts cover various input formats including lext, tables, and conversation. We give an overview of the results in Figure 2) Under both few- shot and zero-shot settings, PoT outperforms CoT significantly across all the evaluated datasets. Under the few-shot setting, the average gain over CoT is around 8% for the MWP datasets and 15% for the financial datasets. Under the zero-shot setting, the average gain over CoT is around 12% for the MWP datasets. PoT combined with self-consistency (SC) also outperforms CoT+SC (Wang et al. 2022b by an average of 10% across all datasets. Our PoT+SC achieves the best-known results on all the evaluated MWP datasets and
Assuming each addition correct with 90% chance, after 50 additions, the likelihood of correct output less than 1%.
2

---

Published in Transactions on Machine Learning Research (10/2023)
IoCoT II PoT
76.4
71.0
73.2
89
64.5
61.4
54.1
GSM8K
AQuA
SVAMP
TabMWP
FinQA
ConyFin Fin
TATQA
LOCoT-SC IPPO--SC
80
87.-5
73.4
68-1
67.33
47.
GSM8K
AQuA
SVAMP
TabMWP
FinQA
ConyFin
TATQA
luzs-CoTlozs-Pe
70.8
65.2
60.4
57
52.5
43.9
40.5
2:6 2
GSM8K
AQuA
SVAMP
TabMWP
FinQA
Comy Fin
TATQA
Figure 2: Few-shot (upper), Few-shot + SC (middle) and Zero-Shi Shot (lower) Performance overview of Codex PoT and Codex CoT across different datasets.
near best-known results on the financial datasets (excluding GPT-4 (OpenAl 2023 ). Finally, we conduct comprchensive ablation studies to understand the different components of PoT.
2 Program of Thoughts
2.1 Preliminaries
In-context learning has been described in Brown et al. (2020): Chen et al. (0021a); Chowdhery al. (2022 Rae al. (2021). Compared with fine-tuning, in-context learning (1) only takes few annotations/ demon- strations a prompt, and (2) performs inference without training the model parameters. With in-context learning, LLMs receive the input-output exemplans as the prefix, followed by an input problem, and generate outputs imitating the exemplars. More recently, 'chain of thoughts prompting' (Wei et al. 2022) has been proposed as a specific type of in-context learning where the exemplar's output contains the 'thought process' or rationale instead of just an output. This approach has been shown elicit LLMs' strong reasoning capabilities various kinds of tasks.
3

---

Published in Transactions on Machine Learning Research (10/2023)
Exemplars
Toulouse has twice many sheep Charleton. Charleston has 4 times as many sheep as Seattle. How many sheep do Toulouse, Charleston, and Seattle have together if Seattle has sheep?
# Question:
Thought:
Questior
Thought:
Questior
Thought:
Questior
Thought:
Questior
Thoughts:
Question:
Answer this question by implementing solver() function. def solver)::
Train 1 departs 6 AM from city and arrive city B in AM. Train 2 departs at 7:30AM from city and arrive in City at 11 AM. both trains depart from their station simuttaneously, when will they meet?
Let's write Python program step by step. and then return the answer
Firstly, we need following var
Prompt
Prompt
LLM
LLM
train1 travel time # 7:30 AM refers to 7.5
Scaètle 30
Charleston Seattle Charlaston
train2_ trein2_travel_ twwvel time 11 77.5
Symbol("D", positive-True) train _speed ddistanoe_of _____o___ohh_ brain bravell time train2_ distance_off cityy/ train2, travel time ana bisttancepp_cttt (trainl_spead train2 _ppedd
distance_of htt_ooo___ty____eeee cltt___end__________
# Then, need calculate the sum of the three sum Seattle Charleston Toulouse Finally, return sum
Figure 3: Left: Few-shot PoT prompting, Right: Zero-shot PoT prompling.
2.2 Program of Thoughts
Besides natural language, programs can also be used to express our thought processes. By using semantically meaningful variable names, a program can also be a natural representation to convey human thoughts. For ex- ample, in the lower example in Figure 1 we first create an unknown variable named interest rate. Then we bind "summation in two years with interest rate' to the variable sum in_ Sa_tt________ awo__eerr___ years_with, with_XXX___ XXX_xixereettt interest and write down the equation expressing their mathematical relations with interest paate. These equations packaged into the 'solve' function provided by 'SymPy'. The program is executed with Python to solve the equations to derive the answer variable interest_rate
Unlike CoT, PoT relegates some computation to an external process (a Python interpreter). The LLMs are only responsible for expressing the 'reasoning process' in the programming language. In contrast, CoT aims to use LLMs to perform both reasoning and computation. We argue that such an approach is more expressive and accurate terms of nurericaa reasoning.
The 'program of thoughts' is different from generating equations directly, where the generation target would be solve(20000 11:0000001++000 (1+ 2000 = 20000 + 1000, z)). As observed by Wei et al. (022) for CoT, directly generating such equations is challenging for LLMss. PoT differs from equation generation in two aspects: (1) Pol breaks down the equation into multi-step 'thought' process, and (2) PoT binds semantic meanings to variables to help ground the model in language. We found that this sort of 'thoughtful' process can elicit language models' reasoning capabilities and generate more accurate programs. We provide a detailed comparison the experimental section.
We show the proposed PoT prompting method Figure under the few-shot and zero-shot settings. Under the few-shot setting, few exemplars of (question, 'program of thoughts") pairs will be prefixed as demon- strations to teach the LLM how to generate 'thoughtful' programs. Under the zero-shot setting, the prompt only contains an instruction without any exemplar demonsttratio.. Unlike zero-shot CoT (Kojima et al. 2022), which requires an extra step to extract the answer from the 'chain of thoughts', zero-shot PoT can return the answer straightforwardly without extra steps.
In zero-shot PoT, a caveat is that LLM can fall back to generating a reasoning chain in comments rather than in the program. Therefore, propose to suppress '#' token logits to encourage it to generate programs.
4

---

Published in Transactions on Machine Learning Research (10/2023)
LLM
prompting
CoT
Answer
LLM
Question
PoT
Progarm
Result
EOS
Answer
Figure PoT combined with CoT for multi-stage reasoning.
2.3 PoT as an Intermediate Step
For certain problems requiring additional textual reasoning, we propose to utilize PoT to tackle the compu- tation part. The program generated by PoT can be executed to provide intermediate result, which is further combined with the question to derive the final answer with COT. We depict the whole process in Figure 8
During demonstration, reasoning needs to be used. If LLM outputs 'keep prompting' in the end, we will adopt the execution results from Pol as imput to further prompt LLMs to derive the answer through CoT.
present LLMs with examples to teach it predict whether to additional CoT
For instance, in the left example in F Figure 3 the program will be executed to return float number 'ans=2.05'; which means that after 2.05 hours the two trains will meet. However, directly adding 2.05 to AM does not make sense because 2.05 hour needs to be translated to minutes to obtain the standard HH:MM time format to make it aligned with provided option in the multi-choice questions. Please note that this prompting strategy is only needed for the AQuA because the other dalasets can all be solved by PoT-only prompling.
3 Experiments
3.1
Experimental Setup
Datasets We summarize evaluated datasets in Table We use the test set for all the evaluated datasets except TATQA. These datasets are highly heterogencous in terms of their input formats. We conduct comprehensive experiments on this broad spectrum of datasets to show the generalizability and applicability of PoT prompting.
Dataset
Split
Example
Domain
Input
Output
GSM8K Cobbe et al.
2021
Test
1318
MWP
Question
Number
AQuA Ling al. 2017
Test
253
MWP
Question
Option
SVAMP Patel et al.
2021
Test
1000
MWP
Question
Number
MultiArith (Roy Roth
2015
Test
600
MWP
Question
Number
TabMWP [Lu et al. 2022
Test
7861
MWP
Table + Question
Number + Text
FinQA (Chen et 20216
Test
1147
Finance
Table + Text Question
Number + Binary
ConvFinQA (Chen al.
2022
Test
421
Finance
Table + Text + Conversation
Number + Binary
TATQA Zhu 2021
Dev
1668
Finance
Table + Text Question
Number + Text
Table 1: Summarization of all the datasets being evaluated.
To incorporate the diverse inputs, we propose to lincarize these inputs in the prompt. For table inputs, we adopt the same strategy as Chen (2022) to linearize table inlo a lext string, The columns of the table are separated by 'I' and the rows are separated by "(n' a table cell is empty, is filled by *.. For text- hybrid inputs, we soparate tables and text with 'nn'. For conversational history, we also separate conversation turns by 'nn.. The prompt is constructed by the concatenation of task instruction, text, linearized table, and question. For conversational question answering, simply concatenate all the dialog history in the prompt..
5

---

Published in Transactions on Machine Learning Research (10/2023)
Implementation Details We mainly use the OpenAI Codex (code-davincc-002) A for experi- ments. We also tested GPT-3 (1ext-davincii0002) Ghat (gpt-turbo-3.5) CodeGen Nijkamp et al. CodeTT5+ (Wang et al. 2023b) and Xgen for ablation experiments. We use Python 3.8 with the SymPy library to execute the generated program. For the lew-shot setting, we use 4-8 shots for all the datasets, based on their difficulty. For simple datasets like FinQA (Chen et al. 202Tb), we tend to use fewer shots, while for more challenging datasets like AQuA (Ling ett al., 2017) and TATQA (Zhu et al., 2021), we use 8 shots cover more diverse problems. The examples are taken from the training set. We generally write prompts for 10-20 examples and then tune the exemplar selection a small validation set to choose the best 4-8 shots for the full set evaluation.
2022) Codegenn-66B-mults and (edegen--66--mone
To elicit the LLM's capability to perform multi-step reasoning, we found a prompt to encourage LLMs to generate reasonable programs without demonstration. The detailed prompt is shown in Figure 3] However, a caveat that LLM can fall back to generating a reasoning chain in comments rather than the program. Therefore, we suppress the '#' token logits by a small bias to decrease its probability to avoid such cases. In our preliminary study, we found that -2 as the bias can achieve the best result. We found that this simple strategy can greatly improve our performance.
Metrics We adopt exact match scores as our evaluation metrics for GSM8K, SVAMP, and MultiArith datasets. We will round the predicted number to a specific precision and then compare it with the reference number. For the AQuA dataset, use PoT to compute the intermediate answer and then prompt the LLM again to output the closest option to measure the accuracy. For TabMWP, ConvFinQA, and TATQA datasets, we use the official evaluation scripts provided on Github. For FinQA, we relax the evaluation for CoT because LLMs cannot perform the computation precisely (especially with high-precision floats and large numbers), so we adopt mathh.sscosse with relative tolerance of 0.001 to compare answers.
Baselines We report results for three different models including Codex (Chen et al. 2021a). GPT-3 Brown
et al. 2020 PaLM (Chowdhery et al. 2022) and LaMDA (Thoppilan et al., 2022). We consider two types
of prediction strategics including direct answer output and chain of thought to derive the answer. Since
PaLM API is not public, we only list PaLM results reported from previous work (Wei et al. 2022 Wang
ct al.
2022b). We also leverage an external calculator as suggested in Wei et al. (2022) for all the equations generated by CoT, which is demoted as CoT cale. Besides greedy decoding, we use self-consistency (Wang et al.
b022b with CoT, taking the majority vote over 40 different completions as the prediction.
3.2 Main Results
Few-shot Results We give our few-shot results in Table 2 improves on TaAeKK/AQA/Tabaa to its simplicity. For financial QA datasets, PoT improves over CoT by roughly 20% on EinAA/ConFFin and 8% on TATQA, The larger improvements in FinQA and ConvFinQA are mainly due to miscalculations on LLMs for large numbers (c.g. in the millions). CoT adopts LLMs to perform the computation, which is highly prone to miscalculation errors, while PoT adopts a highly precise external computer to solve the problem. As an ablation, we also compare with CoTlc which leverages external calculator to correct the calculation results in the generated 'chain of thoughts!. The experiments show that adding an external calculator only shows mild improvement over CoT on MWP datasets, much behind PoT. The main reason for poor performance of 'calculator' is due to its rigid post-processing step, which can lead to low recall in terms of calibrating the calculation results.
On MWP datasets, PoT with greedy decoding by more than 8%. On SVAMP, the improvement is 4% mainly due
Few-shot + Scff-Consistency Results We leverage solf-consistency (SC) decoding to understand the
upper bound of our method. This sampling- based decoding algorithm can greatly reduce randomness in
the generation procedure and boosts performance. Specifically, we set a temperature of 0.4 and K=40
throughout our experiments. According to Table 2 we found that PoT + SC still outperforms CoT + SC
hattps://openai. http://pwwa..ccooooo//////oaaaaa 6 httpss//blogsslestrreaaess /xgen/
http://wwwwwyyy.orr///o/ html
6

---

Published in Transactions on Machine Learning Research (10/2023)
Model
# Params
GSMSK
AQuA
SVAMP
TabWMP
FinQA
ConvFin
TATQA
Avg
Line-tuned or few-shot prompt
Published SoTA
78.0
52.0
86.8
68.2
68.0
68.9
73.6
70.7
Few-shot prompt (Greedy Decoding)
Codex Direct
175B
19.7
29.5
69.9
59.4
25.6
40.0
55.0
42.7
Codex CoT
175B
63.1
45.3
76.1
65.2
10.4
15.6
61.4
56.7
GPT-3 Direct
175B
15.6
24.8
65.7
57.1
14.4
29.1
37.9
34.9
GPT-3 CoT
175B
46.9
35.8
68.9
62.9
26.1
37.4
42.5
45.7
PaLM Direct
540B
17.9
25.2
69.4
PaLM CoT
540B
56.9
35.8
79.0
Codex CoTCcae
175B
65.4
45.3
77.0
65.8
GPT-3 CoTeate
175B
49.6
35.8
70.3
63.4
PaLM Colaaate
540B
58.6
35.8
79.8
PoT-Codex
175B
71.6
54.1
85.2
73.2
64.5
64.6
69.0
68.9
Few-shot prompt Self-Consistency Decoding)
LaMDA CoT-SC
137B
27.7
26.8
53.5
Codex CoT-SC
175B
78.0
52.0
86.8
75.4
44.4
47.9
63.2
63.9
PaLM CoT-SC
540B
74.4
48.3
86.6
PoT-SC-Codex
175B
80.0
58.6
89.1
81.8
68.1
67.3
70.2
73.6
Few-shot prompt (GPTT-4
CoT-GPT4
175B
92.0
72.4
97.0
58.2
PoT-GPT4
175B
97.2
84.4
97.4
74.0
The few-shot results for different datasets. Published SoTA includes the best-known results (ex- cluding results obtained by GPT-4). On GSM8K, AQuA and SVAMP; the prior SoTA results CoT self-consistency decoding (Wang et al. 2022b). On FinQA, the prior best result is from Wang et al. 2002a On ConvFinQA, the prior best result is achieved by FinQANet (Chen et al. 2022).
Table 2:
On TabWMP (Lu et al.
2022), the prior best result is achieved by Dynamic Prompt Learning
(Lu et al. 2022). On TATQA, the
SoTA result is by RegHNT [Lei et al. 2022
Model
# Parrans
GSM8K
AQuA
SVAMP
TabMWP
MultiArith
Avg
Zero-shot Direct (GPT-3)
175B
12.6
22.4
58.7
38.9
22.7
31.0
Zero-shot CoT (GPT-33
31.9
63.7
53.5
79.3
53.7
Zero-shot Col (PaLM)
540B
43.0
66.1
Zero-shot PoT (Ours)
175B
57.0
43.9
70.8
66.5
92.2
66.1
Table 3:
The zero-shot results for different datasets. The baseline results are taken from Kojima et al.
2022
MWP datasets with notable margins. On financial datasets, we observe that self-consistency decoding is less impactful for both PoT and COT. Similarly, PoT + SC outperforms CoT + SC by roughly 20% on FinQA/ConvFinde /ConvFinQA and 7% on TATQA.
We also evaluate the zero-shot performance of PoT and compare with Kojima et al. 22022) in Table 3 As can be seen, zero-shot PoT significantly outperfomms zero-shot CoT across all the MWP datasets evaluated. Compared to few-shot prompting, zero-shot PoT outperforms zero-shot CoT (Kojima ct al. 2022 by an even larger margin. On the evaluated datasets, PoT's outperforms CoT by an average of 12%. On TabMWP, zero-shot PoT even higher than few-shot COT. These results show the great potential to directly generalize to many unseen numerical tasks without any dataset-speeific exemplars.
Zero-shot Results

---

Published in Transactions on Machine Learning Research (10/2023)
Model
# Params
GSM8K
SVAMP
code-davinci-002
175B
71.6
85.2
text-davinci-002
175B
60.4
80.1
8pt-3.5-turbo
76.3
88.2
codegen-16B-muli
16B
8.2
29.2
codegen-16B-mono
16B
12.7
codeT5+
16B
12.5
38.5
Xgen
7B
11.0
40.6
Table PoT prompting performance with different backend model.
10vl I v2210
0.74
0.65
0.67
0.69
0.65
0.7
0.62
0.58
0.6
2-shots
4- 4-shots
6-shots
8-shots
o.
0.06
0.63
0.68
0.58
2-shots
4-shots
6-shots
8-shots
Figure 5: Exemplar sensitivity analysis for GSM8K and FinQA, where vl, v2 and v3 three versions of k-shot demonstration sampled from the pool.
3.3 Ablation Studies
We performed multiple ablation studies under the few-shot sctting understand the importance of different factors in PoT including the backbone models, prompt engineering, etc.
Backend Ablation To understand PoT's performance different backbone models, we compare the performance of text-davinci-002,, code-davinci-0022, gpt-3.5-turbo, codegen-16B-mont codegen-16B-malee CodeT5+ and XGen. We choose three representative datasets GSM8K, SVAMP, and FinQA to analyze the results. We show our experimental results in Table 4 As can be seen, gpt-3.5-turbo can achieve the highest score to outperform codcx (codo-davinci-002) by a remarkable margin. In contrast, text-davinci- 002 is weaker than code-davinci-0002, which mainly because the following text-based instruction luning undermines the models' capabilitios to generate code. A concerning fact we found that the open source model like codegen Nijkamp al. (2022) significantly behind across different benchmarks. We conjecture that such huge gap could be attributed to non-sufficient pre-training and model size.
Sensitivity to Exemplars To better understand how sensitive PoT is w.r.t different exemplars, con- duct sensitivity analysis. Specifically, we wrote 20 total exemplars. For k-shot lcarning, we randomly sample = (2, 4, 6, 8) out of the 20 exemplars three times as v1, v2, and v3. We will use these randomly sampled exemplars as demonstrations for PoT. We summarize our sensitivity analysis in Figure First of all, we found that increasing the number of shots helps more for GSMSK than FinQA. This mainly due the diversity of questions in GSMSK. By adding more exemplars, the language models can better generalize to diverse questions. Another observation is that when given fewer exemplars, PoT's performance variance is
8

---

Published in Transactions on Machine Learning Research (10/2023)
Model
GSMsK
GSM8K-Hard
SVAMP
ASDIV
ADDSUR
MULTIARITH
PaL
72.0
61.2
79.4
79.6
92.5
99.2
Pol
71.6
61.8
85.2
85.2
92.2
99.5
Table Comparison of PoT against contemporary work PaL.
(Gao et al. 2022
Method
GSM8K
SVAMP
FinQA
PoT
71.6
85.2
64.5
PoT Binding
60.2
83.8
61.6
PoT MultiStep
45.8
81.9
58.9
Table 6: Comparison between PoT and equation generation on three different datasets.
larger. When K=2, the performance variance be as large 7% for both datasets. With more exemplars, the performance becomes more stable.
We also compare PoT with another more recent related approach like PaL (Gao et al. 2022). According to to Table 5 found that method is in general better than Pal., especially SVAMP and ASDIV. Our results are 6% higher than their prompting method.
Comparison with PaL
Semantic Binding and Multi-Step Reasoning The two core properties of 'program of thoughts' are: (1) multiple steps: breaking down the thought process into the step-by-step program, (2) semantic binding: associating semantic meaning to the variable names. To better understand how these two properties con- tribute, we compared with two variants. One variant is to remove the semantic binding and simply use a, b, as the variable names. The other variant is to directly predict the final mathematical equation to compute the results. We show our findings in Table 6) As can be seen, removing the binding will in general hurt the model's performance. On more complex questions involving more variables like GSM8K, the performance drop larger. Similarly, prompting LLMs directly generate the target equations is also very challenging. Breaking down the target equation into multiple reasoning steps helps boost performance.
Breakdown Analysis We perform further analysis to determine which kinds of problems CoT and PoT differ most in performance. We use AQuA (Ling al.) 2017) as our testbed for this. Specifically, we mamually classify the questions in AQuA into several calegories including geometry, polynomial, symbolie, arithmetic, combinatorics, linear equation, iterative and probability. We show the accuracy for each subcategory in Fig- The major categories (1) linear equations, (2) arithmetic, (3) combinatorics, (4) probability, and (5) iterative. The largest improvements of PoT are in the categories 'lincar polynomial equation', 'iterative', 'symbolic', and 'combinatorios!. These questions require more complex arithmetic or symbolic skills to solve. contrast, on 'arithmetic', 'probability', and 'geometric' questions, PoT and CoT perform similarly. Such observation reflects our assumption that 'program' more effective on more challenging problems.
ure 6
InPoTDICoT
72
G2
655
410
40
40
geo
poly
symb
arith
comb
lin-eq
prob
Figure 6: PoT and Col's breakdown accuracy across different types of questions.

---

Published in Transactions on Machine Learning Research (10/2023)
Type 1 Error: incorrect Value Grounding
Ruad following lext and table, und then write vode quesi on:
Comparative information of the Company's overall revenues by end-use markets and geography for years endec June 30,
2019, 2018 and 2017 were as follows:
Context: (S millions) 2019 2018 2017
Aerospace and Defense $1,327.9 | $1,182.3 $973.3
Medical 205.0 175.3 125.5
Distribution 136.4| 131.7| 1 118.7
What the change in Distribution
from 2018?
Question:
What the change the beginning balance between fiscal years 2019 and 2018?
distribution, In 2018 1317
The model generate the computation logic correctly however, the model accldentally assign 131.7 and 136.4 as 1317 and 1364. The model ground the value of relevant variables incorrectly.
Output:
distribution In 2019 1364 ans cistribution 20_9019 distribution _
units 'million' <
Type Error: Incorrect Logic Ganeration
Read the following text and table, and then write code arswer quest Net operating losses and tax attributes generated by domestic and foreign entities in closed years and utilized in open years are subject adjjstmmnt by the tax authorities. 2019 2018
Context:
Balance at the beginning of the fiscal year $1,264| | $1,626
apse in statute limitations | -29 -58
Balance the end fiscal year $11255 $1,264
Question:
What is the change in the beginning balance between fiscal years 2019 and 2018?
beginning balancce_ 2018 1626
variables values grounded correctly, however, the change of buginning balance should be computed by thu value of 2019 subtracted by the value of 2018. The model generate computation logic incorrecly.
beginning__baie baalila_llaaeeeeeeeeeeee_ 2011aaacee_211110__1 2019 = 1264
Output:
ans beginning balance__018 beginning.bbalncee.200 balance__019 X
'thousand'
Figure 7: Error cases on TAT-QA dev set using Fore-yy method.
Error Analysis We considered two types of crrors: (1) value grounding error, and (2) logic generation error. The first type indicates that the model fails to assign correct values to the variables relevant to the question. The second type indicates that the model fails to generate the correct computation process to answer the question based on the defined variables. Figure 7 shows an example of cach type of error. the upper example, the model fetches the value of the variables incorrectly while the computation logic is correct. In the lower example, the model grounded relevant variables correctly but fails to generate proper computation logic answer the question. We manually examined the errors made in the TAT-QA results. Among the 198 failure cases of numerical reasoning questions with the PoT (greedy) method, 47% have value grounding errors and 33% have logic errors. In 15% both types of errors occurred and in 5% we believe the answer is actually correct. We found that the majority of the errors are value grounding errors, which is also common for other methods such as CoT.
4
Related Work
4.1
Mathematical Reasoning in NLP
Mathematical reasoning skills are essential for general-purpose intelligent systems, which have attracted a significant amount of attention from the community. Earlier, there have been studies in understanding NLP models' capabilitios to solve arithmetic/ caltimttttcalllbaaaiii algebraic questions (Hosseini et al.. 2014 Koncel-Kedziorskt et al.
2015 Roy & Roth 2015 Ling al.) 2017 Roy & Roth 2018). Recently, more challenging datasets (Dual
et al. 2019: Saxton al. 2019: Miao et al. 2020 Amini et al 2019 Hendrycks et al., 2021 Patel et al. 2021
have been proposed to increase the difficulty, diversity or adversarial robustness. LiLA (Mishra et al.
2022 proposes to assemble large set of mathematical datasots into a unified datasct. LiLA also annotates
Python programs as the generation larget for solving ma mathematical problems. However, LiLA (Mishra et al. p is mostly focused on dataset unification. Our work aims to understand how to generate "thoughtful programs' to best clicit L.L''s reasoning capability. Besides, we also investigate how to solve math problems without any exemplars. Austin et al. (2021) propose to evaluate LLMs capabilities to synthesize code on two curated datasets MBPP and MathQA-Python
10

---

Published in Transactions on Machine Learning Research (10/2023)
4.2 In-context Learning with LLMs
GPT-3 [Brown et al. 2020 demonstrated strong capability to perform few-shot predictions, where the model is given description of the task in natural language with few examples. Scaling model size, data, and computing are crucial to enable this learning ability. Recently, [ae et al. (2021); Smith et al. (2022); Chowdhery et al. (2022): ; Du et al. (2022) have proposed to train different types of LLMs with different training recipes. The capability to follow few-shot exemplars to solve unseen tasks is not existent on smaller LMs, but only emerge as the model scales up (Kaplan et al., 2020). Recently, there have been several works Xie et al., 2021; Min al., 2022) aiming to understand how and why in-context learning works. Another concurrent work similar to ours is BINDER (Cheng al. 2022). which applies Codex to synthesize 'soft' SQL queries to answer questions from tables.
4.3 Chain of Reasoning with LLMs
Although LLMs have demonstrated remarkable success across a range of NLP tasks, their ability reason is often seen as limitation. Recently, CoT (Wei et al., 2022 Kojima et al.) 2022 Wang et al. 2022b)
proposed to enable LLM's capability to perform reasoning tasks by demonstrating 'natural language rationales' Suzgun et al. 20222) have shown that CoT can already surpass human performance on challenging
BIG-Bench tasks. Later on, several other works (Drozdov et al.) 2022 Zhou et al.) 2022 Nye et al. 2021
also propose different approaches to utilize LLMs to solve reasoning tasks by allowing intermediate steps. ReAct Yao ct al. (2022) propose to loverage external tools like search engine to enhance the LLM reasoning skills. Our method can be seen as augmenting CoT with external lools (Python) to enable robust numerical reasoning. Another contemporary work (Gao et al.) 2022 was proposed at the same time as ours adopt hybrid text/code reasoning address math questions.
4.4 Discussion about Contemporary Work
Recently, there has been several follow-up work on top of PoT including self-critic (Gou et al.) 2023). self- eval (Xie et al. 2023), plan-and-solve (Wang et al.) 2023a]. These methods propose to enhance LLMs capabilitics to solve math problems with PoT. self-critic (Gou et al.] 2023) and self-eval (Xie et al. 2021 both adopt self-evaluation to enhance the robustness of the generated program. plan-and-solve (Wang al. 2023a) instead adopt more detailed planning instruction to help LLMs create high-level reasoning plan. These methods all prove to bring decent improvements over PoT on different math reasoning datasets.
Another line of work related to ours is Tool-use in transformer models (Schick et al. 2023 Paranjape et al. 2023). These work propose to adopt different tools to help the language models ground on external world. These work generalizes our Python program into more general AP1 calls to include search engine, string extraction, etc. By generalization, LLMs can unlock its capabilities to solve more complex reasoning and grounding problems in real-world scenarios.
5 Discussion
In this work, we have verified that our prompting methods can work efficiently on numerical reasoning tasks like math or finance problem solving. We also study how to combine PoT with CoT to combine the merits of both prompting approaches. We believe PoT is suitable for problems which require highly symbolic reasoning skills. For semantic reasoning tasks like commonsense reasoning (StrategyQA), we conjecture that PoT is not the best option. In contrast, CoT can solve more broader reasoning tasks.
6 Conclusions
In this work, we investigate how to disentangle computation from reasoning in solving numerical problems. By 'program of thoughts' prompting, we are able to clicit L.LMs' abilities to generate accurate programs express complex reasoning procedure, while also allows computation to be separately handled by an external program interpreter. This approach is able to boost the performance of LLMS on several math datasets
11

---

Published in Transactions on Machine Learning Research (10/2023)
significantly. We believe our work inspire more work to combine symbolic execution with LLMs to achieve better performance on other symbolic reasoning tasks.
Limitations
Our work aims at combining LLM with symbolic execution to solve challenging math problems. PoT would require execution of 'generated code' from LLMs, which could contain certain dangerous or risky code snippets like 'import os; os. (s.rmdir()), elc. We have blocked the LLM from importing any additional modules and restrict it to using the pre-defined modules. Such brutal-force blocking works reasonable for math QA, however, for other unknown symbolic tasks, it might hurt the PoT"s generalization. Another limitation is that PoT still struggles with AQuA dataset with complex algebraic questions with only 58% accuracy. It's mainly due to the diversity questions in AQuA, which the demonstration cannot possibly cover. Therefore, the future research should discuss how to further prompt LLMs to generate code for highly diversified Math questions.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedzzorshi, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, el al. Program synthesis with large language models. arXiv preprint Xie:22108. 07732, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Mark Chen, Jerry Tworek, Hcewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiu preprint ar arXiv:2107.03377 033744 2021a.
Wenhu Chen. Large language models are few (1)-shot table reasoners. arXiv preprint arXiv:2210.06710 2022.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Tana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Naturol Language Processing, pp. 3697-3711, 3711, 2021b.
Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. arXiv preprint arXiv:2210.080 03849, 2022.
Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, el al. Binding language models in symbolic languages. ar preprint arXiv:2210.022 02875,, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ar preprint orXiv:2204.02311 2022.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint or 2rxiv:2100.11 141641 2021.
12

---

Published in Transactions on Machine Learning Research (10/2023)
Andrew Drozdov, Nathanacl Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. Compositional semantic parsing with large language models. arv preprinl arXiv:2209.15003 2022.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling language models with mixture- of-experts. In International Conference on Machine Learning, pp.
PMLR, 2022.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Associalion for Compulational Linguislies: Human Language Technologies, Volume (Long and Short Papers), pp. 2368-2378, 2019.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamic Callan, and Graham Neubig. Pal: Progran-aided language models. arXiv preprinl arXiv:2211 11.112211.00435, 2022.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, arXiv preprint arXiv:2305. 11738, 2023.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. orXiv preprint arXiv:2103. 03..07877 2021.
Mohammad Javad Hosscini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning solve arithmetic word problems with verb calegorization. In EMNLP, pp. 523 533, 2014.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXie preprint arXiv:2001.0888 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint drXim:2205.11916, 2022.
Rik Koncel-Kedziorse Hannanch Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Pars- ing algebraic word problems into equations. Transactions of the Associalion for Computational Linguislics, 2015.
Fangyu Lei, Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. Answering numerical reasoning questions in table-text hybrid contents with graph- encoder and trco-based docoder. In Proceedings of the 29th International Conference on Computational Linguisties, pp. 1379-1393 2022.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annuul Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 158 167, 2017.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tammay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi stritusructured mathematical reasoning. arXiv preprint Xr:::22209 14610, 2022.
Shen- Yun Miao, Chao-Chur Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 975 -84,,,, 2020.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arxiv:200 12837, 2022.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arxiv preprinl arXiv:2203. 13/74, 2022.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyzz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 0rXiv:2112.00111 2021.
OpenAL. Gpt-4 technical report. arxiv preprint wxXi::23330087// 2023.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirxi, Luke Zettlemoyer, and Marco Tulio Ribeiro. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprinl rrXi:2333300011, 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Online, June 2021. Association
Computational Linguisties: Human Language Technologies, pp. for Computational Linguistics. doi: 10.18653/v1 mag.:///m11222...macmmm..... mwwcl-main. 168. URL https://accaanthologg.
org/2021. naacl -main. 168
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. Scaling language models: Methods, analysis insights from training gopher. arXiv preprint arXim:2112. 11446, 2021.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Melhods in Natural Language Processing, pp. 1743 1752, 2015.
Subhro Roy and Dan Roth. Mapping to declarative knowledge for word problem solving. Transactions of the Association for Computational Linguistics, 6:159-172, 2018.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXie preprint arXiv:1904.018 2019.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint Xrv::2230 04761, 2023.
Shaden Smith, Mostofa Patwary Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train megairon-turing nlg 530b, large-scale generalive language model. arXie preprinl arXiv:2201.1 11990, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Lo, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. ar preprint or Xiv:2220 10.0220, 2022.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint Xiv:22201 08239, 2022.
Bin Wang, Jiangzhou Ju, Yunlin Mao, Xin-Yu Yu Dai, Shujian Huang, and Jiajun Chen. A numerical reasoning question answering system with fino-grained retriever and the ensemble of multiple generators for finqa. arXiv preprint ar xrXi::2206.0850 2022a.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ec-Peng Lim. Plan- and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXie:3305.0488 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXie preprint or Xiv:2203 11 11171, 2022b.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint orXiv:2005.07922 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarlen Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 2022.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Hepresentations, 2021.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Yen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-ovaluation guided decoding. orXiv preprint arXio:2305.0060 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Syner- gizing reasoning and acting in language models. In NewrIPS 2022 Foundation Models for Decision Making Workshop, 2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. trost-to-moot prompting enables complex reasoning in large language models. Xiv preprint arXiv:2205. 10625, 2022.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume Long Papers), pp-
3277-3287, 2021.
15

---

Published in Transactions on Machine Learning Research (10/2023)
7 Appendix
7.1 PoT as intermediate step
We demonstrate the workflow in Figure
LLM
Keep prompting
CoT
Answer
LLM
Question
PoT
Progarm
Result
EOS
Answer
Figure We adopt PoT prompt language models to first generate an intermediate answer and then continue to prompt large models to generate the final answer.
We write the pseudo code as follows:
# Function PoT( Input) -> Output
# Input: question
# Ouptut: program
# Function Prompt(Input) -> Output
# Input: question + intermediate
# Ouptut: answer
program = PoT( (sucssiioo)
exec program)
if isintance (ans, dict),
list items (O)
extra according to the program: /
extra - ans 0] + "u=u" + ans [1]
pred = Prompt question + extra)
clse:
pred ans
return pred
PoT intermediate step is able to address complex questions which require both symbolic and com- monsense reasoning.
7.2 Exemplars for Prompting
To enable better reproducibility, also put prompts and exemplars for GSMSK dataset and AQuA dataset in the following pages:
16

---

Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Python code, return ans total_eggs 16
eaten_eggs
baked eggs= sold_eggs eggs total_eggs" feggs-eate_eegee baked_eggs eggs
dollars_per_eedd
ans sold _ggs
dollars_perr ___________________________
Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? Python code, return ans bolts of
2 bolts_ of white fiber num of wwwebbllueeeeee / ans = bolts_oooooo of_ blue_fiber fiber + bolts_of hooo_whit___ppbbeeeeeeeeee_e
Question: Josh decides to try flipping house. He buys house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?
Python code, return ans
cost_of_originee of_orifiiall___oooo __ouseeoueee = 80000
increase_rane sataasse_erteeeeee 150 / 100
value _owee_o__aaeeeee house (1 + increase_rate) cost_off Lowioooooriiiiaaohoeo house
cost of repair 50000
ans value house cost of larioffrrepppa repair cost_oof__s_oeeoeooooooooeeel oowoooooiaaalaaaleeeaaaaaaae house
Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need give her chickens in the final meal of the day the size of Wendi's flock is 20 chickens? Python code, return ans numb of cf__chcccens = 20
Question:
cups cups_for_eaad lass_for_each____ fft__eeahhhhhhhhhee=a=__ =
jos__for_all_chine = num_ sal_off_hhhhe of_chhikens cups_for_each chicken
sups_in_the_mne
= 15
gyps_in_the_afteee
= 25
ans = cups for all all_cchhckeeeeeeaeee cups in tih__monning- cups in tee_ttee__eeeee affeernotonnn
Question: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants buy 16 glasses. How much does need pay for them?
Python code, return ans num_ 16 firstgglas__ooe cost second second_glass____ glaas_coosseettsssaaaaasssssssso 0.6 ans
for in range(num r: if % == 0:
ans += first_glass_coo cost
else:
ans += second_glass_com

---

Question: Marissa is hiking 12-mile trail. She took hour to walk the first 4 miles, then another hour to walk the next two miles. she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance?
Python code, return ans average_mil La____mmeee______________ per_hour- total cotal_trail_miles 12
remaining_miles total_trail miles 4-2
total total_hours hours total twwttttttailltnnttt trril_miles average_miia mile__er__oou hour
remaining_ potal_hours- ans remaining wwwemmlllsss. remaining heouuss hours
Question: Carlos is planting lemon tree. The tree will cost $90 to plant. Each year it will grow lemons, which he can sell for $1.5 each. It costs $3 year to water and feed the tree. How many years will it tak before he starts earning money the lemon tree? # Python code, return ans
total_cost==
cost of Sir_wwatrinng..mm and ueddedingggaaee = 3
cost_ of hum_of of__o_ffloo__eerr___ yearrr_ea== =
= 1.5
ans = 0
while total_ total_costt > 0:
total_cost += cost_of cost__ofwwttrring_... and__eeding
total__ots - num wewelleoooooeeeeeeeee cost_oo_eead Lac_fffeac__emmae lemon ans
Question: When Freda cooks canned tomatoes into sauce, they lose half their volume. Each 16 ounce of tomatoes that she uses contains three tomatoes. Freda's last batch tomato sauce made 32 ounces sauce. How many tomatoes did Freda use?
Python code, return lose_ rate 0.5
num_ tomato contained in_per__opn un_per_ounce__ane sunce__auce 3 15 ounce lice__saue_iie ji__ain_________________________ batch 32
num fom_tooato in last batch ounce sauce ji last___stttttt batch num twttatt_oontalllll pe_pper_oncc____ sauce ans = num tomato_in in last__tt___________ | / (i1-lose_raas rate)
Jordan wanted to surprise her mom with a homemade birthday cake. From reading the instructions, she knew would take 20 minutes to make the cake batter and 30 minutes to bake the cake. The cake would require 2 hours to cool and an additional 10 minutes to frost the cake. she plans to make the cake all on the same day, what is the latest time of day that Jordan can start making the cake to be ready to serve it at 5:00 pm?
Question:
Python code, return ans
minutes_to to_ to_eakeee__ooooo batter 20
minutes_to to_ taa_ttt_bbbbeeeeeee cake 30
60
minutes _ttees_to___oo_ stt_s__tt_ooos_aaao_eeeee 10
total total_minutes minutes minutes Laaiieesttt___m__
tatreesstt_mae_eee_ee matter minutes_ to tateeestt_bbkb_eaeee _ake minutes Siinute__to_cool__e cake
minutes toauutte_tt_ffot__ee__e_
total hours lotta_mmnuutes
ans = total hours

---

# Write Python Code to solve the following questions. Store your result as variable named 'ans'. from sympy import Symbol from sympy import simplify import math
from sympy import solve_it
solve_ soove_it(qquation,, variable): solving the equations and return the variable value.
Question: flight of 600 km, an aircraft was slowed down due bad weather. Its average speed for the trip was reduced by 200 km/hr and the time of flight increased by 30 minutes. The duration of the flight is:
Answer option: ['A]1 hour', 'B)2 hours', 'C)3 hours', 'D)4 hours', 'E)5 hours']
duration = Symbol(' positive=True)
delay = 30 60
total_ total_disntin disntace 600
original_ _se__d_peee___eeeee total_doa
_ace__ddstaceeeee / duration
reduced speed__peeee__eeeeee dita__disntace/ (duration + delay)
solution solve saapeettttorriillspppp seeedd_seeee__eeeeeeeeee @ecluced_speedt 200, duration) ans solution[duuratin]
Question: M men agree to purchase a gift for Rs. If 3 men drop out how much more will each have to contribute towards the purchase of the gift?
# Answer options: ['A)D/(M-3)', 'B)MD/3', 'C)M/(D-3)', 'D)3D/(M2-3MI) 'E)None of these']
M SyMbol('M')
D Sy'
cost _eetr_eefree__e cocpore_droootttttee = D/ M
cost cost_after fes___tte_ddroootttae D/ (M-3) 33
mms=simplifyccost_aftee
Question: A sum money at simple interest amounts to Rs. 815 in 3 years and to Rs. 854 in 4 years. The sum is:
# Answer option: Rs.) 650', Rs. 690', 'C)Rs. 698', 'D)Rs. 700', NE)None of these'] deposit Symbol positive=True) interest Symbol("nterrest, positive=True)
money_in_3_yeaa deposit + 3 interest
money_in_4_veare deposit + interest
solution = solee=ssoooooeeoooooiiee _atpeeeeey_n____ee__e_ years
__yarss____s_____ 815, money_in in____ears 854], [deposit, interest])
solution[deposit]
# Question: Find out which of the following values is the multiple of X, if it is divisible by 9 and 12? # Answer option: ['A]36', 'B)15', 'C)17', 'D)5', 'E)7']
options [36, 15, 17, 5, 7]
for option in options: if option % 9 0 and option % 12 0:
ans = option break

---

# Question: 35% of the employees of company are men. 60% of the men in the company speak French and 40% of the employees of the company speak French. What % of the women in the company who do not speak French?
# Answer option: ['A]4%', 'B)10%', 'C)96%' 'D)90.12%; 'E)70.77%')
num_women 65
men_speaking_ 0.6 35
employess_speaking
0.4 100
women_spe aaaien_speaking_freeee french employees_speekkngggeaee sreeaking_french.. men_mpeeeeeteeee men_speaking__rrncce french
women_ saaeen_notsspaae fwwpa//fneeeeeeeeeue_e___ womenn- loonen_speaking
ans women_ hww_nnot_peeaai__freeeeeeee
num_women
# Question: In one hour, a boat goes 11 km/hr along the stream and 5 km/hr against the stream. The speed of the boat in still water (in km/hr) is:
Answer option: ['A]4 kmph', 'B)5 kmph', 'C)6 kmph', 'D)7 kmph', 'E)8 kmph'] boat_speed haapabboooobba_speeeeaeee stream stream_speed=e
positive=True)
positive=True)
along_stream__seeed==
ggainst_stream__spedd==
solution solleiinas seeattt___eeee___e + stream s_send__edd_eeeee streastrrmmmeee srreeam_speed, boat boat_speed stream srreamm_speed
lraiins_sstraamspeeeee swweetttrrspate [boat_speet dtream_speeds
ans solution[boat_speed
Question: The difference between simple interest and C.I. the same rate for Rs.5000 for years in Rs.72. The rate of interest is?
# Answer option: ['A)10%', 'B)12%', 'C)6%', 'D)8%', 'E)4%'] interest Latresttrttttttte Symbol( rate', positive=True)
amount 5000
amount ceannntwwtthsiippeeeee inntimpleiintrrsttteee = amount (1 + interest_rate / /100)
= amount * (1 + interest_ rate / /00) ** 2
souuiion=sssveejiitammountt setttt_wtthh_eeeee wwwpl/oodmmmeeereeeeee amount amoountwwthhsimpple loapll_iittettttt 72, interest_rate
ans solutionlinterrsttratee
# Question: The area of a rectangle is 15 square centimeters and the perimeter is 16 centimeters. What are the dimensions of the rectangle?
# Answer option: ['A]2&4', 'B)3&5', 'C)4&6', 'D)5&7', 'E)6&8'] width Symbol('width', positive=True) height Symbol('height' positive=True)
area 15
permimeter = 16
solution solve solve_it([widds it([width> height area, (width + height) permimeter), [width, height])
ans (silutionwwidth), solution[height))


Reviews:
[{"review_id": "jafp1QXB6Q", "reviewer": "Reviewer_7mvy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1). This paper introduces a novel method for assessing LLMs, utilizing their coding proficiencies to tackle mathematical problems. Extensive comparative analysis between PoT and CoT underscores the potency of PoT through multiple dimensions of evaluation.\n2). The investigation into PoT is thorough, encompassing an in-depth comparation between PoT and CoT. Additionally, it explores the prowess of PoT in few-shot learning scenarios involving varying shot quantities.\n\nWeaknesses:\n1). Certain technical intricacies require more lucid elucidation. For instance:\n       a) Clarification is needed regarding the term \"bias\" in the context of \"-2 as the bias\" mentioned on page #6, first line.\n       b) Illustrative instances of \"few-shot\" applications could enhance comprehension.\n2). The paper's impact could be strengthened if a similar study of GPT-4 is conducted with it's exterior applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces an innovative approach to LLM prompting, termed PoT, which harnesses the code generation capabilities of LLMs to address mathematical challenges. The study showcases their experimental work on PoT, contrasting the outcomes with the conventional CoT methodology, revealing substantial enhancements in performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Describe more on how the \"#\" is suppressed in the research.\n2. Provide illustrative examples showcasing instances of few-shot learning exemplars.", "broader_impact_concerns": "I have no ethical concerns in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2o3aGAJL6I", "reviewer": "Reviewer_k1bc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nA reasonable extension of chain-of-thoughts prompting strategy that makes a lot of sense (letting LLMs generate executable statements that are executed by external tools and not the LLMs)\nGood performance improvance wrt CoT on a variety of datasets where the input modalities vary from text to tables and conversations and covers two domains - math word problems and finance.\nThe paper is very clear and easy to understand\nWeakness:\nIt would be nice to differentiate this paper’s contribution from LLMs ability to use external tools (e.g. APIs). Seen through that lens, this paper investigates the ability of LLMs to produce inputs (programs) for a particular kind of tool (python interpreter). Is that a correct way to see it? If not, how do you differentiate this work?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces program-of-thought prompting in which a large language model (LLM), instead of generating rationales in natural language, generates programs that can be executed by an external program interpreter. The paper argues that LLMs are error prone when performing numerical computation (esp with inputs of large magnitude) and hence it proposes to decouple computation from reasoning. The authors rigorously evaluate their method on 8 datasets and find that their performance outperforms Chain-of-thought (CoT) based prompting strategies significantly. The LLM used for the model is Codex ((code-davinci-002). They also compare to  GPT-3 (text-davinci-002), ChatGPT (gpt-turbo-3.5), and CodeGen.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In sec 2.3, it is unclear how you make the model decide when to stop or use the intermediate result and the question to continue prompting. Also, in the specific example used, it seems like no additional prompting is needed because in Fig 3 (left), it seems the model is capable of automatically transforming 7:30 to 7.5, therefore it should also be able to do the reverse i.e. convert 2:05 to HH:MM. Please address this in the next draft as knowing when to stop or continue prompting is very critical to understand how your model operates and will help me make the decision for the paper. \nPage 5 (Therefore, we suppress the ‘#’ token logits by -2…) -> can you mention this before in the paper (e.g. in Sec 2.2). This is an important detail and an average reader might miss it. Can you also justify how you came up with ”-2”? Was it done using a validation set?\nSec 3.2 - How do you explain the difference in performance between CoT + calc and PoT? Is it because PoT is explicitly prompted to generate programs? Did you try using the same prompts for CoT and PoT experiments?\n", "broader_impact_concerns": "There is a limitation section which I thought was well-written.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LEw3dKnLne", "reviewer": "Reviewer_Dt1q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Given the considerable attention of chain of thought prompting techniques in the community, this paper provides a reasonable alternative by wrapping natural language problems into programs and employing external tools to execute those programs. There are various benefits after using PoT. On one hand, the input could be simplified; thus saving the overhead against the length limit. On the other hand, PoT has the potential to correct mistakes in the intermediate steps before producing the answer.\n\n- The paper conducts extensive experiments on five math word problem datasets and three financial datasets. These datasets cover various input formats, including text, tables, and conversation. This paper also takes into account the influence of different backend LLMs and learning paradigms (few-shot learning and zero-shot learning), and sensitivity to demonstration exemplars, which improves the credibility of the findings.\n\n- This work reports impressive results compared with the standard CoT techniques for numerical reasoning tasks. The effectiveness of the proposed technique is well justified.\n\n**Weaknesses**\n\n- The proposed PoT technique --- while reasonable --- is based on the assumption that the concerned problem is decomposable into programs. It remains unclear what will happen if the assumption is not satisfied. Adding a clarification would help position the work with more clear and precise application scenarios.\n\n- As there are many recent studies in the literature that have prompted LLMs by program synthesis or symbolization to solve reasoning problems [1-6], it is highly recommended to discuss those existing studies and show how this work differs from them either in the idea or methodology and compare the performance in the experiment table.\n\n- There are still various questions, technical mistakes, and stylistic suggestions (see the detailed comments below) that are not resolved yet.\n\n**Questions & Comments**\n\n- This paper considers the Codex and text-davinci-002 as the backend LLMs, and Codex has achieved much better performance. As Codex has been officially shutting down, it is interesting to see if PoT also performs impressively with other public backends to help reproducibility, such as CodeGen [7]. \n\n- In Section 2.3: \"For instance, in the left example in Figure 3, the program will generate a float number 2.05. However, adding 2.05 to 11 AM cannot be easily handled by a Python program.\" --- 2.05 is not shown in Figure 3. Please check if the text mismatches the figure.\n\n- The implementation prompt is not unclear. In Figure 3, the prompt is \"# Answer this question by implementing a solver() function \\n def solver(): \\n # Let's write a Python program step by step \\n First we need to define the following variable\", while in Section 3 the implemented prompt is claimed to be \"Let's write a Python program step by step.\" The inconsistent prompts may raise confusion on reproducibility.\n\n- \"We generally write prompts for 10-20 examples, and then tune the exemplar selection on a small validation set to choose the best 4-8 shots for the full set evaluation\" --- It is highly recommended to provide the selected exemplars to help reproduction.\n\n- \"Unlike Codex, GPT-3 is not optimized for generating programs, and one would expect degraded performance with GPT-3 as the LLM.\" --- The argument is incorrect.  GPT-3 represents text-davinci-002 in this work, which is a supervised instruction-tuned model based on code-davinci-002. It has already been optimized for generating programs.\n\n**Suggestions**\n\n- The presentation of this paper can be improved. This paper confuses the use of parenthetical citations and textual citations. This issue happens throughout the whole paper. Please proofread the paper to avoid those mistakes.\n\n- When introducing the chain of thought prompting work, it is suggested to cite [8] as it introduced the basic idea before being rebranded \"chain of thought.\"\n\n**References**\n\n[1] Gao, Luyu, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. \"Pal: Program-aided language models.\" In International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023.\n\n[2] Yao, Shunyu, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. \"Tree of thoughts: Deliberate problem solving with large language models.\" arXiv preprint arXiv:2305.10601 (2023).\n\n[3] Cheng, Zhoujun, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong et al. \"Binding Language Models in Symbolic Languages.\" In The Eleventh International Conference on Learning Representations. 2022.\n\n[4] Gou, Zhibin, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. \"Critic: Large language models can self-correct with tool-interactive critiquing.\" arXiv preprint arXiv:2305.11738 (2023).\n\n[5] Wang, Lei, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. \"Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.\" arXiv preprint arXiv:2305.04091 (2023).\n\n[6] Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. \"ReAct: Synergizing Reasoning and Acting in Language Models.\" In The Eleventh International Conference on Learning Representations. 2022.\n\n[7] Nijkamp, Erik, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. \"CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.\" In The Eleventh International Conference on Learning Representations. 2022.\n\n[8] Nye, Maxwell, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan et al. \"Show your work: Scratchpads for intermediate computation with language models.\" arXiv preprint arXiv:2112.00114 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a program of thoughts (PoT) technique, which uses large language models such as GPT-3.5 and Codex to generate text and programming language statements before inferring the answer. The programming language statements are executable by the Python interpreter. PoT is evaluated on five math word problem datasets and three financial-QA datasets in both few-shot and zero-shot settings. Results show that PoT achieves substantial performance gains over previous methods, and PoT has been further enhanced by self-consistency decoding.\n\nThis work advances the research in the field of mathematical reasoning by proposing a technique called PoT, which induces large language models to generate programs to express complex reasoning procedures before producing the answer. Compared with previous chain-of-thought techniques, PoT allows computation to be separately handled by an external program interpreter. Therefore, it is more suitable for arithmetic problems that require accurate calculations. This work also reports impressive results on widely-used arithmetic datasets.\n\nBeyond the performance gains, this paper found that associating semantic meaning to the variable names is critical, which performs much better than directly using variable names.  In addition, breaking down the target equation into multiple reasoning steps helps boost performance, as prompting LLMs to directly generate the target equations is very challenging. The error analysis reveals that PoT still suffers from value grounding errors and logic generation errors, which indicate prospect optimization directions for future studies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparing existing studies that also prompt LLMs to generate programs to solve reasoning problems.\n\n2. Resolving the confusing and problematic claims.\n\n3. Improving the presentation quality of the paper.", "broader_impact_concerns": "I do not find any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a clear contribution, backed up by strong empirical evidence. The authors have addressed all the reviewer comments in their response and revised version. The reviewers unanimously recommend acceptance.
Decision: Accept (poster)